Work

A unified model for image understanding and generation based on cross-modal representation learning

Year: 2026
Type: article
Abstract: Unified image understanding and text-conditioned image generation require models to coordinate discriminative semantic representations with generation-compatible latent representations. Existing approaches often rely on either strongly shared representations, which may cause task interference, or separated branches, which may weaken semantic transfer. This paper proposes CSB-UG, a unified understanding–generation framework based on a cross-modal semantic bridge. CSB-UG projects visual features, textual semantics, and generative latents into an intermediate bridge space, while task-adaptive adapters preserve task-specific transformations. It further introduces controlled bidirectional collaboration and a patch-token consistency objective to enhance semantic guidance, latent feedback, and local alignment. Experiments on MS-COCO Captions, VQAv2, Visual Genome, and GenEval evaluate retrieval, visual question answering, captioning, text-to-image generation, generation alignment, and fine-grained semantic consistency. Results show that CSB-UG achieves competitive performance across the evaluated understanding and generation tasks, with improvements in retrieval, captioning, generation alignment, and local semantic consistency. Qualitative analyses further suggest improved constraint preservation and object-, attribute-, and relation-level alignment, while failure cases show that complex spatial relations, counting, long-prompt following, and rare attributes remain challenging. These results indicate that bridge-based semantic coordination is a useful design choice for unified image understanding and generation under the evaluated settings.
Author: Xiaoping Gu
Language: English
Open Access status: gold
Estimated APC paid: $1,590
OpenAlex ID: W7235404721
FWCI:
Cites: 29
Cited by:
Related to: 10