A unified model for image understanding and generation based on cross-modal representation learning
Year: 2026
Type: article
Abstract: Unified image understanding and text-conditioned image generation require models to coordinate discriminative semantic representations with generation-compatible latent representations. Existing approaches often rely on either strongly shared representations, which may cause task interference, or separated branches, which may weaken semantic transfer. This paper proposes CSB-UG, a unified understanding–generation framework based on a cross-modal semantic bridge. CSB-UG projects visual features, textual semantics, and generative latents into an intermediate bridge space, while task-adaptive adapters preserve task-specific transformations. It further introduces controlled bidirectional collaboration and a patch-token consistency objective to enhance semantic guidance, latent feedback, and local alignment. Experiments on MS-COCO Captions, VQAv2, Visual Genome, and GenEval evaluate retrieval, visual question answering, captioning, text-to-image generation, generation alignment, and fine-grained semantic consistency. Results show that CSB-UG achieves competitive performance across the evaluated understanding and generation tasks, with improvements in retrieval, captioning, generation alignment, and local semantic consistency. Qualitative analyses further suggest improved constraint preservation and object-, attribute-, and relation-level alignment, while failure cases show that complex spatial relations, counting, long-prompt following, and rare attributes remain challenging. These results indicate that bridge-based semantic coordination is a useful design choice for unified image understanding and generation under the evaluated settings.
Source: Discover Artificial Intelligence
Author: Xiaoping Gu
Institution: Zhixing College of Hubei University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: gold
Estimated APC paid: $1,590
OpenAlex ID: W7235404721
