ContextDiff: Context-aware controllable generation with text-to-image diffusion models
Year: 2026
Type: article
Abstract: Recent advancements in image generation have explored in-context learning by presenting a query image alongside context examples and text prompts. Despite these efforts, the generated images often exhibit diminished quality and fidelity when text prompts are omitted, highlighting a significant limitation in the models’ ability to fully exploit visual context. To address this, we introduce Context-Aware Controllable Generation (ContextDiff), a pioneering task designed to generate semantically precise and contextually rich images within specified categories. Our approach establishes a context space and develops a mapping network that aligns this space with the powerful Contrastive Language-Image Pre-training (CLIP) framework, thereby providing a concrete representation of abstract context. To ensure semantic diversity and contextual fidelity, we propose three novel loss functions. Extensive experiments and user studies demonstrate that our model significantly outperforms existing models, resulting in marked improvements in image quality and fidelity.
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $4,150
OpenAlex ID: W7171267145