MATdiff: Mask-aware transformer with diffusion model for large-mask image inpainting
Year: 2026
Type: article
Abstract: Diffusion models are widely used in image inpainting, but their inherent randomness may cause semantic inconsistencies between inpainted regions and known parts of the image. To address this, we propose Mask-Aware Transformer with Diffusion Model (MATdiff), a latent diffusion model that first employs a transformer-based MAT as a conditional encoder and subsequently embeds the semantic features extracted from MAT into the reverse diffusion process. MATdiff exploits the long-range dependencies modeled by transformers while mitigating the randomness of diffusion models, resulting in inpainted regions with semantic coherence and visual fidelity to the visible content in large-mask image inpainting. During the training process of MATdiff, we modify the input layer of the pre-trained unconditional latent diffusion model to accommodate MATdiff, then freeze the conditional encoder. Finally, we conduct end-to-end optimization on the entire MATdiff. Our training method achieves remarkable results in generating plausible overall structures for image inpainting. Extensive experiments on the CelebA-HQ face dataset validate the state-of-the-art image inpainting performance of MATdiff. For large-mask image inpainting, our model notably improves on MAT, with gains of 5.94% (FID), 3.13% (LPIPS), 18.75% (P-IDS), and 5.14% (U-IDS).
Source: Neurocomputing
Authors: Suxia Wang, Wenjing Zhao, Jianbing Liang, Weimin Zhang, Shuang Zhao
Institution: National University of Defense Technology
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $2,670
Awards: G1732787128, G2648029711, G439547191
OpenAlex ID: W7128712454