High-fidelity text refinement for ControlNet-guided latent diffusion in document inpainting
Year: 2026
Type: article
Abstract: Blind document image inpainting (DII) aims to restore degraded document scans without prior knowledge of noise locations, yet existing methods either produce over-smoothed text or introduce pseudo-characters. We propose a novel high-fidelity, text-guided restoration framework based on a ControlNet-guided Latent Diffusion Model (CGLDM). First, we extract and refine noisy OCR outputs using a two-stage Vision Language Model (VLM) and Large Language Model (LLM) pipeline, leveraging both global document context and local text cues to deliver near ground-truth textual fidelity. Next, these refined text features, together with an initial visually restored image, condition a latent diffusion process that progressively denoises and reconstructs clean image latents. To suppress patch-wise background inconsistencies inherent in high-resolution processing, we introduce an explicit feature-alignment loss of diffusion model training that enforces agreement between the predicted features and the VAE-encoded features of the ground-truth image. Finally, we propose using PNG file size as a novel non-reference metric—under preserved textual legibility—to quantify restoration quality in ill-posed settings. Extensive experiments on the FUNSD-ZH dataset demonstrate that our approach outperforms state-of-the-art methods in both perceptual image quality and OCR legibility, while achieving superior compression efficiency, thereby offering a practical, trustworthy solution for real-world document recovery.
Source: Expert Systems with Applications
Institutions: University of Nottingham Ningbo China, Jilin University
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
Estimated APC paid: $3,500
Awards: G374571540, G8667151263
OpenAlex ID: W7166880339