Year: 2024
Type: preprint
Abstract: While diffusion models have significantly advanced the quality of image generation their capability to accurately and coherently render text within these images remains a substantial challenge. Conventional diffusion-based methods for scene text generation are typically limited by their reliance on an intermediate layout output. This dependency often results in a constrained diversity of text styles and fonts an inherent limitation stemming from the deterministic nature of the layout generation phase. To address these challenges this paper introduces SceneTextGen a novel diffusion-based model specifically designed to circumvent the need for a predefined layout stage. By doing so SceneTextGen facilitates a more natural and varied representation of text. The novelty of SceneTextGen lies in its integration of three key components: a character-level encoder for capturing detailed typographic properties coupled with a character-level instance segmentation model and a word-level spotting model to address the issues of unwanted text generation and minor character inaccuracies. We validate the performance of our method by demonstrating improved character recognition rates on generated images across different public visual text datasets in comparison to both standard diffusion based methods and text specific methods.
Source: arXiv (Cornell University)
Authors: Zhangli, Qilong, Jiang, Jindong, Liu, Di, Yu, Licheng, Dai, Xiaoliang, Ramchandani, Ankit, Pang, Guan, Metaxas, Dimitris N., Krishnan, Praveen
Language: English
Subfield: Computer Vision and Pattern Recognition
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
Funder: National Science Foundation
Awards: Collaborative Research: HCC: Medium: Linguistically-Driven Sign Recognition from Continuous Signing for American Sign Language (ASL), NSF Convergence Accelerator Track H: AI-based Tools to Enhance Access and Opportunities for the Deaf, Center: IUCRC Phase II Rutgers University: Center for Accelerated and Real Time Analytics (CARTA), CDS&E: Collaborative Research: Private Data Analytics Synthesis, and Sharing for Large-Scale Multi-Modal Smart City Mobility Research
OpenAlex ID: W4399400482