Diffusion with Truncated Blocks: Fast and High-Quality Text Generation using Truncated Block Generation
Year: 2026
Type: conference-paper
Abstract: Diffusion-based Large Language Models (dLLMs) are emerging as a powerful alternative to traditional autoregressive models.These models learn to generate text by iteratively denoising masked sequences.In this work, we identify a critical problem in dLLMs: the model's attention is wastefully expended on uninformative mask tokens, diluting its focus on meaningful context.We term this phenomenon "attention dilution".We further show that this artifact is amplified by token-level noising, whereas models employing sequence-level noise exhibit a reduced effect.To resolve this problem, we introduce Truncated Block Generation, a novel sampling algorithm that not only mitigates attention dilution but also enables faster inference and flexible-length sequence generation.Extensive experiments validate our analysis and demonstrate the marked effectiveness of our proposed method in enhancing both the performance and efficiency of dLLMs.
Authors: Yuyan Zhou, Weiyu Chen, James Kwok
Institution: Hong Kong University of Science and Technology
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: hybrid
OpenAlex ID: W7166850100