Year: 2026
Type: conference-paper
Abstract: Although uniform diffusion language models (UDLMs) represent a promising diffusion paradigm, scaling them remains challenging.We identify the core obstacle as an overuniform training objective and condition-target confusion during sampling.To address these, we propose Less Uniform Diffusion (LUDI), a novel UDLM framework.Specifically, we (i) introduce a less uniform loss that directs each reverse transition toward the clean token, and (ii) equip the model with per-token time embeddings that supply token-level corruption hints, enabling confidence-based few-step sampling.Experiments across scales show that LUDI yields cleaner supervision and improves few-step generation.We further continue-train a 7B autoregressive model into LUDI-7B, resulting in a UDLM capable of complex reasoning.It achieves 3 token-per-step speedup over AR decoding and a competitive performance compared with masked diffusion baselines, revealing that the full potential of UDLMs for complex generation remains to be unlocked.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7214928843