Rethinking Diffusion Large Language Models via Time-aware Weighting and Block-oriented Refinements
Year: 2026
Type: conference-paper
Abstract: Diffusion Large Language Models (DLLMs) have recently emerged as a parallel alternative to autoregressive models, offering bidirectional context modeling and parallel decoding.However, developing highly performant DLLMs is hindered by lifecycle challenges, including suboptimal timestep weighting in pre-training, train-inference distribution gaps in post-training, and underutilized caching during inference.To address these obstacles, we introduce ReDLLM, a 30Bparameter Mixture-of-Experts DLLM.It incorporates a time-aware weighting strategy during continual pre-training that scales gradients according to denoising difficulty.Furthermore, we propose two block-oriented refinements: a terminal-block treatment to narrow the train-inference gap, and a block-aware inference pipeline featuring inline block prefill and block-level scheduling to maximize KVcache reuse and parallelization.Extensive evaluations demonstrate that ReDLLM surpasses existing DLLMs at the same active-parameter scale, achieves generation quality competitive with 100B DLLMs, and attains the highest inference throughput among evaluated models.
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
OpenAlex ID: W7224312005