Year: 2026
Type: conference-paper
Abstract: Masked Diffusion Models (MDMs) have emerged as a compelling paradigm for language modeling, offering the capability for efficient parallel text generation.However, existing parallel sampling methods typically update multiple masked tokens independently and ignore the complex mutual dependencies among the masked tokens.This independent updating mechanism lacks global coordination and might lead to incoherent outputs.To address this limitation, we propose Representationbased Masked Diffusion Model (RMDM), a framework that leverages the text representation to explicitly encode global semantics and help to parallel update tokens more precisely.Specifically, we first encode text into a continuous semantic space using a pretrained encoder and learn an invertible transformation that normalizes the representation distribution to a Gaussian prior, facilitating efficient sampling during generation.Conditioned on this latent semantic representation, we train a masked diffusion model to learn the conditional text distribution, where the representation serves as global semantic guidance to coordinate parallel token updates and faithfully approximate the target distribution.Empirical results demonstrate that RMDM significantly improves generation quality, particularly in aggressive fewstep sampling regimes.
Authors: Yangrong Hu, Ding Huang, Xueyu Zhou, Jian Huang
Institutions: Hong Kong Polytechnic University, ByteDance (China)
Language: English
Topic: Topic Modeling
Subfield: Artificial Intelligence
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
Funder: Hong Kong Polytechnic University
Award: G1258512577
OpenAlex ID: W7213190479