Dystruct: Dynamically Structured Diffusion Language Model Decoding via Bayesian Inference
Abstract
A Bayesian structured decoding framework enables flexible-length generation in diffusion language models by dynamically inferring structural properties during decoding without requiring retraining.
Diffusion language models (DLMs) have recently emerged as a promising alternative to autoregressive models, primarily due to their ability to enable parallel decoding. Despite this advantage, most existing DLMs rely on a fixed generation length specified prior to decoding, which restricts their flexibility in real-world applications. While a few recent works attempt to support flexible-length generation, they typically suffer from notable limitations: some require costly retraining to accommodate variable-length outputs, while others depend solely on local confidence signals during decoding. Such local criteria fail to capture the evolving structure of the sequence, often resulting in suboptimal generation quality. In this paper, we propose a training-free, Bayesian structured decoding framework that formulates flexible-length generation as a dynamic structural inference problem. Our approach formulates flexible-length generation as a dynamic structural inference problem, jointly computing the expansion length, the block boundaries, and the decoding schedule. At each window expansion step, the method integrates local uncertainty with structural signals via a unified mechanism that supports dynamic structured generation, including both flexible block expansion and block organization, while maintaining coherence. Extensive experiments across multiple benchmarks demonstrate that our approach significantly improves generation quality and flexibility over existing fixed-length and flexible-length baselines. These results highlight the advantage of Bayesian structured decoding for diffusion language model, providing a principled and efficient solution for structured text generation.
Community
DyStruct is a training-free Bayesian decoding framework that enables flexible-length generation in discrete Diffusion Language Models (DLMs).
While discrete diffusion models offer the architectural advantage of parallel decoding, they are typically constrained by fixed sequence lengths. Existing methods for variable-length generation rely on strictly left-to-right truncation heuristics—which force premature token commitments—or require costly custom alignment training.
DyStruct formulates sequence expansion as a pure inference-time structural problem, utilizing a Bayesian framework to dynamically determine expansion size, block partitioning, and decoding order. The method executes non-monotonically: a Chinese Restaurant Process (CRP) prior and context-aware Gibbs scheduling actively search for and anchor stable sequence segments first (such as initial setups and final answer formats). These stable anchors are then used to bidirectionally constrain highly unstable intermediate reasoning steps.
By allocating unmasking iterations based strictly on structural instability, the algorithm naturally terminates early on rigid tasks (such as arithmetic templates) to optimize compute, while reserving deep refinement steps for complex logic. Evaluated on LLaDA-8B and Dream-7B, this approach yields strict accuracy improvements across mathematical reasoning and code synthesis, including a +4.4 exact match increase on Big-Bench Hard.
Get this paper in your agent:
hf papers read 2605.09820 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper