multinomial diffusion

Multinomial Diffusion is a discrete diffusion model where the forward process corrupts categorical data using a categorical (multinomial) noise distribution — at each timestep, each token has a probability of being replaced by any other token in the vocabulary according to a multinomial transition matrix.

Multinomial Diffusion Details

• Transition Matrix: q(x_t | x_t-1) = Cat(x_t; Q_t x_t-1) — categorical distribution over vocabulary.
• Uniform Noise: The simplest scheme transitions toward a uniform distribution over all tokens.
• Absorbing: Alternative scheme transitions toward a single [MASK] token — absorbing state diffusion.
• Reverse: p_theta(x_t-1 | x_t) = Cat(x_t-1; pi_theta(x_t, t)) — neural network predicts clean token probabilities.

Why It Matters

• Natural Fit: Multinomial diffusion is mathematically natural for text, categorical features, and one-hot encoded data.
• D3PM: Structured Denoising Diffusion Models (Austin et al., 2021) formalized multinomial and absorbing diffusion.
• Flexibility: Different transition matrices enable different noise schedules — uniform, absorbing, or token-similarity-based.

Multinomial Diffusion is random token scrambling and unscrambling — a discrete diffusion process using categorical transitions for generating text, molecules, and other categorical data.