how text generation is evolving
DiffusionGemma
What if a language model generated 256 tokens at once instead of one at a time? That's the core idea — and it changes everything about latency.
The problem: autoregressive models
one token at a time
hardware utilisation during generation
memory bandwidth (the bottleneck)
Autoregressive models produce one token per step. The GPU spends most of its time loading weights from memory, not actually computing.
Result: GPU compute sits idle between each token. Huge hardware wasted — and every extra token adds latency for the user.
The fix: a 256-token canvas
all at once
DiffusionGemma opens a canvas of 256 token slots simultaneously, seeded with random noise — instead of starting from an empty sequence.
It then refines all 256 positions in parallel across multiple denoising passes. Hardware is now compute-bound, not memory-bound.
First attempt: masked diffusion
rigid, locks tokens in
✕
Early text diffusion replaced tokens with a [MASK] placeholder. The model predicts the right word and fills it in.
But once a [MASK] is filled, it locks in. If later context makes that word wrong, it can never be corrected. ✕
Uniform State Diffusion
iterative canvas refinement
pass 1 / 4
Instead of [MASK], noise is random vocabulary tokens. The model scans the whole canvas and updates every position in parallel.
Tokens with low confidence get re-noised — replaced with a fresh random token — so the model can correct mistakes in later passes.
After several passes the canvas converges: noise becomes coherent text, with continuous error correction along the way.
Two modes, one model
dynamic toggling
Gemma 4 26B backbone
single model, fine-tuned to
switch between two modes
↙
↘
Prefill mode
causal attention →
Reads the prompt left-to-right.
Writes to the KV cache.
Runs once at start, then once
per completed canvas.
Denoising mode
↔ bidirectional attention
Refines the 256-token canvas.
Every canvas token can attend
to all other canvas tokens
and the KV cache.
↻ repeat per canvas
Prefill is causal (left-to-right, like normal attention). Denoising is bidirectional — every canvas token sees every other — which is why the model can refine all positions at once.
Long-form text: chaining canvases
block diffusion
A single canvas is only 256 tokens. For longer output, DiffusionGemma chains canvases together.
Each completed canvas is appended to the KV cache via a prefill step, then a fresh 256-slot canvas is opened and denoised — autoregression at the block level, diffusion within each block.
Standard LLMs generate one token at a time — memory-bound, high latency per user.
DiffusionGemma opens a 256-token canvas, fills it with noise, and denoises it iteratively — compute-bound, ultra-low latency.
Errors can always be corrected mid-pass. Canvases chain for long text. One model, two attention modes.
source: ai.google.dev/gemma/docs/diffusiongemma/explained · schematic, not to scale