Videos
Short animated explainers on how modern AI actually works — built to make one idea click.
Featured
DiffusionGemma, animated
How Google's text-diffusion model generates 256 tokens at once — noise, denoising passes, and two attention modes.
A ~85-second animated explainer of DiffusionGemma, Google's open-weight model that generates text the way image diffusion models generate pixels. Autoregressive LLMs emit one token per step, leaving the GPU memory-bound and mostly idle; DiffusionGemma instead opens a 256-token canvas seeded with random vocabulary noise and refines every position in parallel across several denoising passes — Uniform State Diffusion. Unlike earlier masked diffusion, where a filled-in [MASK] token is locked forever, low-confidence tokens can be re-noised and corrected in later passes. Under the hood one Gemma backbone toggles between two modes: a causal prefill mode that reads the prompt and writes the KV cache, and a bidirectional denoising mode where every canvas token attends to every other. For long-form text, completed canvases are appended to the KV cache and a fresh canvas is opened — autoregression at the block level, diffusion within each block. Press play on the featured player above.
References
Primary sources
More videos
Mamba vs Attention, animated
How two model families remember context — seen through Soofi S (hybrid Mamba) and Gemma 4 (all-attention).
A ~75-second animated explainer of the core architectural choice behind today's long-context models. Attention keeps a full key-value cache that grows with every token — perfect recall, but rising memory and slower decoding. Mamba keeps a fixed-size recurrent state — flat cost at any context length, at the price of softer recall of exact old details. The comparison is grounded in two recent open-weight models: Germany's Soofi S, a hybrid that runs mostly on Mamba-2 layers with only a handful of attention layers, and Google's Gemma 4, which stays all-attention but slows cache growth with local sliding windows. Press play on the featured player above; the layer-by-layer stacks near the end show how each model is actually assembled.
Does Mamba just forget more than attention?
Yes, but with important nuances — it's more subtle than a simple "forgets more."
What Mamba forgets vs what it keeps
The forgetting isn't random or uniform. Remember:
is learned and input-dependent. The model trains itself to set close to 1 for tokens it should preserve, and close to 0 for tokens it should discard. So in practice it learns selective forgetting — not blanket amnesia.
The distinction that matters is what kind of information you need:
| Task type | Example | Attention | Mamba |
|---|---|---|---|
| Reasoning over nearby context | "Given the paragraph above, answer…" | ✅ | ✅ good |
| Long-range pattern / theme | "What's the overall argument of this essay?" | ✅ | ✅ good |
| Exact verbatim recall | "What was the 3rd word on page 1?" | ✅ | ⚠️ risky |
| Counting over full context | "How many times does X appear?" | ✅ | ❌ often fails |
| Code reasoning | "Fix the bug in this function" | ✅ | ✅ good |
The Soofi S evidence
The paper actually confirms this directly. On the RULER benchmark at 1M tokens, the common word extraction (CWE) task — which requires counting word frequencies across the entire document — Soofi S collapses to ~3% accuracy beyond 32K tokens, while Nemotron (same architecture, different long-context data) holds at 60–64%.
But on every other RULER subtask — retrieval, question answering, reasoning — the two models stay within 4–6 points of each other all the way to 1M tokens.
So the failure is very specific. It's not "Mamba forgets everything at long context." It's more precisely:
Mamba struggles when the answer requires integrating exact counts or verbatim fragments spread across the entire context.
Why attention doesn't have this problem
Mathematically, attention always has:
Token is sitting there untouched at row 1, no matter what is. The softmax score can always attend back to it with full precision. There's no decay, no compression, no eigenvalue shrinking it away.
The practical picture
For 99% of real deployment tasks — coding assistants, document Q&A, chat, reasoning, summarization — Mamba's selective compression is good enough, and the speed/memory advantage at long context is enormous.
The gap only shows up in the narrow class of tasks that need exact, lossless recall of something specific buried thousands of tokens back. That's a real limitation, but it's a much smaller slice of real-world use than it might sound.
The Soofi team actually name this explicitly as their main target for the next iteration — adding retrieval- and aggregation-style synthetic data to the long-context training phase to teach the model to be more aggressive about preserving those signals in .
References
Foundational papers