What Mamba Forgets — and What It Keeps

I recently built an animated explainer of Mamba vs Attention. This is the deeper dive it points to — on exactly what Mamba's fixed-size state gives up, and what it doesn't.

Does Mamba just forget more than attention?

Yes, but with important nuances — it's more subtle than a simple "forgets more."

What Mamba forgets vs what it keeps

The forgetting isn't random or uniform. Remember:

$h_t = A_t \, h_{t-1} + B_t \, x_t$

$A_t$ is learned and input-dependent. The model trains itself to set $A_t$ close to 1 for tokens it should preserve, and close to 0 for tokens it should discard. So in practice it learns selective forgetting — not blanket amnesia.

The distinction that matters is what kind of information you need:

Task type	Example	Attention	Mamba
Reasoning over nearby context	"Given the paragraph above, answer…"	✅	✅ good
Long-range pattern / theme	"What's the overall argument of this essay?"	✅	✅ good
Exact verbatim recall	"What was the 3rd word on page 1?"	✅	⚠️ risky
Counting over full context	"How many times does X appear?"	✅	❌ often fails
Code reasoning	"Fix the bug in this function"	✅	✅ good

The Soofi S evidence

The paper actually confirms this directly. On the RULER benchmark at 1M tokens, the common word extraction (CWE) task — which requires counting word frequencies across the entire document — Soofi S collapses to ~3% accuracy beyond 32K tokens, while Nemotron (same architecture, different long-context data) holds at 60–64%.

But on every other RULER subtask — retrieval, question answering, reasoning — the two models stay within 4–6 points of each other all the way to 1M tokens.

So the failure is very specific. It's not "Mamba forgets everything at long context." It's more precisely:

Mamba struggles when the answer requires integrating exact counts or verbatim fragments spread across the entire context.

Why attention doesn't have this problem

Mathematically, attention always has:

$K_t = \begin{bmatrix} k_1 \\ k_2 \\ \vdots \\ k_t \end{bmatrix}$

Token $k_1$ is sitting there untouched at row 1, no matter what $t$ is. The softmax score can always attend back to it with full precision. There's no decay, no compression, no eigenvalue shrinking it away.

The practical picture

For 99% of real deployment tasks — coding assistants, document Q&A, chat, reasoning, summarization — Mamba's selective compression is good enough, and the speed/memory advantage at long context is enormous.

The gap only shows up in the narrow class of tasks that need exact, lossless recall of something specific buried thousands of tokens back. That's a real limitation, but it's a much smaller slice of real-world use than it might sound.

The Soofi team actually name this explicitly as their main target for the next iteration — adding retrieval- and aggregation-style synthetic data to the long-context training phase to teach the model to be more aggressive about preserving those signals in $h_t$ .