Context Length Has Been Frozen at 1M for 30 Months

Here is a number from the arcade dataset that surprised me when I went looking for it.

Gemini 1.5 introduced a 1-million-token context window in February 2024. Today, eleven models in the dataset advertise a 1M context: Claude Opus 5, Inkling, Gemini 3.6 Flash, GLM-5.2, DeepSeek v4 and V4-Flash, MiniMax-M3, Laguna S 2.1, and more. Every major lab, three model generations, thirty months.

Not one has ever exceeded it. The maximum context across 242 releases has been exactly 1.0M since early 2024, while parameter counts went from 176B to 2.8T and prices fell two orders of magnitude. Everything else moved. That number sat still. Pokee's own technical report independently confirms the plateau in its related-work section: frontier models "have about 1M context window."

So when Pokee-Isaac 28B launched claiming 10 million tokens, the interesting question wasn't whether it's impressive. It's whether the wall was ever technical.

Why 1M held

A standard decoder-only transformer runs softmax attention at every layer, and every layer keeps a KV cache proportional to sequence length. Double the context, double the cache, at every layer, for every concurrent request. Somewhere past a million tokens the memory required to hold one conversation exceeds what you can profitably dedicate to it, and the economics of serving collapse before the architecture does. 1M is roughly where "we could" and "we can afford to" diverge.

The labs advertising 1M weren't hitting a capability ceiling. They were hitting a serving-cost ceiling.

The RULER table, and how to read its zeros

RULER is the benchmark that matters here, because advertised context length is the most inflated statistic in AI. It holds task difficulty constant while scaling context, isolating how a model's usable context tracks its nominal one. Pokee used NVIDIA's official data-preparation pipeline and scoring suite.

* context-overflow error. ˢ vendor self-reported, not measured by Pokee.

Those zeros need reading carefully, and the report is scrupulous about it rather than hiding behind them. Claude Haiku 4.5 caps at a 200K native context and Qwen 3.5 122B at 262K — they score 0.0 from 256K onward because they cannot accept the input, not because they tried and degraded. Nemotron has a 1M native window but no purchasable endpoint serves beyond 262K.

So "rivals fail past 2M" is true but flattering in its phrasing. The honest version: most of the panel doesn't have a context window that long to begin with, and the two that reach 1M (Luna, Gemini Flash Lite) overflow there. Isaac is the only model in the panel that stays above 93% across the entire sweep — which remains a real result, just a narrower one than "everyone else fails."

The sharper differentiator is MRCR v2, which distributes multiple needles through a long conversation and penalises cross-needle interference. Isaac scores 0.607 / 0.743 / 0.500 at 256K/512K/1M. GPT-5.6 Luna scores 95.0 on RULER at 256K and then collapses to 0.050 on MRCR at 1M. The report draws the conclusion itself: multi-needle disambiguation is a far harder test than single-target retrieval. That's a point against the industry's favourite benchmark, made by a vendor who could have quietly omitted it.

The report is more honest than its own marketing

The marketing calls Isaac the "world's first real 10M-token context frontier-class model." The technical report claims something narrower and much more defensible: that Isaac "matches or exceeds the strongest cost-optimized cloud systems." And it says why, explicitly:

We deliberately exclude frontier flagship models, such as GPT-5.6 Sol, Claude Opus 5, and Gemini 3.1 Pro, as they cost roughly an order of magnitude more per token, remain strictly cloud-hosted, and address a fundamentally different deployment envelope.

The panel is Luna, Gemini 3.5 Flash Lite, Claude Haiku 4.5, Nemotron 3 Super 120B and Qwen 3.5 122B — cheap tiers and self-hostable open models. Isaac is the smallest by a wide margin, under a quarter of the open-weight baselines' parameters.

The methodology holds to the same standard. Vendor self-reported figures are marked with a superscript and "never averaged into our evaluations." Context overflows, exhausted retries and infrastructure failures are all scored as zero rather than dropped. Harness, prompts and sampling parameters are held fixed across every baseline. BFCL is verified programmatically with no LLM judge. The DTAP run carries an explicit noise floor: "per-domain ASR gaps within roughly ±3 points should be treated as noise."

And then there is this, on their own headline agentic win — Isaac 70.94 against Luna's 70.61 on BFCL v4:

The margin over Luna is 0.33 points and should be read as parity rather than as a decisive lead.

A vendor talking down its own benchmark victory is rare enough to note. The evaluation section is more rigorous than most vendor reports and several academic ones — which makes the one real omission, the architecture, stand out more sharply rather than less.

Where Isaac actually wins and loses

Reading the full agentic panel rather than the highlights:

Long context and safety are decisive wins. Function calling is a tie. Shell execution and multi-server tool orchestration go to GPT-5.6 Luna. Even within τ³, Isaac loses banking to Gemini (0.186 vs 0.203) and telecom to Qwen (0.912 vs 0.947).

One small inconsistency: the report states Terminal-Bench is "the one benchmark in this report where a cloud baseline finishes ahead of Isaac," but on MCP-Atlas two cloud baselines finish ahead of it. A slip rather than a distortion — both tables are printed accurately — but the sentence overstates.

The pricing needs the long-context rates, not the headline ones

Isaac's $0.15/$1.00 is easy to compare against GPT-5.6 Luna's headline $0.20/$1.20, and that comparison is wrong. The report is careful about why: Luna is billed at 2× input and 1.5× output above 272K tokens, which puts it at $0.40/$1.80 at the lengths actually evaluated here. Gemini 3.5 Flash Lite is $0.30/$2.50 flat.

Isaac undercuts Luna by $0.25 input and $0.80 output at long context, and three of the six models can't be quoted at all because nothing serves them there. Worth noting the report labels Isaac's rates "provisional and subject to confirmation at launch", so they aren't settled yet.

Efficiency, and a counterintuitive result

On a single B200, at 10M context: TTFT 72.9s, prefill 137,200 tok/s, decode 337 tok/s.

The interesting property is that prefill throughput rises with context — roughly 42,400 tok/s at 1M against 137,200 at 10M on identical hardware. A tenfold increase in context costs about three times the time-to-first-token rather than ten. Decode is essentially flat across the sweep (~335 tok/s at both lengths). That combination is what makes the window usable in practice rather than merely addressable.

Portability is measured rather than projected, which is unusual:

• Intel Arc Pro B70: 1,087–1,500 tok/s prefill (3.6–5× stock llama.cpp on the same hardware), 58.8 tok/s decode (2.3×)
• Intel Panther Lake, 12-core Xe3: 150.7 prefill / 22.84 decode, fully on the SoC, no discrete GPU
• Qualcomm Snapdragon X2 Elite: 124.95 prefill / 23.54 decode, fully on-device
• AMD adaptation in progress

On Pinchbench's 116-task SuperClaw suite, the 28B dense Isaac running locally at TP-2 scores 0.9567 against cloud-hosted GLM-5 (744B MoE) at 0.929 and locally-served Qwen3-Coder-Next (80B MoE) at 0.866.

What's still undisclosed, and what Pokee admits

The architecture. The report says "non-decoder-only" and stops — no attention mechanism, no memory scheme, nothing. Given that the entire 10M result rests on that undisclosed design, it's the one place where a technical report genuinely withholds the technical part. Everything else here is checkable; this isn't.

We do know some weights are fine-tuned from Qwen3.6-27B (Apache 2.0), with the rest trained from scratch. That makes this the second case in three days of a Western product built on Chinese open weights as substrate, after Sakana's Namazu on DeepSeek and Kimi K2.6. Chinese open models are becoming what other companies build on, in Tokyo and San Francisco alike.

The report's stated limitations are unusually direct, and one is important:

• Text only. No image, audio or video, which is why it's excluded from vision-dependent portions of several benchmarks.
• Coding is not optimised, and not evaluated. "Coding was not a training priority for our first ever Pokee-Isaac model, and this report does not evaluate it directly… Isaac's placement there should be read as neither evidence of coding strength nor evidence of its absence." They flag it as the most commercially significant gap they chose not to close.
• Hardware adaptation is partial. AMD pending.

What this is actually for

The abstract frames the whole thing as a sovereignty argument rather than a benchmark one: long-context agentic capability has been "delivered almost exclusively from the cloud," which leaves "regulated, sovereign, and on-device settings, where data is not permitted to leave the boundary, with no path to it at all."

That places Isaac alongside A.X K2, Soofi S and Namazu — but attacking sovereignty from the deployment envelope rather than the training run or the licence. And there's a second-order argument in the conclusion worth flagging for anyone building agent systems: a large share of today's agent machinery — memory hierarchies, context compression, observation pruning, sharding across sub-agents — exists to ration a scarce resource. Make the resource abundant inside the boundary and "part of that machinery becomes optional rather than required: a trajectory can be kept as it is, rather than compressed, summarized, or split."

If the 10M window holds up independently, that's the more consequential claim than any number in the tables.

What to expect next

• Independent RULER and MRCR runs at 10M remain the whole story. Pokee's methodology is good enough that I'd expect it to replicate; that expectation is not the same as replication.
• Expect the 1M plateau to break generally. A wall that held thirty months held for economic reasons, and one vendor demonstrating a serving profile that goes past it changes what "1M" reads as.
• Coding is the gap to watch, and Pokee says so first. They didn't train for it and didn't measure it. Whoever closes that on a locally-deployable long-context model has the whole package.
• Watch whether the architecture is ever disclosed. A v0 report can reasonably hold it back. A v1 that still doesn't is a different signal.