FreeToken is a new open-source serving system for Mixture-of-Experts (MoE) models, released by researchers at UC Berkeley and UT Austin — including Matei Zaharia and Ion Stoica, both established names in serving-systems research (Zaharia co-created Apache Spark; Stoica co-created Ray/Anyscale, and both are affiliated with the lab behind vLLM and SGLang, which FreeToken builds on). The paper's framing is worth stating plainly because it's correct as far as it goes: open-weight models are catching up to proprietary ones in capability, but releasing weights only determines who can obtain a model, not who can afford to run it at scale — frontier open models still assume datacenter-class GPU clusters. FreeToken's actual claim is that a personal machine, treated as a unified pool of GPU, CPU, and memory rather than "a small GPU," can serve far more than that framing suggests.
The problem MoE creates, not just the one it solves
Mixture-of-Experts architectures are usually described as the trick that makes big models cheap to run: each layer holds hundreds of experts, but a token only activates a handful. That's true for computation. It doesn't help memory — the full expert pool for a 284B or 753B parameter model still has to live somewhere, and most of it won't fit on a consumer GPU's VRAM. The paper identifies why existing edge-serving tools (llama.cpp, KTransformers, Ollama) fall short of what the hardware could theoretically do: prefill has to touch nearly every expert regardless of routing, so it streams almost the entire pool over PCIe every time; decode routing shifts token by token, so any placement fixed at load time misses most of the traffic; and consumer hardware varies enormously in how its GPU, PCIe, and memory bandwidth actually balance against each other, so no single fixed strategy works across machines.
The mechanism: measure the machine, then do the math
FreeToken's central idea is a live, closed-form policy rather than a fixed rule. During decode, when an expert isn't already cached on the GPU, the system has two ways to serve it: pull it over PCIe into the cache, or execute it in place on the CPU where it already lives. Both options draw on the same host-memory bandwidth, so the paper derives the optimal split directly from two bandwidth numbers it measures on the actual deployed machine — PCIe transfer bandwidth and CPU-side expert-execution bandwidth — with the split ratio, called q*, following directly from balancing the two branches' completion times against each other. It's a genuinely elegant piece of engineering: rather than tuning a heuristic per device, the system profiles the hardware once and lets a formula decide, live, how much of each cache miss goes where.
Two supporting mechanisms round this out. A shared LRU cache exploits the fact that consecutive decode steps tend to route to overlapping experts, so scarce GPU memory tracks the model's actual current working set instead of a placement frozen at load time. And for prefill, the system anchors checkpoints of recurrent/attention state at the same semantic boundaries — thinking blocks, tool calls, tool outputs — that agent harnesses like OpenClaw, OpenCode, and SWE-agent already use as their own edit points when they truncate context. That's a well-chosen detail: instead of guessing where a context edit might land, FreeToken checkpoints exactly where real agent frameworks are already documented to cut, so an edited context can resume from the nearest surviving anchor and re-prefill only the genuinely new suffix.
The headline numbers, and what they're being measured against
Counting the paper's own reported results directly: on an RTX 5090, FreeToken sustains 77–83 tok/s on Qwen3.6-35B-A3B and 22–25 tok/s on DeepSeek-V4-Flash (284B parameters, 13B active), a reported 1.5–2.3x higher decode throughput than the strongest of four baselines (llama.cpp, Ollama, KTransformers, MoE-Infinity) across four real agentic workloads. Two results stand out beyond the raw multiplier. First, stability under agentic use: FreeToken's decode rate reportedly stays within 12% of its single-turn rate across three multi-turn workloads, while the most context-sensitive baseline in the comparison, KTransformers on DeepSeek-V4-Flash, had already lost 31% of its single-turn rate by the second workload — a reminder that a benchmark run on a single isolated prompt can meaningfully overstate how a system performs once real agent context accumulates. Second, tail latency: the paper reports FreeToken's worst-case time-to-first-token stays under 44 seconds across every workload tested, while every baseline crosses 150 seconds somewhere (llama.cpp at 232s, Ollama at 179s, KTransformers at 946s) — past the timeout thresholds real agent clients actually enforce (the paper cites OpenClaw's 120-second idle watchdog and Claude Code's roughly ten-minute default), which makes tail latency an availability question for real agent use, not just a speed statistic.
Across five consumer systems the paper reports a 1.3–2.1x decode throughput improvement, with two results worth naming directly. On an 8GB RTX 4060 laptop, FreeToken reportedly serves a 35B model at 39.3 tok/s — faster than the 33 tok/s median decode speed the paper attributes to Codex in production, citing a separate measurement paper rather than a same-setup comparison, worth flagging as a cross-paper reference rather than an apples-to-apples one. On a single RTX PRO 6000 workstation GPU, FreeToken reportedly serves the 753B-parameter GLM-5.2 at 14.9 tok/s against llama.cpp's 7.3 (2.0x) with comparable time-to-first-token — while KTransformers, per the paper, has no working path to serve that model on the same box at all, since its CPU kernels don't support GLM-5.2's NVFP4 weight format.
The methodology behind these numbers is unusually disclosed
Worth crediting directly: three of the paper's six test systems are rented dual-socket servers whose real CPUs would far exceed any actual edge machine, so the authors deliberately cap them to 6 CPU threads and pin them to the GPU's NUMA node to emulate consumer-class host bandwidth — and they show this emulation lands in the same bandwidth range their two genuinely consumer machines (a real desktop and a real laptop) reach at full thread count, rather than asserting the emulation is valid without checking it. Baselines are held to bit-identical weight formats rather than compared across different quantizations. And the agentic workloads are real harnesses — OpenCode and Claude Code driving an actual SWE-bench repository issue, OpenClaw running a real email/calendar agent kit — rather than synthetic single-turn prompts. That's a genuinely more rigorous evaluation setup than a lot of serving-systems papers use, and it's worth naming as such independent of whether the final numbers hold up under outside scrutiny.
What isn't independently checked yet
This is an arXiv preprint, submitted August 17, 2026 — not yet peer-reviewed, and every number in the paper is the authors' own measurement, not an independently reproduced one. That the authors are established systems researchers with real track records in this exact space is a reason to take the methodology seriously, not a substitute for someone outside the author list rerunning the comparison. The code is open (Apache 2.0, on GitHub) and a downloadable build exists at flashml.ai, which is precisely what makes independent reproduction possible here — unlike a hosted API benchmark, anyone with the listed hardware could actually rerun this.
What to expect next
- Watch for independent benchmarking. Because FreeToken is installable rather than API-gated, third-party reruns on the same or comparable hardware are actually feasible here — a real test this kind of systems paper doesn't always get.
- Watch for peer review. The engineering is detailed and the evaluation methodology is unusually transparent about its own emulation choices, but neither substitutes for the paper going through independent review.
- Watch adoption on the two extremes the paper highlights. Whether an 8GB laptop running a 35B model at usable speed, and a single workstation GPU running a 753B model at all, actually change what people run locally — versus staying a benchmark result — is the real-world test of the "open weights into open access" claim.
References: arXiv — FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution (2608.16157) · GitHub — FlashML-org/FreeToken · related coverage: Open Weights You Can't Actually Run · Frontier Arcade: trends & predictions