Chinchilla, Explained — and the Meta FAIR Paper That Just Found Its Blind Spot

In 2022, DeepMind trained a 70-billion-parameter model on 1.4 trillion tokens, compared it to their own 280-billion-parameter Gopher trained on 300 billion tokens using the same compute budget, and the smaller model won — comfortably. The smaller model was called Chinchilla, and the paper explaining why it won became one of the most consequential documents in modern AI: the reason every serious pretraining run since has thought hard about how many tokens per parameter, not just how many parameters.

A Meta FAIR paper published today goes back to that formula, tests an assumption built into its math, finds the assumption doesn't hold, and shows the gap matters more the bigger you scale.

What Chinchilla actually is

Before Chinchilla, the dominant scaling recipe came from Kaplan et al. (2020) at OpenAI, and the practical reading of it was: when you get more compute, spend most of it on a bigger model. That's the lineage that produced GPT-3 — 175 billion parameters, trained on roughly 300 billion tokens, a ratio of under 2 tokens per parameter.

Chinchilla — formally Hoffmann et al., "Training Compute-Optimal Large Language Models" — re-ran that question properly: for a fixed training compute budget $C$, what split between model size $N$ (parameters) and dataset size $D$ (tokens) actually minimizes loss? Their answer, after training hundreds of models at varying sizes and token budgets: model size and data should scale at roughly the same rate. Not "bigger model, whatever data you happen to have" — both grow together. The practical rule of thumb that fell out of it is now industry folklore: roughly 20 tokens per parameter for compute-optimal training. Chinchilla itself was trained at closer to that ratio, and it beat Gopher, GPT-3, and Megatron-Turing NLG — all larger, all trained on comparatively little data — at the same or lower training cost.

The formula, and the assumption hiding in it

Chinchilla models the loss as a sum of three terms:

$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$

$E$ is the irreducible loss — the entropy of language itself, the floor no model reaches no matter how large or how much data it sees. $A/N^{\alpha}$ is the error from having a finite model — too few parameters to represent everything the data could teach it. $B/D^{\beta}$ is the error from having finite data — even a perfect model can't learn what it hasn't seen enough of. Fit $A$, $B$, $\alpha$, $\beta$ and $E$ to a grid of training runs, and you can predict the loss of a run you haven't done yet — including, critically, runs far larger than anything in your grid. That's the entire economic value of a scaling law: it lets a lab spend a small fraction of a frontier training budget on small experiments and extrapolate the expensive decision.

Notice the structure, though: it's a sum. $N$ and $D$ each get their own independent term, added together. That's a modeling choice, not a physical law, and it has a specific mathematical consequence — take the mixed second derivative and it's identically zero:

$\frac{\partial^2 L}{\partial N \, \partial D} \equiv 0$

In plain terms: Chinchilla's formula assumes that how much a bigger model helps doesn't depend on how much data you have, and how much more data helps doesn't depend on how big your model is. The two axes are assumed to not talk to each other at all.

How it's used — and its known asterisk

Given a compute budget $C = 6ND$ (the standard FLOPs approximation for a transformer forward-backward pass), minimizing Chinchilla's loss subject to that constraint has a closed-form answer: $N^*$ and $D^*$ each scale as a power of $C$, and — because Chinchilla's fitted $\alpha$ and $\beta$ came out close to equal — that power is close to $0.5$ for both. Double your compute, and compute-optimal training roughly doubles both your model and your token count.

Two things are worth knowing before treating this as gospel. First, an independent replication (Besiroglu et al., 2024) found that the confidence intervals reported around Chinchilla's original fitted numbers were implausibly tight given the underlying data — likely an error in the original paper's Table A.3 — though the replication's own re-fit broadly reconfirmed the headline ~20-tokens-per-parameter finding. Second, and more consequential in practice: Chinchilla optimizes for training-compute efficiency, not for the total cost of a model you're going to serve to millions of users for years. A smaller model trained on more tokens than Chinchilla-optimal costs more to train and less to run forever afterward — which is exactly why LLaMA and most production models since have been deliberately trained well past the Chinchilla-optimal ratio. The formula answers "what minimizes loss for this many FLOPs," not "what minimizes total cost of ownership," and those are different questions with different optimal ratios.

Skaling: testing the independence assumption directly

That brings us to today's paper. Rather than starting from a functional form and fitting it, Videau, Youbi-Idrissi, Lopez-Paz and Ahuja (FAIR at Meta) start from the data and ask the question the additive formula answers by assumption: is the mixed derivative actually zero?

They estimate it directly — nonparametrically, without committing to any scaling-law formula first — on two grids of real training runs: Farseer, a public dataset of 404 model/token-budget configurations, and their own SK-Grid, 134 configurations trained specifically for this paper. Using two independent estimators (a local moving-least-squares fit and a Gaussian process), both agree: $\partial^2 L / \partial N \partial D$ is not zero anywhere on the grid. It's consistently negative, and it follows its own power-law decay. Scaling model size and data together lowers the loss by more than the sum of scaling each one alone — a real synergy between the two axes that an additive formula cannot represent no matter how well its individual terms are fit.

The fix: one extra parameter

Their proposed Skaling law keeps Chinchilla's interpretable inner structure — the same $A$, $B$, $\alpha$, $\beta$ — but changes how the two terms combine:

$L(N, D) = \left( \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \right)^{k} + E$

One new parameter, $k$, wrapping the whole bracket. Set $k = 1$ and this is exactly Chinchilla — the two laws are nested, not competitors from scratch. For any $k \neq 1$, the outer exponent reintroduces exactly the kind of coupling between $N$ and $D$ that the earlier Kaplan formula had (and that Chinchilla's redesign had discarded), without re-tangling Chinchilla's cleanly separable inner terms. It's a minimal patch, deliberately: one parameter, addressing one diagnosed structural flaw.

What changes when you fix it

The gains concentrate exactly where you'd predict if the diagnosis is right: at the boundaries of the training grid, where $N$ and $D$ are most imbalanced — a huge model starved of data, or a small model drowning in it. That's precisely where forcing independence should hurt most, and it does. On the Farseer grid, single-axis extrapolation error falls from 1.48% to 0.47% (larger model sizes) and from 1.98% to 0.88% (larger token budgets). On the authors' own grid, far-extrapolation error drops from 5.17% to 0.70%. Skaling also beats a separately-published, far more heavily parameterized 9-parameter scaling law at these same boundaries — evidence the gain comes from matching the right structure, not just adding more free parameters to fit with.

The result worth sitting with longest is the compute-extrapolation test: predicting the most expensive runs in the dataset using only cheap, low-compute runs to fit the law — the actual real-world use case for a scaling law. Split into undertrained, near-optimal, and overtrained regimes by token-to-parameter ratio, Chinchilla is weakest exactly in the optimal band — 3.47% MAPE, its worst regime — while Skaling holds under 1% in every regime, a 3.9× reduction overall. The additive law is least reliable precisely where a lab actually plans to train.

And the practical number: extrapolated one order of magnitude past the current data, to $2\times10^{25}$ FLOPs — genuine frontier-model territory — Chinchilla's closed-form optimum predicts the token-to-parameter ratio staying roughly flat, drifting up toward ~380 tokens per parameter. Skaling, and the raw nonparametric gradient estimate independently, predict the ratio should instead fall to 20–40. More than a 10× disagreement, at exactly the scale where frontier labs like DeepSeek lock in a fixed ratio and build a training run around it.

The honest caveats

The paper is careful not to oversell this. The coupling strength isn't universal: on Farseer's code-domain counterpart and on the original Chinchilla measurements themselves, the fitted $k$ comes out much closer to 1 ($k \approx 0.77$–$0.90$) than on the paper's primary grids ($k \approx 0.31$–$0.45$), and Skaling's advantage shrinks correspondingly — when the true surface really is close to additive, there's nothing for the extra parameter to buy you. Nor is the direction of the correction universal: on Farseer the fitted exponents imply the optimal ratio should fall as compute grows, but on the authors' own SK-Grid the exponents point the other way, toward a rising ratio. Their conclusion is appropriately narrow — coupling changes large-scale allocation, and which direction it changes it in depends on the fitted data and architecture, not on a universal constant.

What to expect next

• Watch for a cheap-profiling recipe to spread. The paper's "L-shape" sampling strategy — sweep data volume for small models, sweep model size at a small fixed token budget, skip the expensive interior — recovers full-grid accuracy at roughly 10× less profiling compute, because the boundary-anchored functional form can infer the interaction from the edges alone. That's a genuinely practical result independent of whether Skaling itself gets adopted.
• Watch whether the "20 tokens per parameter" folklore gets a scale-dependent asterisk. If Skaling's prediction holds up at real frontier compute, the right ratio isn't a constant — it's a number that moves with scale, and the direction it moves depends on your architecture and data mix.
• Watch for independent replication at production scale. Everything here is fit and cross-validated on grids up to $10^{22}$ FLOPs and extrapolated an order of magnitude beyond; whether the 10×-larger optimal-ratio gap survives an actual frontier training run is the test that will actually settle it.

References: Videau, Youbi-Idrissi, Lopez-Paz & Ahuja — Skaling: Chinchilla's Exponents Meet Kaplan's Coupling (arXiv:2608.07222) · Hoffmann et al. — Training Compute-Optimal Large Language Models (arXiv:2203.15556) · Kaplan et al. — Scaling Laws for Neural Language Models (arXiv:2001.08361) · Besiroglu et al. — Chinchilla Scaling: A Replication Attempt (arXiv:2404.10102) · related coverage: Understanding AI Model Quantization · What Mamba Forgets