Running a language model usually means a GPU with tens or hundreds of gigabytes of memory. This project runs one on an ESP32-S3 — a $8 microcontroller with 512KB of SRAM, 8MB of PSRAM, and 16MB of flash — entirely on-chip, no network connection involved, streaming generated text to a small attached display at roughly 9.5 tokens per second.
Small language models on microcontrollers aren't new, but the previous mark was around 260,000 parameters. This one is 28.9 million — about 100x larger — which raises the obvious problem: none of the ESP32-S3's memory tiers is anywhere near big enough to hold a model that size, let alone run it.
First, what an "embedding" even is
Before the trick makes sense, it helps to see the ordinary thing it replaces. A language model doesn't work with words — it works with numbers. The first thing that happens to any input is a lookup: the model keeps a giant table with one row per possible token (this model's vocabulary has 32,768 of them), and each row is a short list of numbers — the token's "embedding" — that stands in for that token everywhere else in the network. It's a dictionary, essentially: token in, one row out, the other 32,767 rows ignored.
The catch is where that dictionary has to live. In an ordinary transformer, the embedding table is a weight matrix like any other, and neural-network weights are expected to sit in fast memory for the model's entire lifetime, because in principle any of them could matter on any given step. A small model's table might only be a few megabytes, but it scales with vocabulary size, and it adds up fast: Gemma 3n's 256,000-token vocabulary alone works out to over a gigabyte, sitting fully loaded before the model has done a single step of actual reasoning — Google's own developer blog uses exactly this framing to introduce the problem PLE solves.
The fix: give every layer its own small lookup, and let it live somewhere slower
Per-Layer Embeddings (PLE) is the architectural change Google introduced with the Gemma 3n family, previewed at Google I/O and fully released in June 2025. The idea has two parts, and the second is the one people usually skip past.
First: instead of one embedding per token, looked up once at the input and then carried unchanged through every layer, PLE gives each transformer layer its own small, separate embedding table for the same vocabulary. So a token doesn't get one lookup — it gets one per layer, and each layer mixes its own small vector into the computation as the token passes through, rather than relying purely on the original input embedding to carry all the meaning.
Second, and this is the part that actually saves memory: because each per-layer table only ever gets touched by a simple row lookup — never multiplied through, never part of a matrix computation — it doesn't need to sit in the same fast memory as the weights that do get computed on. An independent technical breakdown of Gemma 3n puts a number on it: for Gemma 3n's E2B model, each of the 30 layers gets a 256-number vector per token, for 30 × 256 = 7,680 values — which, quantized down to 4-bit, comes out to just under 4KB per token. Compare that to loading a full ~1GB+ table, and you can see why Google reports that the E2B and E4B models, with 5B and 8B raw parameters respectively, run with a memory footprint closer to a 2B and 4B model — the rest of the parameter count is embeddings that get streamed in a few kilobytes at a time rather than held permanently in the accelerator's memory.
Why this maps so neatly onto a microcontroller
An ESP32-S3 doesn't have a CPU/accelerator split the way a phone does — but it does have exactly the split PLE is designed around: a small pool of fast memory (SRAM) built for computation, and a much larger, much slower pool (flash) that's fine for reads it doesn't need to do quickly. This project applies the same idea at a far smaller scale: 6 layers instead of 30, a 128-number per-layer vector instead of 256, and a 32,768-token vocabulary. That's roughly 25M of the model's 28.9M total parameters living as a lookup table in flash, while only the compute-heavy core stays in fast memory:
| Tier | Capacity | Holds |
|---|---|---|
| SRAM | 512KB | The compute core — active every token |
| PSRAM | 8MB | Output head and working memory |
| Flash | 16MB | The ~25M-parameter, per-layer embedding lookup table |
At inference time, generating one token means reading 6 rows from that flash table — one per layer — for a total of about 450 bytes, rather than anything close to the full 25M-parameter structure. The author's own write-up frames it plainly: applying "Gemma-style Per-Layer Embeddings to a microcontroller SRAM/flash hierarchy," and, as far as they're aware, the first time this specific idea has been adapted to hardware this constrained. It's the whole reason a 100x-larger model fits where a 260K-parameter one used to be the ceiling — not a smarter compression trick, but moving 87% of the parameter count somewhere that never had to be fast in the first place.
What's actually on the chip
After 4-bit quantization the full model is 14.9MB, comfortably inside the ESP32-S3's 16MB of flash alongside the firmware. Generation runs at about 9.5 tokens/second end-to-end (9.7 tok/s of pure compute, per the repo's measurements), with output written directly to the connected display — no cloud round-trip anywhere in the loop.
It's trained on Microsoft's TinyStories dataset, so the outputs are exactly what that implies: short, simple, reasonably coherent children's stories. It doesn't answer questions, follow instructions, write code, or hold facts — those weren't the training objective, and nothing about this architecture was aimed at general capability. The point was proving that a model two orders of magnitude larger than the prior microcontroller record could run, coherently, on hardware this small.
What's in the repo
The GitHub repository ships the full pipeline rather than just a binary: ESP32 firmware and wiring instructions, the training code used to produce the model, the quantization pipeline that gets it down to 14.9MB, and a results write-up covering the ablations and on-chip measurements behind the numbers above. It's MIT-licensed, so all of it — training, quantization, and firmware — is there to adapt for other constrained-hardware experiments.