Overview
An open-source project fits a language model into an ESP32-S3 board (configuration N16R8: 512 KB internal SRAM, 8 MB PSRAM, 16 MB Flash, price ~$8). The model holds roughly 28.9M parameters and runs entirely on-chip, with no text sent to a server.
Reported performance:
- End-to-end generation: ~9.5 tokens/s
- Pure compute measurement: 9.72 tokens/s
- Dense core → SRAM
- Large lookup table → Flash (each token reads only a few rows)
- Output head → PSRAM (loaded at boot)
- Model file size: ~14.9 MB, 4-bit quantized into a 16 MB partition
- Flash lookup cost: ~0.12 ms per token (≈0.7% of an isolated bandwidth test)
- Bottleneck: PSRAM reads on the output head + scalar compute, not Flash bandwidth
- Answer questions
- Follow instructions
- Write code
- Provide factual knowledge
- The author corrected earlier parameter counts and documented multiple speed numbers in
RESULTS.md. - The current 9.5 tokens/s is the latest end-to-end result; 58 tokens/s is a memory-bandwidth upper bound — they should not be confused.
- SIMD is not yet used; the head is still PSRAM-bandwidth-bound.
- Next steps are likely to focus on fewer bytes read per token, not on stacking more parameters.
- Project repository: https://github.com/slvDev/esp32-ai
- Experimental results: https://github.com/slvDev/esp32-ai/blob/main/RESULTS.md
- TinyStories: https://arxiv.org/abs/2305.07759
What the "28.9M" actually means
28.9M is not a 28.9M-parameter general-purpose LLM. The parameters are split as:
| Component | Parameters | |---|---| | Dense core | ~559K | | Output head | ~3.1M | | Per-Layer Embeddings table | ~25M |
Memory placement:
Design inspiration
The architecture borrows the Per-Layer Embeddings idea from the Gemma family: not every parameter must participate in every step in the same way. On desktop GPUs, putting weights in slow storage is usually bad. On microcontrollers, Flash is *much* larger than SRAM, so reorganizing the memory hierarchy can be the only way to make a model fit at all.
Measurements on real silicon:
Capability boundaries (stated by the author)
The model is trained on TinyStories. It can write short, coherent stories, but it cannot:
It qualifies as *"a language model running on a microcontroller,"* but not as an embodied agent that can directly control a robot: no vision input, no action space, no sensor feedback loop, no tool calling, no real-time guarantees.
Why this matters for embodied AI
The headline takeaway "$8 chip replaces the robot mainboard" is misleading. Two practical implications:
1. Edge devices can host tiny, local language interactions, status explanations, and offline prompts, reducing network dependence. 2. Model designers can start treating the memory hierarchy as part of model architecture, rather than training a model first and then trying to compress it.
For robotics, this approach could later support low-risk use cases such as voice commands, device self-checks, fault descriptions, and local policy indexing. That still requires new data, interfaces, and safety evaluations.
Engineering notes worth flagging
Takeaway
The project proves that parameter storage scale can be decoupled from high-speed memory capacity on microcontrollers. It does not prove that a small chip has LLM-grade reasoning ability. For embodied systems, that second claim still has to be demonstrated with closed-loop tasks, latency, power, disconnection, and safety tests — not with parameter counts.