TinyNav: $20 ESP32 Runs End-to-End Autonomous Driving with a 23k-Parameter CNN

Forum topic · ✨步子哥 · 2026-03-15

Summary

TinyNav is a TinyML project that demonstrates end-to-end autonomous driving on a ~$20 ESP32-P4 microcontroller. The system pairs a Waveshare ESP32-P4-WIFI6-M (dual-core 360 MHz, 32 MB PSRAM) with a Sipeed MaixSense A010 Time-of-Flight depth camera. Raw 100x100 depth maps are binned to 24x24, and 20 consecutive frames are stacked as channels into a 24x24x20 spatiotemporal block processed by a compact 2D CNN with only 23,000 parameters and two heads: steering (-1 to 1) and throttle (0 to 1). Training data was collected by teleoperating a car on a physical track with movable walls, recording depth-action pairs with horizontal-flip augmentation. Post-training INT8 quantization via TensorFlow Lite Micro retained 99.84% steering and 99.79% throttle accuracy, with 30 ms inference per 20-frame window. One core runs inference while the other handles sensing and motor control. Validation showed ~0.6 correlation with human commands, and Grad-CAM revealed the steering head focuses on wall-top edges while the throttle head watches approaching obstacles. On the training track the car completed 40 collision-free laps. The authors openly discuss limits: ~50k parameter ceiling, indoor-only training data, no odometry, forward-only driving. Code and datasets are open, making TinyNav a replicable engineering template for low-cost robot navigation.

TinyNav: End-to-End Autonomous Driving on a $20 Microcontroller

TinyNav, a project by students at Queen's University, shows that a tiny ESP32 microcontroller can drive a small robot autonomously through a track — no GPU, no SLAM, no cloud. The core insight comes from optical flow: nearby objects produce fast-moving image features while distant ones move slowly, so a sequence of frames carries depth and motion information without explicit 3D reconstruction.

Hardware

MCU: Waveshare ESP32-P4-WIFI6-M — dual-core 360 MHz, 32 MB PSRAM, 768 KB cache (~$20)
Sensor: Sipeed MaixSense A010 Time-of-Flight camera, which emits modulated IR pulses and measures per-pixel return time to output a depth map directly
Raw 100x100 resolution is 4x4-binned on the sensor to 25x25, then resized to 24x24

ToF depth input avoids the 2D-to-3D ambiguity of RGB: instead of a gray wall, the model sees "obstacle at 1.2 m."

Data Collection

Rather than simulation, the team built a physical track with movable walls (varying widths, curves, dead ends), teleoperated the car lap after lap, and recorded depth frames paired with human steering/throttle commands. Horizontal flipping augmented the data; a 60/40 train/test split produced the final image-action dataset.

Model: 23k Parameters of Spatiotemporal Magic

LSTMs and 3D convolutions aren't feasible on ESP32, so TinyNav stacks 20 consecutive depth frames as channels into a 24x24x20 tensor and processes it with a plain 2D CNN, which implicitly captures motion. The network uses a few conv/pooling layers and two parallel fully-connected heads:

Steering head: output in [-1, 1]
Throttle head: output in [0, 1]

Total: 23,000 parameters — end-to-end learning straight from sensor to actuators.

INT8 Quantization

Post-training quantization with TensorFlow Lite Micro converts weights to 8-bit integers:

Steering accuracy retained: 99.84%
Throttle accuracy retained: 99.79%
Inference on a 20-frame window: ~30 ms — real-time capable

Quantization correlation

Dual-Core Pipeline

One ESP32 core runs CNN inference; the other handles sensor capture and motor commands, communicating through shared memory. A slightly slow inference never blocks the control loop.

Validation

Predicted steering/throttle correlate with ground truth at ~0.6 — respectable for 23k parameters
Distribution plots cover the full decision range (no lazy zero-steering outputs)
Grad-CAM shows the steering head attends to upper wall-top edges; the throttle head focuses on approaching walls and slows in dead ends
40 consecutive collision-free laps on the training track; new layouts completed with only occasional light scrapes

Prediction distributions

Grad-CAM examples

Honest Limitations

~50k parameters is roughly the ESP32 ceiling — complex scenes are out of reach
Training data comes only from low-wall indoor tracks; chairs, people, stairs, and outdoor lighting are unseen
No wheel encoders or odometry — pure vision accumulates drift
Forward driving only; reverse dynamics are untrained

Future work could add cheap encoders, multi-sensor fusion, and sim-to-real transfer. The code and dataset are fully open, making TinyNav an excellent replicable starting point for cost-sensitive robotics education, inspection, and smart-home applications.

References

1. R. David et al., "Tensorflow lite micro: Embedded machine learning on tinyml systems," arXiv preprint arXiv:2010.08678, 2021. 2. Espressif Systems, "Esp-nn: Optimised neural network functions for espressif chipsets," GitHub repository, 2026. 3. A. G. Howard et al., "Mobilenets: Efficient convolutional neural networks for mobile vision applications," arXiv preprint arXiv:1704.04861, 2017. 4. M. Sandler et al., "Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation," arXiv preprint arXiv:1801.04381, 2018. 5. R. R. Selvaraju et al., "Grad-cam: Visual explanations from deep networks via gradient-based localization," arXiv preprint arXiv:1610.02391, 2016.