TinyNav: 20-Dollar ESP32 Microcontroller Achieves End-to-End Autonomous Driving with a 23k-Parameter CNN

Forum topic · ✨步子哥 · 2026-03-15

Summary

TinyNav is a project by Queen's University students demonstrating end-to-end autonomous navigation on an ESP32-P4 microcontroller costing roughly $20. The system pairs a Waveshare ESP32-P4-WIFI6-M (dual-core 360 MHz, 32 MB PSRAM) with a Sipeed MaixSense A010 time-of-flight depth camera. Instead of RGB images, it uses depth maps motivated by optical flow principles: nearby objects shift faster than distant ones, giving rich 3D information. Twenty consecutive 24x24 depth frames are stacked into a 20-channel input, allowing a plain 2D CNN (only 23,000 parameters) with two fully connected heads to regress steering (-1 to 1) and throttle (0 to 1). Training data was collected by teleoperating a physical car on a configurable indoor track, with horizontal-flip augmentation and a 60/40 train-test split. Post-training INT8 quantization via TensorFlow Lite Micro retained 99.8% of steering/throttle accuracy and enabled 30 ms inference. Dual-core parallelism separates inference from control. On the training track the car completed 40 laps collision-free; Grad-CAM confirmed the model attends to wall edges and open space. The authors acknowledge limits: ~50k parameter ceiling, indoor-only training data, no odometry, and forward-only driving. Code and datasets are open-sourced, making it a reproducible TinyML robotics reference.

Overview

TinyNav is a project from Queen's University students showing that end-to-end autonomous driving is possible on hardware costing about $20. Built around an ESP32-P4 microcontroller, it replaces heavy SLAM + GPU pipelines with a tiny convolutional neural network fed by time-of-flight depth data — an approach grounded in the principle of optical flow.

Why Optical Flow Matters

Human vision perceives the world not as static images but as a dynamic flow field: when you turn your head, nearby objects sweep past quickly while distant ones drift slowly. This relative motion carries depth information for free. A monitor displaying a static image fails this test — every pixel moves at the same speed when you shift perspective — whereas a mirror responds correctly. Optical flow (the motion vector field of pixels across frames) lets a robot infer relative velocity and depth without full 3D reconstruction: roadside trees that "flow" fast are close; distant mountains barely move.

Hardware

MCU: Waveshare ESP32-P4-WIFI6-M — dual-core 360 MHz, 32 MB PSRAM, 768 KB cache (~$20)
Camera: Sipeed MaixSense A010 time-of-flight (ToF) depth camera. ToF emits modulated infrared pulses and measures round-trip time per pixel, outputting distance directly. This avoids the 2D-to-3D ambiguity of RGB: a wall is not "a gray rectangle" but "an obstacle 1.2 m ahead."
Raw 100x100 depth frames are 4x4-binned on-sensor to 25x25, then resized to 24x24 to fit the microcontroller's budget.

Data Collection

Rather than simulation, the team built a physical track with movable wall panels (varying widths, curves, surfaces, dead ends) and teleoperated the car around it, recording depth maps paired with human steering and throttle commands. Data was augmented with horizontal flips and split 60/40 into train/test sets — an image-action dataset of real demonstrations.

Model: Temporal Stacking in a Tiny CNN

Navigation is continuous, but LSTMs and 3D convolutions are impractical on an ESP32. TinyNav's trick: stack 20 consecutive depth frames as 20 channels, forming a 24x24x20 spatiotemporal block that a standard 2D CNN can process — implicitly capturing motion and speed. The network is a few conv + pooling layers followed by two parallel fully connected heads:

steering: output in [-1, 1]
throttle: output in [0, 1]

Total: 23,000 parameters, end-to-end from raw sensor data to control commands, skipping explicit perception/mapping/planning modules.

INT8 Quantization

Post-training quantization via TensorFlow Lite Micro converts float weights to 8-bit integers:

Steering accuracy retained: 99.84%
Throttle accuracy retained: 99.79%
Inference latency (20-frame window + decision): ~30 ms — sufficient for real-time control

Dual-Core Pipeline

One core runs CNN inference; the other handles sensor acquisition and motor commands, communicating through shared memory. A slow inference frame never blocks the control loop, keeping the car smooth.

Evaluation

Correlation: predicted steering/throttle vs. ground truth correlation of ~0.6 — solid for a 23k-parameter model; distributions show full-range decisions (no lazy zero-steering collapse).
Grad-CAM: the steering head attends to upper wall edges (most stable channel/obstacle discriminator); the throttle head looks at open space ahead on straights and fixes on approaching walls in dead ends, slowing appropriately.
Field test: 40 consecutive collision-free laps on the training track; completed an unseen new layout with occasional light scrapes.

Quantization correlation Prediction distributions Grad-CAM examples

Limitations & Future Work

The authors are candid about constraints:

~50k parameter ceiling on the ESP32 caps model complexity
Training data covers only an indoor track with low walls; no chairs, pedestrians, stairs, or outdoor lighting — weak generalization
No wheel encoders/odometry, so pure vision drift accumulates error
Forward driving only; reverse dynamics are untrained

Future directions: low-cost encoders, multi-sensor fusion, and sim-to-real transfer learning.

Verdict

TinyNav is a compelling engineering exemplar for TinyML robotics: not algorithmically novel, but a complete, honest, highly reproducible closed loop (code and dataset open-sourced) that shows intelligence depends more on clever design than on big hardware. It is an excellent starting point for low-cost robotics in education, inspection, and smart home applications.

References

1. R. David et al., "Tensorflow lite micro: Embedded machine learning on tinyml systems," arXiv preprint arXiv:2010.08678, 2021. 2. Espressif Systems, "Esp-nn: Optimised neural network functions for espressif chipsets," GitHub repository, 2026. 3. A. G. Howard et al., "Mobilenets: Efficient convolutional neural networks for mobile vision applications," arXiv preprint arXiv:1704.04861, 2017. 4. M. Sandler et al., "Inverted residuals and linear bottlenecks," arXiv preprint arXiv:1801.04381, 2018. 5. R. R. Selvaraju et al., "Grad-cam: Visual explanations from deep networks via gradient-based localization," arXiv preprint arXiv:1610.02391, 2016.