Code Grimoire: Reimplementing Ilya Sutskever's 30 Legendary AI Papers in Pure NumPy

Forum topic · ✨步子哥 · 2026-01-18

Summary

The GitHub repository sutskever-30-implementations reproduces the 30 papers on Ilya Sutskever's famous reading list — the list he reportedly told John Carmack contains 90% of what matters in AI — using nothing but pure NumPy, with no PyTorch or TensorFlow dependencies. Each paper is implemented from scratch in Jupyter Notebooks with hand-written gradients and synthetic data, covering topics from cellular automata and RNNs to LSTMs, Pointer Networks, AlexNet, ResNet and pre-activation variants, dilated convolutions, graph neural networks, the Transformer, Bahdanau attention, VAEs, Relational RNNs (~1100 lines of manual backprop) evaluated on bAbI tasks, Neural Turing Machines, CTC-based speech recognition, scaling laws, MDL/compression theory, AIXI and Kolmogorov complexity, multi-token prediction, Dense Passage Retrieval, RAG, and the 'Lost in the Middle' long-context phenomenon. The project is framed as an educational pilgrimage: rather than calling APIs, learners rebuild the mathematical core of modern deep learning with minimal tools, gaining an intuition for why these architectures actually work.

Introduction

According to the post, Ilya Sutskever once told John Carmack that truly reading his list of 30 papers would give you 90% of AI's essence. The GitHub repository sutskever-30-implementations turns that list into runnable code — every paper implemented from scratch in pure NumPy, no PyTorch or TensorFlow. Each implementation uses synthetic data, hand-derived gradients, and Jupyter Notebooks that guide you step by step.

Key Points — the 30 papers, grouped by theme

Foundations of complexity and sequence modeling

A First Law of Complex Dynamics — how complexity emerges from simple cellular automata rules via entropy/complexity growth.
The Unreasonable Effectiveness of RNNs (Karpathy) — character-level generation, with vanishing gradients visible on the time axis.
Understanding LSTMs (Colah) — forget/input/output gates reimplemented in a few hundred lines of NumPy.
Regularization for RNNs — variational dropout, weight penalties, path normalization, with before/after training curves.

Architecture milestones

Keeping Neural Networks Simple (MDL pruning) — minimum description length as a pruning criterion.
Pointer Networks — attention as a pointer over output indices for sorting, convex hulls, TSP.
AlexNet (2012) — 8 layers built from scratch: ReLU, dropout, data augmentation.
Order Matters (seq2seq on sets) — permutation invariance via insertion-sort tasks.
GPipe — micro-batching and pipeline parallelism (serially simulated in NumPy).
ResNet / Identity Mappings — skip connections vs. pre-activation blocks; gradient flow past 100 layers.
Dilated Convolutions — exponentially growing receptive fields for segmentation.

Attention, graphs, and memory

Attention Is All You Need — scaled dot-product, multi-head self-attention, positional encoding, full encoder–decoder in NumPy.
Bahdanau vs. Luong attention — additive vs. multiplicative alignment.
Neural Message Passing (GNNs) — message/aggregate/update loops on synthetic graphs.
Neural Turing Machine — differentiable read/write heads and content addressing.
Relational RNNs — the hardest implementation (~1,100 lines of manual backprop); relational memory decisively beats a plain LSTM across the 20 bAbI reasoning tasks. Comparison plots included:

bAbI task example

LSTM vs Relational RNN curves

Task difficulty scaling

Generative models, theory, and modern LLM topics

Variational Autoencoders — ELBO and the reparameterization trick.
Coffee Automaton — irreversibility, entropy, and Landauer's principle.
Deep Speech 2 / CTC — end-to-end speech recognition without forced alignment.
Scaling Laws / Chinchilla — fitting power-law trade-offs between compute, data, and parameters.
MDL revisited, Kolmogorov complexity, and AIXI — compression-as-understanding, algorithmic probability, Solomonoff induction (conceptual implementations).
CS231n notes — kNN through CNN backpropagation.
Multi-token prediction — sample efficiency and speculative decoding.
Dense Passage Retrieval + RAG — dual-encoder retrieval and Sequence/Token RAG paradigms, contrasted against pure generation.
Lost in the Middle — reproducing the mid-context performance drop in long-context Transformers.

Conclusion

The author frames the repository as a manifesto for deep understanding: mastery is not memorizing formulas or calling APIs, but being able to rebuild the magic from scratch with the plainest possible tool.