A GitHub repository called sutskever-30-implementations (https://github.com/pageman/sutskever-30-implementations) re-implements the 30 papers from Ilya Sutskever's well-known reading list — the list he reportedly told John Carmack contains 90% of what one needs to understand about AI — entirely in pure NumPy, with no PyTorch, TensorFlow, or other deep learning frameworks.
Each implementation is a from-scratch pilgrimage rather than a code port: Jupyter Notebooks guide the reader step by step, synthetic data is generated on the fly, and gradients are computed and verified by hand. A few pip installs are enough to run anything from an RNN generating text to a full Transformer.
What the 30 implementations cover
- Complexity from simple rules — *The First Law of Complex Dynamics*: how entropy and complexity grow in cellular automata, explaining emergent unpredictability.
- RNNs — Karpathy's *The Unreasonable Effectiveness of Recurrent Neural Networks*: character-level generation with backpropagation through time, plus visible vanishing gradients.
- LSTMs — a NumPy recreation of Colah's canonical post, implementing forget, input, and output gates to handle long-range dependencies.
- Regularization — variational dropout, weight penalties, and path normalization for sequence models, with comparative training curves.
- Pruning & MDL — *Keeping Neural Networks Simple* via the Minimum Description Length principle: the best model is the shortest description of the data.
- Pointer Networks — attention as an explicit pointer over inputs, applied to sorting, convex hulls, and TSP.
- AlexNet — an eight-layer NumPy CNN reproducing ReLU, Dropout, and data augmentation that won ImageNet 2012.
- Order Matters — making seq2seq models permutation-invariant for set inputs via a sorting task.
- GPipe — micro-batching and pipelining concepts for training giant models (illustrated, not truly parallelized, in NumPy).
- ResNet & pre-activation ResNet — skip connections that let gradients survive hundreds of layers, and identity-mapping variants that further stabilize training.
- Dilated convolutions — exponentially growing receptive fields without extra parameters, shown on semantic segmentation.
- Graph neural networks — message passing, aggregation, and updates on synthetic graph data.
- Bahdanau attention — additive attention for neural machine translation, contrasted with Luong attention.
- Transformer — *Attention Is All You Need* built step by step: scaled dot-product attention, multi-head self-attention, positional encoding, and the encoder-decoder stack.
- VAEs — ELBO and the reparameterization trick, with latent-space sampling.
- Relational RNNs — the hardest implementation (~1,100 lines of manual gradient computation), with a relational memory module that outperforms a plain LSTM across 20 bAbI reasoning tasks. Visualizations include:
- !bAbI task example
- !LSTM vs Relational RNN comparison
- !difficulty scaling
- Irreversibility and thermodynamics — a philosophical paper on entropy and Landauer's principle, framed through a coffee-cup vortex.
- Neural Turing Machines — differentiable read/write heads and memory tape, with soft addressing mimicking discrete operations.
- Deep Speech 2 / CTC — Connectionist Temporal Classification for end-to-end speech recognition without forced alignment.
- Scaling laws — fitting power-law curves for compute, data, and parameters (Chinchilla-style analysis).
- Kolmogorov complexity — shortest-program definitions of intrinsic complexity; why digits of pi look random but compress trivially.
- CS231n notes — from kNN to a full CNN with hand-derived backpropagation.
- Multi-token prediction — predicting multiple future tokens to improve sample efficiency and enable speculative decoding.
- Dense Passage Retrieval (DPR) — the dual-encoder architecture with in-batch negatives, foundational to modern RAG systems.
- RAG — sequence-level and token-level retrieval-augmented generation, compared against pure generation on long-tail knowledge.
- Lost in the Middle — experiments reproducing the position bias where Transformers degrade on information in the middle of long contexts.
- AIXI / machine superintelligence — a conceptual NumPy treatment of Solomonoff induction and algorithmic probability from Hutter's thesis.
Why it matters
The repository is less a code collection than a manifesto for deep understanding: instead of calling APIs, you rebuild every mechanism — attention, gates, skip connections, CTC loss — with the plainest tool available. Working through the notebooks traces the full arc of modern AI, from RNNs to Transformers, from pixels to reasoning, from compression to general intelligence.
References
1. pageman/sutskever-30-implementations GitHub repository: pure-NumPy implementations of all 30 papers, 2024–2025. 2. Aman.ai — Ilya Sutskever's Top 30 Papers Primer: the original recommendation list and commentary. 3. dzyim/ilya-sutskever-recommended-reading: an earlier collection of the reading list. 4. Relational RNNs and the bAbI benchmarks: Facebook AI Research, 2018. 5. *Attention Is All You Need*: Vaswani et al., Google, 2017.