Key points
- Project overview: Talkie-1930 is a 13B-parameter language model trained on ~260B tokens of pre-1931 public-domain text (books, newspapers, journals, patents, case law). The cutoff is legally chosen because works from 1929 and earlier are firmly in the U.S. public domain.
- Team: Led by Alec Radford (first author of GPT-1/2, CLIP, Whisper), with David Duvenaud (Neural ODE co-author) handling architecture/training and Nick Levine on data engineering. Radford holds only an undergraduate degree.
- Control design: Two companion models —
talkie-1930-13b-it(instruction-tuned with pre-1931 etiquette manuals, letter-writing guides, encyclopedias, poetry) andtalkie-web-13b-base(same architecture and compute, trained on FineWeb). The web twin enables clean A/B comparison. - Data engineering findings:
- OCR is the dominant bottleneck. Conventional OCR systems deliver only 30% of the learning efficiency of human transcription; lightly cleaned OCR reaches ~70%.
- The team plans a custom OCR system tuned for pre-1930 fonts, long-s (ſ), archaic spellings, and degraded paper.
- Post-training pipeline:
- Stage 1: vintage SFT using etiquette manuals and letter guides.
- Stage 2: online DPO with Claude Sonnet 4.6 as judge; reward scores rose 2.0 → 3.4 / 5.
- Stage 3: rejection-sampled SFT from Claude Opus 4.6 synthetic dialogues.
- Side effect: a 7B variant started adopting modern list-style formatting — the authors call this "time contamination."
- Core finding #1 — emergent Python coding: On HumanEval (164 standard Python problems), Talkie never saw modern code yet produced a working inverse of a Caesar-shift encoder by changing
+5to-5. Capabilities: simple one-liners and minor example modifications work; complex multi-function programs do not. Limited Chinese-community reports cite 4.5% on SWE-bench after 250-sample fine-tuning, though the original paper is not yet released and this figure is unverified. - Core finding #2 — surprisingness curve: Measuring bits-per-byte across ~5,000 NYT "On This Day" events yields three phases — composure pre-1930, shock through the 1930s–40s (New Deal, WWII), and vertigo peaking in the 1950s–60s before plateauing. The peak aligns with postwar reconstruction, the Cold War, the space race, and civil-rights upheaval — moments that shatter a 1930 worldview.
- Data leakage: Despite the cutoff, Talkie reproduces FDR's New Deal specifics, WWII details, UN founding facts, and Churchill-era references. Likely sources: mislabeled reprints, citations in pre-1931 texts pointing forward, modern editorial annotations introduced during digitization, and Claude-judge contamination via DPO.
- Performance vs. modern twin: On core language understanding and math, Talkie-1930 nearly matches
talkie-web-13b-base. General-knowledge gap shrinks by roughly half when anachronistic questions are excluded, suggesting OCR-driven data-quality loss explains most of the residual deficit. - Philosophical implication: Because Python, the internet, transistors, DNA, nuclear weapons, and WWII postdate the training corpus, any correct answer must be derived, not retrieved — partial evidence against the strict "stochastic parrot" view, while still showing reasoning is local and shallow.
- Connection to Demis Hassabis's AGI test (early 2026): can a model trained on pre-1911 data derive general relativity? Talkie does not answer it directly (it already knows special relativity), but supplies the experimental scaffold: given sufficient quality and scale, the answer may be yes.
- Limitations: OCR loss, residual data leakage, weak factual knowledge, narrow programming ability, English-only, and a ceiling near 260B historical English tokens.
- Roadmap: vintage OCR system, GPT-3-class trillion-token vintage model (summer 2026), bootstrapped post-training without Claude, multilingual vintage corpora, and a ChatGPT-class vintage model as a long-term goal.
- Community reaction: Hacker News thread (355 upvotes, 125 comments) debated leakage and modern scaffolding. Simon Willison stressed that historical corpora are inherently noisy and OCR-bound. Historian Benjamin Breen framed vintage LLMs as a possible new humanities discipline.
- Industry implications: Challenges "bigger is better" data orthodoxy, offers a contamination-free evaluation baseline (no pre-1931 test sets exist), and inaugurates the Vintage Language Model category — useful for benchmarking, controlled reasoning experiments, and digital archaeology.
- https://talkie-lm.com/chat
- https://huggingface.co/talkie-lm/talkie-1930-13b-base
- https://huggingface.co/talkie-lm/talkie-1930-13b-it
- https://huggingface.co/talkie-lm/talkie-web-13b-base
- https://talkie-lm.com/
- https://news.ycombinator.com/item?id=43589001
- https://simonwillison.net/2026/Apr/28/talkie/
- https://www.marktechpost.com/2026/04/27/meet-talkie-1930/
- https://byteiota.com/talkie-vintage-llm-1930s-ai-tests-reasoning-vs-memory/
- https://eu.36kr.com/en/p/3787360224681221
- https://resobscura.substack.com/p/vintage-llms
- https://github.com/entanglr/awesome-vintage-llms
- https://github.com/talkie-lm