A widely shared Chinese post claimed that independent researcher secemp had uploaded the entire arXiv corpus to Hugging Face: 3,148,796 papers, 16.1 TB, all LaTeX sources and high-resolution PDFs, hosted at $249/month of personal money, ending with the line: "Once the fire of science is scattered into the hard drives of ordinary people, no one can easily extinguish it."
This audit re-read every cited number against primary sources: the 2,889-line dataset card, the full LICENSE, arXiv's official Terms of Use, and original social media quotes. The verdict: the data is real; the copywriting was embellished.
Key points
- 32 hard numbers checked, 29 exactly correct: 3,148,796 papers; 16.08 TB (16,076,056,758,128 bytes); metadata branch 1.6 GB; paper_text ~2.85M papers / ~70 GB; versions 5,030,612 records / 269 MB; source 6.51 TB; pdf 8.65 TB; paper-level PDF coverage 99.47%; earliest paper April 1986. The only invented figure: "over 6 million version records" — the card's maximum is 5,030,612.
- Nine-config architecture: the dataset is split into on-demand configs (
metadata,versions,files,paper_text,latex,source,pdf,ps,sample). The three index configs together are only 4.35 GB — the full corpus skeleton for trend analysis and retrieval. Note the caliber: pdf raw content is 10.37 TB, compressed Parquet 8.65 TB; the 22.57 TB figure seen elsewhere counts raw bytes including indexed-but-unpublished HTML. - Commercial LLM training: filter the 634,391 commercially usable papers first; cut the rest.
- Local RAG: fine for personal use; check licenses paper-by-paper for public services.
- Version-evolution research:
versions(269 MB) is cleanest — arXiv metadata itself is CC0. - Bandwidth saving: use
snapshot_download(allow_patterns="paper_text/*")or DuckDB remote SQL; unfiltered pulls attempt all 16.08 TB.
Three factual errors in the viral post
| Claim | Reality |
|---|---|
| PostScript "fully preserved" | ps covers only 41.87% of papers |
| S3 had "many corrupted files" | The author only said "my hunch was right"; the card reports a real gap of 0.42% |
| "Data scattered across mirror sites" | arXiv's mirror network was shut down 2024-09-15 |
Also corrected: the arxiv-community/arxiv_dataset is 2.25–3.06 GB, not 1.1 TB; unarXive retained 9M figure captions, 2M table captions, and 742M LaTeX formulas; the commercially-usable share is 20.15% (634,391 papers), not 18.2% (which was CC BY 4.0 alone).
Denominator illusions
"99.47% PDF coverage" is correct but is one of five coverage rates from the same repository: 98.87% (all paper-version pairs), 99.47% (paper-level PDF), 99.51% (paper-level any rendering), 99.54% (retrievable versions with PDF), 99.58% (retrievable versions any rendering). Every percentage must be reported with its denominator.
Of the 21,191 "real gaps": 18,517 (87.4%) are snapshot-boundary artifacts (submissions after the capture date); 1,405 date from 1991–1995 (permanent — arXiv only had PostScript then). Excluding boundary noise, the permanent gap is ~2,674 papers (0.053%), and 19,549 of the 21,191 were never even requested. Meanwhile the dataset already holds PDFs for 30,898/32,341 (95.5%) of retrievable 1991–1995 versions — the oldest papers are among its strongest assets.
The author's self-disclosure
The dataset card records that only the first 289 repair requests honored arXiv's declared 15-second Crawl-delay; the remaining 9,796 ran at a 1.20-second interval (0.83 req/s, 12.5x the published rate), with arXiv never signaling throttling. Quote: "the faster rate was our choice and it is recorded here rather than left out." The author also proactively offers a takedown channel and documented replacing 3 corrupted mirrored PDFs (of 54 flagged candidates; 8 were PNG/JPEG disguised as .pdf).
Licensing: bytes ≠ rights
LICENSE breakdown (denominator 3,148,796): 60.42% arXiv non-exclusive license 1.0; 18.17% CC BY 4.0; 14.38% no record (NULL); remainder NC-ND/NC-SA/CC0/CC BY-SA etc. Author-provided aggregates: CC-licensed + public domain = 25.21%; arXiv non-exclusive or unrecorded = 74.79%; directly filterable commercial-use subset = 20.15% (634,391 papers). NULL does not mean public domain.
arXiv's official Terms state it cannot grant third parties redistribution rights: "arXiv is not the copyright holder on any of the e-prints" and "We are thus unable to grant others the right to distribute arXiv articles." The precise characterization is "third-party redistribution rights were not granted" — a civil permission gap, not a criminal prohibition. Risk falls mainly on hosts and commercial users; personal reading and research use are permitted.
Practical guidance
---
The fire has indeed been scattered — but every ember still carries an invisible thread back to its author. This 16.08 TB's real value is not that it is free, but that it is honest: every break, every violation, every boundary is written on the card.
*Verification date: 2026-09-25. Primary sources: dataset README (2,889 lines), full LICENSE, HF API, arXiv official ToU pages, original X posts. All figures valid as of 2026-09-25.*