What happens when you feed 17th-century Italian to GPT-4?
Your intuition might say: old language, the model surely can't understand it. But this paper (arXiv:2606.27275) reveals a counterintuitive split — the model's perplexity on 17th-century Italian spikes 2.4x (3.2x for academic texts), yet embedding similarity stays above 0.85.
In other words: the model "can't predict the next word," but "knows what the passage is about."
This matters because the field has long used perplexity as a proxy for model understanding — and that proxy may be fundamentally wrong.
A Four-Dimensional Diagnostic Framework
Maria Levchenko (University of Bologna) proposes decomposing "historical language difficulty" into four independent dimensions:
1. Tokenization Cost — how many tokens does the historical text get split into? Finer splits mean lower encoding efficiency 2. Surprisal/Perplexity — how surprised is the model by the next word? 3. Semantic Robustness — embedding similarity: is the model's semantic representation stable? 4. Context Sensitivity — how much does perplexity drop with a small contextual hint?
Prior work treated "historical difficulty" as a single monolithic obstacle, conflating orthographic variation, linguistic distance, and pretraining exposure. The core contribution: these four dimensions vary independently and cannot proxy for one another.
Three Datasets, Three Centuries
- 17th-century Italian (1610–1689): newly digitized corpus from page images, spanning religious treatises to academic prose
- **19th-century Italian (*The Betrothed*): high-exposure control — a classic widely digitized and likely abundant in training data
- 18th-century Russian (civil print books): orthographic stress test — Russian orthography reformed in the 18th century, differing from modern Russian despite linguistic closeness
- 17th-century Italian: on average 2.4x more surprising (academic prose 3.2x)
- 18th-century Russian: only a mild increase
- Title: How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation
- Author: Maria Levchenko
- Affiliation: University of Bologna
- arXiv: https://arxiv.org/abs/2606.27275
- HTML: https://arxiv.org/html/2606.27275v1
- Venue: The 22nd Conference on Information and Research Science Connecting to Digital and Library Science
Core Findings: Encoding Tax ≠ Comprehension Tax
Finding 1: The tokenization tax is mechanical, not related to understanding
17th-century Italian and 18th-century Russian both suffer ~25–30% tokenization inflation. But their prediction difficulty diverges wildly:
Tokenization cost does not correlate with perplexity. BPE tokenizers simply split unseen character combinations more finely — a mechanical process that says nothing about whether the model "understands" the text.
Finding 2: High perplexity ≠ collapsed semantic representation
The most counterintuitive result: perplexity spikes on 17th-century Italian, yet embedding similarity stays above 0.85 on all datasets.
The model can represent the semantics of historical text even when it cannot predict** it. Generation (predicting the next token) and representation (understanding what a passage means) are distinct capabilities using different computational paths.
Analogy: reading classical Chinese, you may not guess the next character (weak generation) but fully grasp the meaning (normal representation).
Finding 3: Genre matters more than era
17th-century religious treatises and academic prose are close in time, yet differ dramatically in perplexity: academic prose is 3.2x more surprising, religious texts only 1.8x.
"Historicalness" is not a uniform dimension — genre may affect model comprehension more than age. Religious texts are likely more common in training data (biblical commentaries and theology are heavily digitized), while 17th-century academic prose (alchemy, early science) is far rarer.
Finding 4: 60% of perplexity vanishes with one sentence
A minimal intervention — adding "The following is 17th-century Italian text" to the prompt — reduced historical perplexity by ~60%.
The model is not helpless with historical language; it simply hasn't activated the right contextual mode. A simple temporal context prompt unlocks its historical-language processing.
Finding 5: Canonical texts are misleading benchmarks
*The Betrothed*, a 19th-century Italian classic, performs near-modern Italian on every dimension: low token inflation, mild perplexity, high embedding similarity.
But that's not because it is "historical" — it's because it is widely digitized, edited, disseminated, and likely abundant in training data. Using canonical texts to evaluate historical-language ability severely overestimates model competence.
Why This Matters
Practical implication for digital humanities: digital libraries can safely use LLMs for semantic retrieval of historical texts, but generation tasks (e.g., automatic translation, modernization) need additional adaptation.
The deeper methodological contribution to NLP at large: perplexity is not a proxy for understanding. Mechanistic interpretability researchers have hinted at this, but this paper uses historical languages as a natural experiment to provide the clearest empirical evidence.
A model can be simultaneously "perplexed" and "understanding" — like a person who "speaks poorly" but "comprehends" a foreign language. Generation and representation are two separate paths; perplexity only measures the former.
Next time a model shows high perplexity on some domain and you're tempted to conclude "the model doesn't understand it" — remember 17th-century Italian. Perhaps the model just can't say it — but it knows.