What happens when you feed a 17th-century Italian academic text to GPT-4?
Your intuition might be: it's an old language, so the model surely can't understand it. But this paper (arXiv:2606.27275) found a counterintuitive split—perplexity on 17th-century Italian rises 2.4x (3.2x for academic prose), yet embedding similarity remains above 0.85.
In other words: the model "can't predict the next token," but "knows what the passage is about."
This matters because we have long treated perplexity as a proxy for model understanding—and this proxy may be fundamentally wrong.
A Four-Dimensional Diagnostic Framework
Author Maria Levchenko (University of Bologna) proposes a framework that decomposes "historical language difficulty" into four independent dimensions:
1. Tokenization Cost: how many tokens does historical text get split into? Finer splits mean lower encoding efficiency. 2. Prediction Uncertainty (Surprisal/Perplexity): how surprised is the model by the next word? 3. Semantic Robustness: embedding similarity—is the model's semantic representation stable? 4. Context Sensitivity: how much does perplexity drop with a small contextual hint?
Earlier work treated "historical difficulty" as a single obstacle, conflating orthographic variation, linguistic distance, and pretraining exposure. The framework's core contribution: these four dimensions can vary independently and cannot substitute for one another.
Three Datasets, Three Centuries
- 17th-century Italian (1610–1689): a newly compiled corpus digitized from original page images, spanning religious treatises to academic prose
- **19th-century Italian (*The Betrothed*): a high-exposure control—widely digitized classic literature, likely abundant in training data
- 18th-century Russian (civil-print books): an orthographic stress test—Russian spelling was reformed in the 18th century, differing sharply from modern Russian while the language itself stays close
- 17th-century Italian: on average 2.4x more surprising (3.2x for academic prose)
- 18th-century Russian: only a mild increase
- Title: How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation
- Author: Maria Levchenko
- Institution: University of Bologna
- arXiv: https://arxiv.org/abs/2606.27275
- HTML: https://arxiv.org/html/2606.27275v1
- Venue: The 22nd Conference on Information and Research Science Connecting to Digital and Library Science
Core Findings: Tokenization Tax ≠ Comprehension Tax
Finding 1: The tokenization tax is mechanical, unrelated to understanding
Both 17th-century Italian and 18th-century Russian suffer ~25–30% tokenization inflation—roughly the same "tax." Yet their prediction difficulty diverges dramatically:
Tokenization cost and perplexity do not correlate. Tokenization is mechanical—BPE tokenizers split unfamiliar character combinations more finely—without reflecting whether the model "understands" the text.
Finding 2: High perplexity ≠ broken semantic representation
The most counterintuitive finding: while perplexity spikes for 17th-century Italian, embedding similarity stays above 0.85 across all datasets.
This means models can represent the semantics of historical text even when they cannot predict it. Generation (predicting the next word) and representation (understanding what a passage means) are distinct abilities using different computational paths.
Analogy: reading Classical Chinese, you may be unable to guess the next character (weak generation) yet fully grasp the meaning (normal representation).
Finding 3: Genre matters more than era
17th-century religious treatises and academic prose are contemporaneous but differ enormously in perplexity: academic prose is 3.2x more surprising, religious texts only 1.8x.
"Historicalness" is not a uniform dimension—genre may affect model comprehension more than age. Religious texts are likely more common in training data (Biblical commentaries and theological works were heavily digitized), while 17th-century academic prose (alchemy, proto-scientific papers) is far rarer.
Finding 4: A single sentence removes 60% of perplexity
A minimal intervention was tested: adding one line to the prompt—"the following is 17th-century Italian text."
That single hint reduced historical perplexity by roughly 60%**.
The model is not helpless with historical languages—it simply hasn't "activated" the right contextual mode. A simple temporal context prompt suffices.
Finding 5: Canonical texts are misleading benchmarks
*The Betrothed*, a 19th-century Italian classic, performs close to modern Italian on every dimension: low token inflation, moderate perplexity, high embedding similarity.
Is that because it is "historical text"? No—because it is widely digitized, edited, disseminated, and likely abundant in training data. Using canonical texts to evaluate historical-language handling seriously overestimates model capability.
Why It Matters
For digital humanities, the practical implication is direct: digital libraries can safely use LLMs for semantic retrieval of historical texts, but generative tasks (auto-translation, modernization) require additional adaptation.
The deeper methodological contribution is for NLP at large: perplexity is not a proxy for understanding. Mechanistic-interpretability researchers have made similar arguments, but this paper uses historical languages as a natural experiment to provide the clearest empirical evidence.
A model can be simultaneously "perplexed" and "comprehending"—just as a person can speak a foreign language poorly yet understand it perfectly. Generation and representation are two paths; perplexity only measures the former.
Next time you see a model with high perplexity on some dataset and conclude "the model doesn't understand this domain"—think of 17th-century Italian. The model may simply be unable to say it, but it knows.