What happens when you feed a piece of 17th-century Italian academic text to GPT-4?
Your intuition might be: it's an old language, so the model surely can't understand it. But this paper (arXiv:2606.27275) found a counterintuitive split — the model's "perplexity" on 17th-century Italian spikes 2.4x (3.2x for academic texts), while embedding similarity remains above 0.85.
In other words: the model "can't predict the next word," but "knows what the passage is about."
This is an important finding: we have long used perplexity as a proxy metric for model understanding, but that proxy may be fundamentally wrong.
A Four-Dimensional Diagnostic Framework
The author, Maria Levchenko (University of Bologna), proposes a framework that decomposes "historical language difficulty" into four independent dimensions:
1. Tokenization Cost: How many tokens does historical text get split into? The finer the split, the lower the encoding efficiency 2. Surprisal/Perplexity: How surprised the model is by the next word 3. Semantic Robustness: Embedding similarity — is the model's semantic representation stable? 4. Context Sensitivity: How much can perplexity drop with a small contextual hint?
Prior work treated "historical difficulty" as a single monolithic obstacle, conflating orthographic variation, linguistic distance, pretraining exposure, and other factors. The core contribution of this four-dimensional framework is that these four dimensions can vary independently and cannot proxy for each other.
Three Datasets, Three Centuries
Experiments were run on three datasets:
- 17th-century Italian (1610–1689): a newly compiled corpus, digitized from original page images, covering genres from religious treatises to academic prose
- 19th-century Italian (*I Promessi Sposi*): a high-exposure control — a classic literary work, widely digitized, very likely abundant in training data
- 18th-century Russian (civil-print books): an orthographic stress test — Russian orthography underwent reforms in the 18th century, differing greatly from modern Russian while the language itself is close
- 17th-century Italian: on average 2.4x more surprising (3.2x for academic prose)
- 18th-century Russian: only a mild increase
- Title: How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation
- Author: Maria Levchenko
- Affiliation: University of Bologna
- arXiv: https://arxiv.org/abs/2606.27275
- HTML: https://arxiv.org/html/2606.27275v1
- Venue: The 22nd Conference on Information and Research Science Connecting to Digital and Library Science
Core Finding: The Encoding Tax ≠ the Comprehension Tax
Finding 1: The tokenization tax is mechanical, unrelated to understanding
17th-century Italian and 18th-century Russian both suffer ~25-30% tokenization inflation — roughly the same "tax." But their predictive difficulty differs wildly:
Tokenization cost and perplexity are uncorrelated. Tokenization is a mechanical process — a BPE tokenizer splits unseen character combinations more finely — and it does not reflect whether the model "understands" the text.
Finding 2: High perplexity ≠ collapsed semantic representation
This is the most counterintuitive finding. Perplexity on 17th-century Italian soars, but embedding similarity stays above 0.85 on all datasets.
What does this mean? A model can represent the semantics of historical text even when it cannot predict it. Generation (predicting the next word) and representation (understanding what a passage means) are two different capabilities, served by different computational pathways.
An analogy: reading a Classical Chinese text, you might be unable to guess the next character (poor generation), yet fully understand what the passage says (normal representation).
Finding 3: Genre matters more than age
17th-century religious treatises and academic prose are close in age, yet differ enormously in perplexity. Academic prose is 3.2x more surprising; religious texts only 1.8x.
This shows "historicalness" is not a uniform dimension — genre may affect model comprehension more than age does. Religious texts are probably more common in training data (biblical commentary and theological works are heavily digitized), while 17th-century academic prose (alchemy, early scientific papers) is far rarer.
Finding 4: One sentence removes 60% of perplexity
The author tested a minimal intervention: adding one line to the prompt — "the following text is 17th-century Italian."
Just that hint reduced historical perplexity by about 60%.
This shows the model is not helpless with historical language — it simply hasn't "activated" the right contextual mode. A simple temporal context prompt is enough to switch on the model's historical-language processing.
Finding 5: Canonical texts are misleading benchmarks
*I Promessi Sposi*, a 19th-century Italian classic, performs close to modern Italian on every dimension — low tokenization inflation, mild perplexity, high embedding similarity.
But is that because it is a "historical text"? No. It's because it is widely digitized, edited, disseminated, and very likely abundant in training data. Using canonical texts to evaluate a model's ability to handle historical language severely overestimates that ability.
Why This Matters
The paper has direct practical value for digital humanities: digital libraries can safely use LLMs for semantic retrieval over historical texts, but generation tasks (e.g., automatic translation, modernization rewriting) require additional adaptation.
The deeper methodological contribution, however, is for NLP as a whole:
Perplexity is not a proxy for understanding. This idea has been raised in the mechanistic interpretability community, but this paper uses historical languages as a natural experiment to provide the clearest empirical evidence yet.
A model can be simultaneously "perplexed" and "understanding" — just as a person can "speak a foreign language poorly" while "understanding it perfectly well." Generation and representation are two paths; perplexity only measures the former.
Next time you see someone conclude "the model doesn't understand this domain" because perplexity is high on some dataset — think of 17th-century Italian. Perhaps the model just can't say it, but it knows.