English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

17th-Century Italian Confuses LLMs 2.4x More, Yet They Still Understand It: Tokenization Tax vs. Comprehension Tax

Forum topic · ✨步子哥 · 2026-06-28

Summary

A paper by Maria Levchenko (University of Bologna, arXiv:2606.27275) shows that GPT-4-class language models find 17th-century Italian academic prose up to 3.2x more perplexing, yet embedding similarity remains above 0.85, indicating stable semantic representation. The study introduces a four-dimensional diagnostic framework separating tokenization cost, prediction surprisal, semantic robustness, and context sensitivity. Across three datasets (17th-century Italian, 19th-century Italian classics, and 18th-century Russian), the authors find that tokenization inflation (~25-30%) does not correlate with perplexity; high perplexity does not imply semantic breakdown; genre matters more than era; a single temporal context prompt reduces historical perplexity by ~60%; and widely digitized classics are misleading benchmarks. The key methodological takeaway: perplexity measures generation, not understanding, so it is a flawed proxy for model comprehension. Digital libraries can safely use LLMs for semantic retrieval of historical texts, but generation tasks need adaptation.

What happens when you feed 17th-century Italian to GPT-4?

Your intuition might say: old language, the model surely can't understand it. But this paper (arXiv:2606.27275) reveals a counterintuitive split — the model's perplexity on 17th-century Italian spikes 2.4x (3.2x for academic texts), yet embedding similarity stays above 0.85.

In other words: the model "can't predict the next word," but "knows what the passage is about."

This matters because the field has long used perplexity as a proxy for model understanding — and that proxy may be fundamentally wrong.

A Four-Dimensional Diagnostic Framework

Maria Levchenko (University of Bologna) proposes decomposing "historical language difficulty" into four independent dimensions:

1. Tokenization Cost — how many tokens does the historical text get split into? Finer splits mean lower encoding efficiency 2. Surprisal/Perplexity — how surprised is the model by the next word? 3. Semantic Robustness — embedding similarity: is the model's semantic representation stable? 4. Context Sensitivity — how much does perplexity drop with a small contextual hint?

Prior work treated "historical difficulty" as a single monolithic obstacle, conflating orthographic variation, linguistic distance, and pretraining exposure. The core contribution: these four dimensions vary independently and cannot proxy for one another.

Three Datasets, Three Centuries

  • 17th-century Italian (1610–1689): newly digitized corpus from page images, spanning religious treatises to academic prose
  • **19th-century Italian (*The Betrothed*): high-exposure control — a classic widely digitized and likely abundant in training data
  • 18th-century Russian (civil print books): orthographic stress test — Russian orthography reformed in the 18th century, differing from modern Russian despite linguistic closeness
  • Core Findings: Encoding Tax ≠ Comprehension Tax

    Finding 1: The tokenization tax is mechanical, not related to understanding

    17th-century Italian and 18th-century Russian both suffer ~25–30% tokenization inflation. But their prediction difficulty diverges wildly:

  • 17th-century Italian: on average 2.4x more surprising (academic prose 3.2x)
  • 18th-century Russian: only a mild increase
  • Tokenization cost does not correlate with perplexity. BPE tokenizers simply split unseen character combinations more finely — a mechanical process that says nothing about whether the model "understands" the text.

    Finding 2: High perplexity ≠ collapsed semantic representation

    The most counterintuitive result: perplexity spikes on 17th-century Italian, yet embedding similarity stays above 0.85 on all datasets.

    The model can represent the semantics of historical text even when it cannot predict** it. Generation (predicting the next token) and representation (understanding what a passage means) are distinct capabilities using different computational paths.

    Analogy: reading classical Chinese, you may not guess the next character (weak generation) but fully grasp the meaning (normal representation).

    Finding 3: Genre matters more than era

    17th-century religious treatises and academic prose are close in time, yet differ dramatically in perplexity: academic prose is 3.2x more surprising, religious texts only 1.8x.

    "Historicalness" is not a uniform dimension — genre may affect model comprehension more than age. Religious texts are likely more common in training data (biblical commentaries and theology are heavily digitized), while 17th-century academic prose (alchemy, early science) is far rarer.

    Finding 4: 60% of perplexity vanishes with one sentence

    A minimal intervention — adding "The following is 17th-century Italian text" to the prompt — reduced historical perplexity by ~60%.

    The model is not helpless with historical language; it simply hasn't activated the right contextual mode. A simple temporal context prompt unlocks its historical-language processing.

    Finding 5: Canonical texts are misleading benchmarks

    *The Betrothed*, a 19th-century Italian classic, performs near-modern Italian on every dimension: low token inflation, mild perplexity, high embedding similarity.

    But that's not because it is "historical" — it's because it is widely digitized, edited, disseminated, and likely abundant in training data. Using canonical texts to evaluate historical-language ability severely overestimates model competence.

    Why This Matters

    Practical implication for digital humanities: digital libraries can safely use LLMs for semantic retrieval of historical texts, but generation tasks (e.g., automatic translation, modernization) need additional adaptation.

    The deeper methodological contribution to NLP at large: perplexity is not a proxy for understanding. Mechanistic interpretability researchers have hinted at this, but this paper uses historical languages as a natural experiment to provide the clearest empirical evidence.

    A model can be simultaneously "perplexed" and "understanding" — like a person who "speaks poorly" but "comprehends" a foreign language. Generation and representation are two separate paths; perplexity only measures the former.

    Next time a model shows high perplexity on some domain and you're tempted to conclude "the model doesn't understand it" — remember 17th-century Italian. Perhaps the model just can't say it — but it knows.

    Paper Information

  • Title: How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation
  • Author: Maria Levchenko
  • Affiliation: University of Bologna
  • arXiv: https://arxiv.org/abs/2606.27275
  • HTML: https://arxiv.org/html/2606.27275v1
  • Venue: The 22nd Conference on Information and Research Science Connecting to Digital and Library Science

Tags

#language-models#perplexity#historical-linguistics#tokenization#digital-humanities#embeddings#nlp-evaluation#17th-century-italian

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178208264