How Surprising Is Historical Italian to Language Models? Tokenization Tax, Comprehension Tax, and a Simple Mitigation

This paper investigates the challenges modern Large Language Models (LLMs) face when processing historical texts. The author argues that 'historical difficulty' is often treated as a monolithic problem, conflating mechanical encoding issues with genuine linguistic comprehension. To address this, the study introduces a diagnostic framework that separates four dimensions: tokenization cost, predictive uncertainty (surprisal), semantic robustness, and context sensitivity. The author evaluates this framework using three distinct datasets: a curated corpus of 17th-century Italian, 19th-century canonical Italian (as a high-exposure control), and 18th-century Russian civil print books.

The study reveals a critical dissociation between how models encode historical text and how well they understand it. While historical texts often suffer from a 'tokenization tax'—where archaic characters like the long 's' cause inefficient subword fragmentation—this mechanical cost does not reliably predict comprehension difficulty. For instance, 18th-century Russian and 17th-century Italian incur similar tokenization penalties, yet the Italian texts are significantly more 'surprising' (higher perplexity) to the model.

Crucially, high surprisal does not mean the model fails to grasp the content. The author demonstrates that even when generation is unstable, embedding similarity remains high (above 0.85), suggesting that modern LLMs can still effectively perform semantic retrieval tasks on historical documents. Finally, the study shows that providing a minimal temporal context prompt (e.g., specifying the century) can reduce historical surprisal by approximately 60%, offering a simple, model-agnostic way to improve performance.

For digital libraries and researchers, these findings are highly encouraging. They suggest that while historical texts may be expensive to process due to tokenization inefficiencies, they do not necessarily require expensive fine-tuning to be useful. Modern LLMs can be safely deployed for semantic search and indexing of historical archives, provided that generative applications are adapted to account for the model's initial temporal misalignment.