Paper Overview
Field: NLP Author: Manuel Pita Published: 2025-07-12 arXiv: 2507.08695
Abstract
A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the question is whether the model follows the construct's theory or reaches the right code by correlated shortcuts.
The paper tests this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined according to the theory's explicit rules. If calibration can close this gap, some portability should survive across models and languages; if not, the construct–model pairing itself may be at fault.
Key Findings
- The study asks whether a calibrated English tool can transfer to AMALIA-9B and European Portuguese — for one construct and one corpus, it cannot.
- Decomposition recovered only about half of AMALIA's holistic performance.
- Error analysis indicates reliance on surface correlation, especially moral outrage appearing near authority figures.
- An open multilingual LLM, under the same instructions, narrowed the gap on the same Portuguese corpus — suggesting the corpus is not the primary explanation.
- AMALIA remains useful for large-scale screening and pre-coding, but is not yet accurate enough for standalone measurement of this construct.
Conclusion
This work is a single counterexample, not a verdict on national models. Its main claim is methodological: sovereign LLM benchmark suites should test not only agreement with human coders, but also the evidence path by which that agreement is achieved.
--- *Collected automatically on 2026-07-13*