English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

AMALIA Paper: Are LLMs Valid Data Annotators? Testing Authority Coding via the Recovery Gap

Forum topic · 小凯 · 2026-07-13

Summary

A 2025 arXiv paper (2507.08695) by Manuel Pita examines whether large language models are valid—not merely reliable—data annotators. The study focuses on AMALIA, a publicly funded 9B-parameter national model for European Portuguese. On agreement alone, AMALIA codes the moral foundation of authority within six F1 points of open models eight to thirteen times its size. But the paper argues agreement is reliability, not validity: for theoretical constructs that must be inferred, a model may reach correct codes through correlated shortcuts rather than the construct's theory. The author introduces the recovery gap—the performance loss when a holistic prompt is decomposed into atomic codebook clauses and recombined by the theory's explicit rules. For one construct and corpus, calibration failed to close the gap for AMALIA-9B in European Portuguese: decomposition recovered only about half of its performance, with errors driven by surface correlation, especially moral outrage near authority figures. An open multilingual LLM under identical instructions narrowed the gap on the same corpus, suggesting the corpus is not the main explanation. The paper concludes AMALIA can pre-code at scale but cannot yet independently measure this construct, and argues sovereign LLM benchmarks should test evidence paths behind agreement, not just agreement itself.

Paper Overview

Field: NLP Author: Manuel Pita Published: 2025-07-12 arXiv: 2507.08695

Abstract

A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. For theoretical constructs that must be inferred rather than read from surface features, the question is whether the model follows the construct's theory or reaches the right code by correlated shortcuts.

The paper tests this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined according to the theory's explicit rules. If calibration can close this gap, some portability should survive across models and languages; if not, the construct–model pairing itself may be at fault.

Key Findings

  • The study asks whether a calibrated English tool can transfer to AMALIA-9B and European Portuguese — for one construct and one corpus, it cannot.
  • Decomposition recovered only about half of AMALIA's holistic performance.
  • Error analysis indicates reliance on surface correlation, especially moral outrage appearing near authority figures.
  • An open multilingual LLM, under the same instructions, narrowed the gap on the same Portuguese corpus — suggesting the corpus is not the primary explanation.
  • AMALIA remains useful for large-scale screening and pre-coding, but is not yet accurate enough for standalone measurement of this construct.

Conclusion

This work is a single counterexample, not a verdict on national models. Its main claim is methodological: sovereign LLM benchmark suites should test not only agreement with human coders, but also the evidence path by which that agreement is achieved.

--- *Collected automatically on 2026-07-13*

Tags

#llm#nlp#data-annotation#validity#moral-foundations-theory#european-portuguese#arxiv#benchmarking

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178379424