Are LLMs Valid Data Annotators? Testing AMALIA on Moral Foundation of Authority

Forum topic · 小凯 · 2026-07-13

Summary

A new arXiv paper (2507.08695) by Manuel Pita examines whether large language models can serve as valid data annotators, not just reliable ones. The study focuses on AMALIA, a publicly funded 9B-parameter language model for European Portuguese, which was asked to code the moral foundation of authority in text. On agreement alone, AMALIA looks competitive: it matches trained human coders to within six F1 points of open models eight to thirteen times its size. However, the paper argues that agreement measures reliability, not validity. To test validity, the author introduces the recovery gap: the performance loss when a holistic annotation prompt is decomposed into the codebook's atomic clauses and recombined according to the theory's explicit rules. For the tested construct and corpus, calibration did not transfer—decomposition recovered only about half of AMALIA's holistic performance. Error analysis suggests the model relies on surface correlates, particularly moral indignation appearing near authority figures. The findings caution researchers using LLM annotators in social science.

Paper Overview

Field: NLP
Author: Manuel Pita
Published: 2025-07-12
arXiv: 2507.08695

Key Points

A national language model gives a linguistic community its own instrument for measuring what its citizens say and value.
AMALIA is a publicly funded 9B-parameter model for European Portuguese.
Asked to code the moral foundation of authority, AMALIA agrees with trained human coders to within six F1 points of open models eight to thirteen times its size.
However, agreement is reliability, not validity.
The paper tests validity via the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule.
For the tested construct and corpus, calibrated performance does not transfer: decomposition recovers only about half of AMALIA's holistic performance.
Error analysis suggests reliance on surface correlations, especially moral indignation near authority figures.

Abstract (from arXiv)

> A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. We test this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. For one construct and one corpus, it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlates...

--- *Auto-collected on 2025-07-13*