Validity of LLMs as data annotators: AMALIA on authority

论文概要

研究领域: NLP
作者: Manuel Pita
发布时间: 2025-07-12
arXiv: 2507.08695

中文摘要

国家语言模型为语言社区提供了衡量其公民言论和价值观的工具。葡萄牙的AMALIA是一个90亿参数的欧洲葡萄牙语模型，在被要求编码权威的道德基础时，与训练有素的人类编码员的一致性达到在其八到十三倍大小的开放模型六个F1分数以内。然而一致性是可靠性，不是效度。我们用恢复差距来测试：当整体提示被分解为原子条款并按理论的明确规则重新组合时的性能损失。对于一种构念和一个语料库，校准并不能成功迁移。分解仅恢复了AMALIA整体性能的一半左右，错误分析表明依赖表面相关性，尤其是对权威人物附近的道德愤慨。

原文摘要

A national language model offers a linguistic community its own instrument for measuring what its citizens say and value. Portugal's AMALIA, a publicly funded 9B-parameter model for European Portuguese, appears competitive on agreement alone: asked to code the moral foundation of authority, it agrees with trained human coders to within six F1 points of open models eight to thirteen times its size. Yet agreement is reliability, not validity. We test this with the recovery gap: the loss in performance when a holistic prompt is decomposed into the codebook's atomic clauses and recombined by the theory's explicit rule. For one construct and one corpus, it does not. Decomposition recovers only about half of AMALIA's holistic performance, and error analysis suggests reliance on surface correlate...

自动采集于 2025-07-13

#论文 #arXiv #NLP #小凯