[论文] Evaluating Cultural Reasoning and Robustness in LLMs for Spanish Polit...
研究领域: NLP 作者: Jorge Valverde-Rebaza, Nathan Garcia-Larios, Karime Ruvalcaba-Perez, Alejandro Rendon, Santiago Sanchez-Ballesteros, Alejandro Martin del Campo 发…
论文概要
研究领域: NLP 作者: Jorge Valverde-Rebaza, Nathan Garcia-Larios, Karime Ruvalcaba-Perez, Alejandro Rendon, Santiago Sanchez-Ballesteros, Alejandro Martin del Campo 发布时间: 2026-11-16 arXiv: 2611.21535
中文摘要
语言模型现在被用来阅读央行文本以进行经济预测,但大多数模型是在所读文档之后撰写的文本上训练的。一个阅读 2021 年 FOMC 声明的模型可能已经知道 2022 年通胀上升,因此其表面技能的一部分可以是对结果的记忆——一种时间形式的测试集污染。我们通过比较现代开放模型与按时间顺序训练的模型(仅在固定日期之前的文本上训练)来衡量这种记忆。在 2000 年至 2026 年的 FOMC 声明和会议纪要上,仅给定日期询问过去的 GDP 结果时,不可能知道答案的模型仅凭对日期和时代的先验仍能达到高达 0.64 的 AUC,因此偶然性是错误的基线。在相同问题上与这些无记忆模型对比,大多数现代模型能回忆哪些年份增长强劲。成员推断分数按时代和文档类型而非模型是否见过这些文档来排序。在实际实时 nowcast 中,我们估计的美国 GDP 的每个前瞻溢价都小于预测比较在宏观经济样本量下能检测到的水平。前瞻审计应直接在同一项目上针对无记忆模型测量记忆。
原文摘要
Large language models (LLMs) are increasingly used to interpret, generate, and mediate public-facing language, which makes their reliability especially consequential in politically sensitive settings. Although sentiment analysis is a common task in this area, affective meaning in social media discourse is often conveyed through culturally situated and pragmatically indirect cues that are not well captured by label-centric evaluation alone. In this paper, we evaluate the robustness of LLMs for sentiment analysis of Spanish political discourse from social media, with emphasis on culturally situated affective interpretation. We introduce a diagnostic protocol that combines three independent human annotation views into Gold and Ambiguous subsets, and extends the corpus with a Synthetic subset ...
*自动采集于 2026-11-18*
#论文 #arXiv #NLP #小凯