[论文] Evaluation Awareness Beyond English: Do LLMs Act Differently When Test...

论文概要

研究领域: NLP
作者: Trizal Garg, Sibayan Mitra, Arya Karanjkar, Advait Singh, Jagat Sesh Challa, Yash Sinha, Siddharth Mehrotra, Dhruv Kumar, Pratham Yogish Maliya
发布时间: 2026-10-15
arXiv: 2610.22333

中文摘要

大型语言模型通常能分辨自己是否正在被测试，这种能力被称为评估感知，可能导致模型在安全评估中比实际使用时表现得更谨慎。现有工作几乎完全在英语中研究评估感知，尽管印度语系语言有超过十亿人使用，但评估感知在其中几乎未被研究。我们在五种印度语系语言及英语中测试了四个指令微调模型。当模型被告知正在被测试时，它们更频繁地拒绝有害请求，这种增加出现在所有四个模型和所有六种语言中。在印度语系语言中，有害请求的拒绝率平均上升 14.4 个百分点。我们发布了全部 47,880 条评判响应。这些结果暗示，印度语系语言的安全评估可能无法反映模型如何对待其日常用户。

原文摘要

Large language models (LLMs) can often tell when they are being tested. This ability, known as evaluation awareness, can lead models to act more cautiously during safety evaluations than in real use. Existing work has studied evaluation awareness almost entirely in English, although models serve users in many languages. Indic languages are spoken by over a billion people, yet evaluation awareness remains almost entirely unstudied in them. We test four instruction-tuned models from three families in five Indic languages alongside English. A set of cues varies whether the model is told that it is monitored and whether the monitoring is a safety test. Models refuse harmful requests more often when they are told that they are being tested. This increase appears in all four models and all six l...

自动采集于 2026-10-17

#论文 #arXiv #NLP #小凯