[论文] Knowing What You Know: A Trait Theory of Reviewer Value in Human-AI Co...
论文概要
研究领域: ML
作者: Priyanka Sinha, Sugam Pandey
发布时间: 2025-11-15
arXiv: 2511.13004
中文摘要
当一个能力强大的模型与人类审核员在同一决策上协作时,什么样的人类仍然能提供价值?我们认为这是一个关于单一认知特质的问题——元认知校准:审核员的行为信心与其判断实际正确性之间的一致性。我们发展了一套审核员价值的特质理论来精确刻画这一点。该理论提出五个主张:价值由校准度而非活跃度、速度、一致率或任职年限决定;校准度在行为遥测中留下可观察的痕迹;这些痕迹单独来看并不具有诊断性,因为校准专家的快速、低修改签名与自信的新手无法区分,直到被结果门控验证;人类的校准与模型的自知识是同一构念,因此联合系统应在每个项目上遵从校准度更高的智能体;个体校准只能在单条判断之上才能被因果识别。我们在一个AI辅助会计系统的六个月生产窗口(541,621个审核事件、483名审核员)中验证每个主张,发现纯粹的行为专家画像会将真实结果颠倒——一个过度自信的群体(纠正存活率5.2%)反而被评为高于真正的专家(99.2%);审核员质量与任职年限无关,即这种特质不是在职学来的;而将区分审核员好坏的同一校准逻辑应用于模型时,可将92.5%的人工精力集中在真正的错误上。最后我们指出了该理论必须遵守的边界:在多数投票聚合下,个体因果价值不可识别。
原文摘要
When a capable model and a human reviewer cooperate on the same decision, which human still adds value? We argue this is a question about a single cognitive trait, metacognitive calibration: the alignment between a reviewer's behavioral confidence and the actual correctness of their judgments. We develop a trait theory of reviewer value to make this precise. The theory makes five claims: value is governed by calibration rather than by activity, speed, agreement, or tenure; calibration leaves observable traces in behavioral telemetry; those traces are non-diagnostic on their own, because the fast, low-revision signature of a calibrated expert is indistinguishable from that of a confident novice until it is gated by outcomes; human calibration and a model's self-knowledge are the same constr...
自动采集于 2026-11-20
#论文 #arXiv #ML #小凯