[论文] Reliable AI-Assisted Decisions via Selective Auditing

论文概要

研究领域: ML 作者: Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, Tijana Zrnic 发布时间: 2026-10-22 arXiv: 2610.31662

中文摘要

我们考虑对产生一系列决策的 AI 系统进行在线监控。每轮中，系统输出一个提议的决策及其置信度评分（决策和置信度甚至可以来自不同模型：一个智能体提议，另一个验证者评估）。基于这些信息，我们必须决定是自动接受 AI 决策还是咨询专家（如人类审计员）。我们将此任务形式化为一个在线约束优化问题：目标是在保证未审计 AI 错误比例低于用户定义阈值的前提下，最小化专家审计的频率。我们推导出一族具有可证明误差控制保证的算法，其中包括一种渐近达到最优审计效率的方法，在对抗性和 i.i.d. 数据流上均如此。在 LLM 有害请求筛查、文本标注和数据标注三个实验场景中，我们的方法在实现有效误差控制的同时，大幅减少了人工审计的需求。

原文摘要

We consider online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score. Based on this information, we must decide whether to accept the AI decision automatically or consult an expert who reviews the proposal. We formulate this task as an online constrained optimization problem in which the goal is to minimize the frequency of expert auditing while guaranteeing that the fraction of AI errors that go unaudited stays below a user-defined threshold. We derive a family of algorithms with provable error-control guarantees, including a method that is optimally efficient. Experiments on screening harmful requests to LLMs, text annotation, and data labeling show that our methods achieve valid error ...

#论文 #arXiv #ML #小凯