Reliable AI-Assisted Decisions via Selective Auditing: Online Error-Control Algorithms (arXiv 2610.31662)

Forum topic · 小凯 · 2026-10-24

Summary

A new paper by Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, and Tijana Zrnic (arXiv 2610.31662) addresses how to monitor AI systems that produce a stream of decisions. In each round, the system outputs a proposed decision plus a confidence score—the decision and score may even come from different models, such as a proposing agent and a separate verifier. The auditor must then choose between automatically accepting the AI decision or escalating to a human expert. The authors formalize this as an online constrained optimization problem: minimize the frequency of expert audits while guaranteeing that the fraction of unaudited AI errors stays below a user-defined threshold. They derive a family of algorithms with provable error-control guarantees, including a method that asymptotically achieves optimal audit efficiency on both adversarial and i.i.d. data streams. Experiments across three tasks—screening harmful requests to LLMs, text annotation, and data labeling—show the methods provide valid error control while substantially reducing the need for human review.

Overview

Research area: Machine Learning Authors: Isaac Gibbs, Elise L. Han, Ryan J. Tibshirani, Tijana Zrnic Published: 2026-10-22 arXiv: 2610.31662

Problem setting

The paper considers online monitoring of an AI system that produces a sequence of decisions. At each round, the system outputs a proposed decision together with a confidence score. Notably, the decision and the confidence score can come from different models: one agent proposes, another verifier evaluates. Based on this information, a monitor must decide whether to accept the AI decision automatically or consult an expert (e.g., a human auditor) who reviews the proposal.

Formulation and contributions

The task is formalized as an online constrained optimization problem: minimize the frequency of expert auditing while guaranteeing that the fraction of AI errors that go unaudited stays below a user-defined threshold.
The authors derive a family of algorithms with provable error-control guarantees.
Among these is a method that is optimally efficient: it asymptotically achieves optimal audit efficiency on both adversarial and i.i.d. data streams.

Experiments

The methods were evaluated in three scenarios:

1. Screening harmful requests to LLMs 2. Text annotation 3. Data labeling

Across all three tasks, the algorithms achieve valid error control while substantially reducing the need for human auditing compared to baseline approaches.