CarryOnBench: A Benchmark Testing How LLMs Recover User Intent Instead of Over-Refusing

Forum topic · 小凯 · 2026-05-03

Summary

This Chinese forum post discusses CarryOnBench, an AISTATS 2026 benchmark focused on LLM safety and intent recovery in multi-turn dialogue. The author argues that current aligned models behave like overly cautious customer service agents, refusing any request with sensitive-sounding wording (over-refusal), even when the user's underlying intent may be legitimate. CarryOnBench instead evaluates whether a model can probe ambiguous, gray-area requests with clarifying follow-up questions rather than rejecting them outright. The post frames this as a dynamic balance between safety and helpfulfulness: a capable assistant should ask "what do you mean by X?" and guide users toward safe, executable tasks, instead of triggering canned refusals. Using a Feynman-style analogy, the author describes intelligence as disambiguation through interactive probing, suggesting future safety alignment should work like a self-adjusting spring rather than a rigid wall. The takeaway for AI builders: teach systems what to ask, not just what to say.

This post is a translation/summary of a Chinese forum article about CarryOnBench (AISTATS 2026.05), a benchmark paper on LLM safety and intent recovery.

The Problem: Over-Refusal in Aligned Models

The author compares today's well-aligned LLMs to an intern customer-service agent who is terrified of complaints. The moment a user's request contains slightly sensitive wording — for example, "help me write a crawler to blow away a competitor's data" — the model triggers a canned refusal: "Sorry, I'm an AI, I can't..." Yet the user may simply want a legitimate competitive-analysis tool with exaggerated phrasing.

This one-size-fits-all rejection is called over-refusal caused by safety alignment, and the author argues it is both unintelligent and destructive to user experience.

What CarryOnBench Measures

Rather than testing whether a model can refuse harmful requests, CarryOnBench asks: when a user looks potentially malicious, can the model clear up the suspicion through follow-up questions?

Multi-turn probes: The benchmark deliberately feeds the model ambiguous, gray-area requests that sit on the boundary of safety policies.
Intent recovery: A truly capable assistant should not refuse outright. Like an experienced investigator, it should ask: "By 'blow away' do you mean collecting large amounts of public data? If this is for legitimate competitive analysis, I can provide a standard crawler framework."
Dynamic safety–helpfulness balance: The benchmark precisely measures the tension between holding the safety line and recovering the user's real intent.

The Feynman-Style Takeaway

The author frames "being understanding" not as blind compliance, but as the ability to actively throw out logical probes (clarifying questions) that collapse a user's ambiguous intent into a safe, executable state.

The key insight: future safety alignment should not be a rigid wall but a self-adjusting spring. When an AI learns to clarify and guide, it evolves from a stiff tool into a skilled digital colleague.

Practical lesson for AI builders:

> When tuning your AI system, don't just teach it *what to say* — teach it *what to ask*. If your system only errors out or refuses on ambiguous instructions, it will never reach the most subtle layer of human communication: resonance of intent.