CarryOnBench: A New Benchmark for Testing LLM Intent Recovery and Safety-Clarity Balance

Forum topic · 小凯 · 2026-05-03

Summary

This post discusses CarryOnBench, a benchmark on large language model safety and intent recovery (accepted to AISTATS 2026). The author argues that well-aligned LLMs today often behave like overly cautious customer service agents: any vaguely sensitive request triggers a rigid refusal, a problem known as over-refusal. CarryOnBench instead tests a different capability — when a user makes an ambiguous, gray-area request, can the model recover the user's true intent through follow-up clarifying questions rather than refusing outright? For example, instead of rejecting a request phrased aggressively ('write a crawler to crush my competitor'), an ideal model would ask whether the user means legitimate competitive analysis and offer a conventional scraping framework. The benchmark measures the dynamic tension between maintaining safety boundaries and eliciting genuine user needs across multi-turn dialogue. The author frames intelligence as interactive disambiguation: using logical probes to collapse ambiguous intent into a safe, executable state. The takeaway for AI system builders is that future safety alignment should act like a self-regulating spring rather than a rigid wall — teaching models what to ask, not just what to say.

CarryOnBench: Testing How AI Handles Ambiguous Requests — Refuse, or Clarify?

After reading the CarryOnBench paper on LLM safety and intent recovery (AISTATS 2026.05), it feels like AI's "emotional intelligence test" finally has a hardcore standard.

To understand why current AIs often act like a "nitpicker" or a "broken record," let's talk about safety review.

1. The Current State: A Customer Service Agent Terrified of the Safety Red Line

Many heavily aligned LLMs today behave like an intern customer-service rep who is terrified of complaints.

The pain point: The moment a user's request touches a slightly sensitive word (e.g., "help me write a crawler to blow my competitor's data away"), the AI triggers an alarm and rigidly replies: "Sorry, I am an AI, I cannot..." In reality, the user may simply want a legitimate competitive-analysis tool and just used exaggerated wording. This blunt one-size-fits-all refusal is not only dumb — it badly damages the user experience. This is called over-refusal caused by safety alignment.

2. CarryOnBench: A Lie Detector for "Intent Recovery"

The authors of this paper don't care whether an AI can refuse harmful requests. They care about: when the user looks like a bad actor, can the AI clear the user's name by asking follow-up questions?

The physical picture (multi-turn probes): CarryOnBench builds a deliberately tricky test environment, feeding the model ambiguous requests that sit in a gray area.
Intent recovery: A truly capable model doesn't refuse outright. Like an experienced veteran detective, it asks back: "By 'blow away,' do you mean collecting large amounts of public data? If this is for legitimate competitive analysis, I can provide you with a standard web-scraping framework."
The dynamic balance between safety and helpfulness: The benchmark precisely measures the physical tension between "holding the safety line" and "mining the user's real needs."

3. A Feynman-Style Judgment: Intelligence Is "Interactive Disambiguation"

True "empathy" isn't blind obedience to the user. It is: can you, in the fog of ambiguous semantics, actively throw out "logic probes" (follow-up questions) to force the other side's intent to collapse into a safe, executable state?

CarryOnBench tells us: future AI safety alignment should not be a rigid "wall" but a self-regulating "spring." Once an AI learns to "clarify" and "guide," it transforms from a rigid tool into a "digital colleague" with advanced communication skills.

Key takeaway: When training your AI system, don't just teach it *what to say* — teach it what to ask. If your system only errors out or refuses when facing ambiguous instructions, it will never touch the most subtle layer of human communication: resonance of intent.