CaRT: Teaching LLM Agents to Know When They Know Enough — When AI Learns to Say "Enough"

Forum topic · ✨步子哥 · 2025-11-08

Summary

This post reviews the Carnegie Mellon paper "CaRT: Teaching LLM Agents to Know When They Know Enough" (arXiv:2510.08517), which tackles a core LLM weakness: deciding when to stop gathering information. Modern LLMs either over-think, wasting compute, or terminate prematurely with wrong answers. CaRT (Counterfactuals and Reasoning for Termination) combines two ideas: generating hard negative counterfactual trajectories—near-identical sequences missing one critical piece of information—and attaching explicit natural-language reasoning to each termination decision, acting as a "verbalized value function." In simulated medical diagnosis (MedQA-USMLE/MedMCQA questions, GPT-4o as doctor, Llama-3.1-8B as patient), CaRT's termination points closely track diagnostic accuracy saturation and generalize robustly to out-of-distribution dermatology questions where SFT baselines fail. Applied to AIME 2025 math reasoning with Qwen3-1.7B, CaRT achieves the highest success rate while using the fewest tokens, showing adaptive test-time compute allocation. Ablations confirm both counterfactuals and reasoning are essential, and that reasoning acts as an implicit regularizer. The author frames this as teaching AI the wisdom of knowing when to stop—critical as AI enters high-stakes decisions.

CaRT: Teaching LLM Agents to Know When They Know Enough

This post discusses a recent Carnegie Mellon paper, "CaRT: Teaching LLM Agents to Know When They Know Enough" (arXiv:2510.08517v1), about a deep AI dilemma: knowing when to stop is harder than knowing how to start.

The Problem: When More Becomes Less

Modern LLMs face a "choice paralysis": they either think endlessly or conclude too hastily. The paper points out that off-the-shelf LLMs can barely predict their own success rates, let alone explore in a principled way. Extra context can even hurt — models may latch onto spurious cues, which is dangerous in domains like medical diagnosis.

Classical optimal stopping theory (e.g., the secretary problem) works well in rule-based settings but struggles in the open-ended world of natural language.

Formalization: Given a question $$x$$ , information gathering produces observations $o_{0:t}$ and reasoning tokens $z_{0:t}$ . A policy $\pi(a_t|x, z_{0:t}, o_{0:t})$ chooses continue or terminate at each step, maximizing reward under a penalty factor $\gamma \in (0,1]$ for over-computation. Supervised fine-tuning (SFT) alone tends to learn superficial rules like "stop when the conversation gets long."

The CaRT Method

CaRT (Counterfactuals and Reasoning for Termination) has two key components:

1. Hard negative counterfactuals: For each optimal termination point, construct a nearly identical trajectory missing one critical piece of information (e.g., a question whose removal drops diagnosis success from ≥50% to <30%). This minimal-change/maximal-contrast design forces the model to focus on information that truly determines success.

2. Explicit natural-language reasoning: Each termination decision is accompanied by verbal reasoning comparing expected outcomes of stopping vs. continuing — effectively a "verbalized value function" that runs a thought experiment before deciding.

Medical Diagnosis Experiments

Setup: 1,133 medium-difficulty questions from MedQA-USMLE and MedMCQA (single-turn success <40%, full-information success ≥20%). GPT-4o plays doctor, Llama-3.1-8B plays patient, up to 20 QA rounds. Every dialogue prefix is labeled with diagnostic accuracy computed over 50 samples by an external diagnostic model.
Results: CaRT's termination points land almost exactly where diagnostic accuracy saturates, beating baselines and SFT. On out-of-distribution dermatology questions, baselines and SFT underperform a naive fixed-budget strategy, while CaRT remains robust — evidence of transferable decision-making rather than surface statistics.
Adding RL post-training made dialogues longer, slightly reducing efficiency — hinting at a delicate balance between exploration and termination.

Math Reasoning Experiments

On AIME 2025, with Qwen3-1.7B outputs segmented into "episodes" (each starting with a logical/strategy-shift sentence), CaRT decides after each episode whether to answer or keep thinking. CaRT achieves the highest success rate with the fewest tokens, demonstrating intelligent allocation of test-time compute — teaching models to use compute cleverly rather than just adding more.

Ablation Insights

Both counterfactuals and reasoning are essential; removing either degrades performance in distinct ways.
Adding a separate confidence-prediction task yields only marginal gains — the reasoning itself already encodes sufficient value information.
CaRT shows an elegant "staircase" termination curve, spiking exactly at information-gain points. Reasoning acts as an implicit regularizer improving generalization.

Limitations and Future Directions

CaRT assumes a fixed information-gathering policy and only optimizes termination timing; in reality, what to ask and when to stop are coupled. Future work could jointly optimize exploration and termination (e.g., curriculum training, dense rewards).
Explicit value estimation or uncertainty modeling could make termination more robust; CaRT's verbalized reasoning suggests value functions might themselves be generative processes.

Conclusion

Echoing the classical Chinese maxim "知止而后有定" (knowing when to stop brings stability), the author argues that intelligence lies not in knowing more, but in knowing what is enough. As AI enters high-stakes domains — medicine, finance, law, education — "when to stop" is becoming as important as "how to act."

References

1. Liu, G., Qu, Y., Schneider, J., Singh, A., & Kumar, A. (2025). *CaRT: Teaching LLM Agents to Know When They Know Enough*. arXiv:2510.08517v1. 2. Setlur, A., et al. (2024). *Rewarding Progress: Scaling up Automated Data Generation for Test-Time Compute*. 3. Gandhi, K., et al. (2024). *Stream of Search (SoS): Learning to Reason, Search, and Generate*. 4. Singhal, P., et al. (2024). *Craft-MD: A Benchmark for Medical Diagnosis with LLMs*. 5. Touvron, H., et al. (2023). *Llama 2: Open Foundation and Fine-Tuned Chat Models*.