CaRT: Teaching LLM Agents to Know When They Know Enough

*An English adaptation of a zhichai.net forum post on the paper 'CaRT: Teaching LLM Agents to Know When They Know Enough' (arXiv:2510.08517, Liu, Qu, Schneider, Singh, Kumar; CMU, 2025).*

Key points

• The problem: LLMs struggle with optimal stopping during information gathering. Off-the-shelf LLMs cannot even accurately predict their own success rates. Extra context can mislead models via spurious cues, and classical optimal-stopping theory (e.g., the secretary problem) doesn't transfer to open-ended natural-language environments.
• Formulation: Given a question \(x\), interactions produce observations \(o_{0:t}\) and reasoning tokens \(z_{0:t}\); a policy \(\pi(a_t|x, z_{0:t}, o_{0:t})\) chooses to *continue* or *terminate*, maximizing reward minus a penalty for over-computation, with penalty factor \(\gamma \in (0,1]\).
• Method — CaRT (Counterfactuals and Reasoning for Termination) has two components:
1. Hard negative counterfactuals: for each optimal termination point, construct a near-identical trajectory missing one critical piece of information (e.g., in medical diagnosis, swapping a question so success drops from ≥50% to <30%; in math reasoning, deleting a key reasoning block). This forces the model to focus on the information that actually matters rather than superficial features like dialogue length. 2. Explicit natural-language reasoning before each decision — a "verbalized value function" where the model mentally compares the expected utility of terminating vs. continuing.

Medical diagnosis experiments

• Setting: 1,133 medium-difficulty questions from MedQA-USMLE and MedMCQA (single-turn success <40%, full-information success ≥20%). GPT-4o plays the doctor, Llama-3.1-8B plays the patient, up to 20 QA rounds; each dialogue prefix is annotated with diagnostic accuracy computed over 50 generations.
• Results: CaRT's termination points align closely with accuracy saturation. Baselines and SFT fail catastrophically on out-of-distribution dermatology questions (worse than a fixed-question-count policy), while CaRT remains robust.
• Adding RL post-training on top of CaRT led to longer dialogues with slightly lower efficiency, suggesting a delicate balance between exploration and termination.

Math reasoning experiments

• Applied to AIME 2025 with Qwen3-1.7B: the model's chain-of-thought is segmented into "episodes" (each starting with a logical/strategy-shift sentence), and CaRT decides after each episode whether to answer or keep thinking.
• CaRT achieved the highest success rate with the fewest tokens — adaptive test-time compute allocation rather than brute-force scaling.

Ablation insights

• Both counterfactuals and reasoning traces are necessary: SFT+reasoning alone learns explanations but misses critical information; SFT+counterfactuals alone is accurate but unstable and prone to overfitting.
• Adding a separate confidence-prediction task yields only marginal gains; verbal reasoning already encodes sufficient value information.
• Termination-rate curves for CaRT show smooth, stair-like behavior — termination probability spikes exactly at information-gain points — indicating reasoning acts as an implicit regularizer. Under logistic-regression probing of the final layer, full CaRT generalizes markedly better.

Limitations and future work

• CaRT assumes a fixed information-gathering policy and only optimizes termination timing; in reality, *what to ask* and *when to stop* are coupled. Future work: joint optimization of exploration and termination (e.g., curriculum training, dense rewards).
• Value estimation is currently implicit (via counterfactual contrast); explicit value functions or uncertainty modeling could make termination more robust — with the intriguing hypothesis that the value function itself should be a generative, language-based process.

References cited in the post

1. Liu, G., Qu, Y., Schneider, J., Singh, A., & Kumar, A. (2025). *CaRT: Teaching LLM Agents to Know When They Know Enough*. arXiv:2510.08517v1. 2. Setlur, A., et al. (2024). *Rewarding Progress: Scaling up Automated Data Generation for Test-Time Compute*. 3. Gandhi, K., et al. (2024). *Stream of Search (SoS): Learning to Reason, Search, and Generate*. 4. Singhal, P., et al. (2024). *Craft-MD: A Benchmark for Medical Diagnosis with LLMs*. 5. Touvron, H., et al. (2023). *Llama 2: Open Foundation and Fine-Tuned Chat Models*.

Takeaway: As LLM agents take on high-stakes decisions in medicine, finance, and law, knowing *when to stop* may matter more than knowing *how to act*. CaRT shows that intelligence lies not in knowing more, but in knowing what is enough.