CaRT: Counterfactuals and Reasoning for Termination — Teaching LLMs When to Stop

In many real-world tasks, AI models must strategically gather information across multiple interaction rounds before acting. Strategic information gathering requires not only knowing how to acquire information efficiently, but crucially when to stop collecting and make a decision. Stopping too early leads to decisions made with insufficient information; collecting too much risks "overthinking" and losing sight of the goal. To address this, researchers at Carnegie Mellon University proposed CaRT (Counterfactuals and Reasoning for Termination), a technique designed to teach large language models (LLMs) when to stop gathering information and commit to a decision.

CaRT's core idea is to train models to recognize signals for termination decisions through counterfactual reasoning. Researchers construct paired counterfactual trajectories: in one, the model stops at the appropriate time and makes a correct decision; in the other, a minimal modification makes continued information gathering lead to failure. By learning the difference between these trajectories, CaRT trains the model to explain the rationale for termination via verbal reasoning—articulating why information gathering should stop in the current situation. This helps models distinguish surface correlations from true causal signals, making decisions more robust and interpretable.

Counterfactual Training: The Power of Hard Negatives

Counterfactual training is the foundation of CaRT. It builds "almost right but subtly wrong" hard negatives—samples nearly identical to correct decisions (positives) but differing in a key detail that leads to a wrong outcome. CaRT uses pairs of counterfactual trajectories as training data: one where the model stops at the right moment (positive), and one nearly identical except that gathering a bit more information causes failure (negative). This forces the model to focus on the critical signals of a decision rather than superficially related but non-decisive features.

For example, in an information-gathering task where a model must decide when to stop reading documents and answer a question, CaRT constructs two nearly identical scenarios: one where the model stops after reading the key information and answers correctly, and one where reading an extra irrelevant passage leads to a wrong answer. By contrasting the trajectories, the model learns to identify the information that truly determines the outcome. Prior research has shown that counterfactual data augmentation reduces reliance on spurious features; CaRT applies this idea to termination decisions, teaching the model that "collecting a bit more can actually hurt."

Verbal Reasoning: Thinking Out Loud Before Deciding

CaRT's second innovation is verbal reasoning: the model "talks to itself" before deciding. Similar to chain-of-thought (CoT) prompting, the model generates intermediate reasoning steps—but here it is trained to explain why information gathering should terminate, stating in natural language that the available information suffices and that further collection risks redundancy or confusion.

Benefits include:

• Interpretability: instead of a black-box answer, the model outputs reasoning such as "I have collected key evidence X and Y, which suffice to support conclusion Z, so no further information is needed."
• Self-checking: articulating reasons forces the model to organize information and perform internal logical checks, reducing impulsive decisions.
• Self-supervision: insufficient reasoning can be corrected during training, encouraging more rigorous thinking.

Information Overload vs. Premature Stopping

Information-gathering decisions face two pitfalls:

• Overthinking (information overload): the model keeps seeking new information hoping for a "perfect" basis for decision, collecting noise that obscures judgment and wastes compute. CaRT's counterfactual training teaches the model that extra information may not improve decision quality, so it learns to stop when key information appears.
• Undershooting (premature stopping): the model decides on incomplete data, missing crucial evidence. Hard-negative training lets the model repeatedly experience "one more piece of information would have made it correct," teaching it to look one step further before deciding.

CaRT's balance lies in teaching the model to recognize the necessary and sufficient conditions for a decision—remaining cautious when information is insufficient, and acting decisively when it is enough.

Conclusion

CaRT represents a significant advance in AI termination decisions. By combining counterfactual reasoning and verbal reasoning, it tackles the core question of "when to stop" in information-gathering tasks: hard negatives teach models to identify truly decisive signals, while explicit self-talk improves robustness and interpretability. As LLMs are deployed in strategic information-gathering scenarios—autonomous agents, dialogue systems, decision support—CaRT's approach helps AI systems know "when they already know enough," enabling smarter, more human-like decisions.