Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Interventions (arXiv 2609.05385)

Paper Overview

Field: Machine Learning Authors: Urja Pawar, Rajitha Ramanayake, Nabeel Kemal, Ashwin Kandath, Owen O'Neill, Guillaume Bourgeon, Houssem Chatbri Published: 2026-09-04 arXiv: 2609.05385

Summary

LLM decision components that operate within agent workflows often produce action-relevant recommendations or judgements together with explanations. Operators may use the named factors in these explanations to monitor a system, diagnose errors, or decide when to escalate an output. Such use assumes the explanations agree with the component's observable decision behaviour.

The paper tests two interpretations of the named factors:

• Necessity: changing a factor would change the output
• Sufficiency: retaining the factor while removing other changeable information would preserve the output

Evaluation Setup

The authors evaluate these interpretations in two synthetic use cases: recommending advisors to clients and judging prompts for harmfulness or risk. Models return an output together with the top three factors that most influenced the decision. Controlled black-box interventions estimate:

• Necessity score: how often perturbing a factor flips the output
• Sufficiency score: how often retaining a factor preserves the output when other information is removed

Key Findings

Across eight models from Claude, GPT, and Gemini families:

• Mean Spearman correlation between cited factor ranking and necessity/sufficiency scores: 0.349 / 0.354 in the advisor recommendation task, and 0.431 / 0.580 in the prompt monitoring task.
• The rate at which an uncited factor scored above the lowest-cited factor: 57.6% (necessity) and 58.1% (sufficiency) for advisor recommendation; 25.8% and 8.9% for prompt monitoring.

Conclusion

The top-three cited factors contain useful information, but under necessity or sufficiency interpretations they do not reliably identify the three most influential factors. The proposed framework provides a black-box reliability test for explanations used in agent supervision.

---

*Auto-collected on 2026-09-09*