Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Tests (arXiv 2609.05385)

Paper Overview

Field: Machine Learning Authors: Urja Pawar, Rajitha Ramanayake, Nabeel Kemal, Ashwin Kandath, Owen O'Neill, Guillaume Bourgeon, Houssem Chatbri Published: 2026-09-04 arXiv: 2609.05385

Summary

LLM decision components that operate within agent workflows often produce action-relevant recommendations or judgements together with explanations. Operators may use these named factors to monitor a system, diagnose errors, or decide when to escalate an output. Such use assumes that explanations agree with the component's observable decision behaviour.

The paper tests two interpretations of the named factors:

• Necessity: changing a factor would change the output.
• Sufficiency: retaining the factor while removing other changeable information would preserve the output.

The authors evaluate these interpretations in two synthetic use cases: recommending advisors to clients and judging prompts for harmfulness or risk. Models return an output plus the top three factors that most influenced the decision. Controlled black-box interventions estimate necessity scores by how often changing a factor flips the output, and sufficiency scores by how often retaining it preserves the output.

Findings

Across eight models from the Claude, GPT, and Gemini families:

• Average Spearman correlation between cited-factor ranking and necessity/sufficiency scores: 0.349 / 0.354 for advisor recommendation; 0.431 / 0.580 for prompt monitoring.
• Rate at which an *uncited* factor scored above the lowest *cited* factor: advisor task 57.6% (necessity) and 58.1% (sufficiency); prompt monitoring 25.8% and 8.9%.

The cited top-three factors contain useful information, but they do not reliably identify the three most influential factors under necessity or sufficiency interpretations. The proposed framework provides a black-box reliability test for explanations used in agent supervision.

--- *Collected automatically on 2026-09-09*