Necessary or Sufficient? Evaluating LLM Explanations with Behavioural Tests

Forum topic · 小凯 · 2026-09-09

Summary

This arXiv paper (2609.05385) by Urja Pawar and colleagues tests whether explanations produced by LLM decision components in agent workflows actually match the models' observable decision behaviour. The authors interpret named factors under two notions: necessity (changing the factor changes the output) and sufficiency (keeping the factor while removing other information preserves the output). Using controlled black-box interventions across two synthetic use cases—recommending advisors to clients and judging prompts for harmfulness—the study evaluates eight Claude, GPT, and Gemini models that return an output plus their top three influential factors. Average Spearman correlations between factor ranking and necessity/sufficiency scores were 0.349 and 0.354 for advisor recommendation, and 0.431 and 0.580 for prompt monitoring. Notably, unmentioned factors scored above the lowest-cited factor in 57.6% (necessity) and 58.1% (sufficiency) of advisor cases, versus 25.8% and 8.9% for prompt monitoring. The conclusion: cited top-three factors carry useful information but do not reliably identify the most influential factors, motivating black-box reliability checks for explanations used in agent oversight.

Paper Overview

Field: Machine Learning
Authors: Urja Pawar, Rajitha Ramanayake, Nabeel Kemal, Ashwin Kandath, Owen O'Neill, Guillaume Bourgeon, Houssem Chatbri
Published: 2026-09-04
arXiv: 2609.05385

Summary

LLM decision components operating within agent workflows often produce action-relevant recommendations or judgements together with explanations. Operators may use the named factors to monitor a system, diagnose errors, or decide when to escalate an output. Such use assumes that the explanations agree with the component's observable decision behaviour.

This paper tests two interpretations of the named factors:

Necessity: changing a factor would change the output.
Sufficiency: retaining the factor while removing other changeable information would preserve the output.

Methodology

The authors evaluate these interpretations in two synthetic use cases:

1. Advisor recommendation: recommending advisors to clients. 2. Prompt monitoring: judging prompts for harmfulness or risk.

Models return an output and the top three factors that most influenced the decision. Controlled black-box interventions estimate:

Necessity score: how often changing a factor changes the output.
Sufficiency score: how often retaining a factor preserves the output.

Key Findings

Across eight Claude/GPT/Gemini models:

Average Spearman correlation between cited factor ranking and necessity/sufficiency scores:
Advisor recommendation: 0.349 (necessity), 0.354 (sufficiency)
Prompt monitoring: 0.431 (necessity), 0.580 (sufficiency)
Rate at which unmentioned factors scored above the lowest-cited factor:
Advisor recommendation: 57.6% (necessity), 58.1% (sufficiency)
Prompt monitoring: 25.8% (necessity), 8.9% (sufficiency)

Conclusion

The cited top-three factors contain useful information, but they do not reliably identify the three most influential factors under either necessity or sufficiency interpretations. The proposed framework offers a black-box reliability test for explanations used in agent supervision.

Necessary or Sufficient? Evaluating LLM Explanations with Behavioural Tests

Summary

Paper Overview

Summary

Methodology

Key Findings

Conclusion

Tags