Learning When to Trust: SCOPE and the MIST Benchmark for Selective Context Trust in LLMs

Forum topic · 小凯 · 2026-08-29

Summary

This article presents an in-depth Chinese-language walkthrough of the paper 'Learning When to Trust via Selective Context Preference Optimization' (arXiv:2608.06377). The paper addresses selective trust: large language models must decide, context by context, whether external information (RAG documents, user-supplied context, tool outputs) should be adopted, resisted, or ignored. The authors introduce MIST, a benchmark that evaluates each question under four matched conditions—Clean, Misleading, Correct-Context, and Irrelevant—preventing models from gaming tests by simply ignoring all external context, plus a metric called SC2W measuring how often models abandon correct answers after exposure to misleading context. Their training method, SCOPE, builds preference pairs for DPO by mining failure cases and balancing them equally across the four condition pairs, teaching models to retain correct answers under misleading context while still exploiting genuinely helpful information. Experiments show mainstream open-source LLMs are broadly vulnerable, that scale alone does not confer selective trust, and that SCOPE substantially reduces SC2W without degrading use of correct context; ablations confirm each condition's contribution. The post closes with discussion of epistemic humility, open challenges, and future directions.

This is an English structured summary of a detailed Chinese forum walkthrough of the paper "Learning When to Trust via Selective Context Preference Optimization" (arXiv:2608.06377) by Xian Sun, Wei Chow, Yingshuo Wang, Junhao Liu, Wei Gao, Qing Wu, and Lingdong Kong (NLP / alignment / in-context learning).

Key points

The problem: selective trust

LLMs increasingly consume external context (RAG documents, user input, tool results). Naive trust exposes models to prompt injection and misinformation; blanket distrust forfeits valuable external knowledge.
The paper argues true robustness comes from discrimination, not resistance: deciding per-context whether to adopt, resist, or ignore a signal.
The post illustrates with four scenarios (correct RAG info, outdated-but-true info, malicious false injection, irrelevant context) showing why a binary trust decision is insufficient.

MIST benchmark

Existing benchmarks (TruthfulQA, HaluEval) test only Clean vs. Misleading conditions, so a model can score well by simply ignoring all external context.
MIST evaluates every question under four matched conditions: Clean, Misleading, Correct-Context, and Irrelevant, isolating the nature of the external context as the only varying factor.
A new metric, SC2W (Selective Context-to-Wrong), measures the fraction of Clean-correct answers that become wrong under Misleading context:

\text{SC2W} = \frac{\text{correct on Clean and wrong on Misleading}}{\text{total correct on Clean}}

Lower SC2W indicates better resistance to misleading context, while Correct-Context accuracy guards against the "ignore everything" shortcut.

SCOPE training method

SCOPE (Selective COntext Preference Optimization) builds DPO preference pairs from mined failure cases where the model answers correctly on Clean, incorrectly on Misleading, and correctly on Correct-Context/Irrelevant—i.e., high-value teaching examples.
Preference pairs are constructed equally across four condition contrasts (25% each): Clean vs. Misleading, Correct-Context vs. Misleading, Irrelevant vs. Misleading, and Clean vs. Correct-Context, balancing resistance to misleading context with exploitation of helpful context.
The DPO objective optimizes:

\mathcal{L}_{\text{DPO}} = -\log \sigma\left(\beta \cdot \log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \cdot \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)

Findings

Baseline open-source models show high SC2W and extreme all-trust or all-distrust tendencies; larger scale alone does not yield selective trust.
After SCOPE training: SC2W drops substantially, Correct-Context accuracy is preserved, and models become stable under irrelevant context—generalizing across architectures and scales.
Ablations: training only on Clean-vs-Misleading pairs causes over-conservatism; removing Clean-vs-Correct-Context pairs leaves helpful context underused; unbalanced condition ratios cause overfitting to the dominant condition.

Open questions raised by the post

Handling multi-source, contradictory evidence; dynamic source-reputation modeling; interpretability of trust decisions; adversarially targeted misinformation.

Reference

Sun, X., Chow, W., Wang, Y., Liu, J., Gao, W., Wu, Q., & Kong, L. (2026). Learning When to Trust via Selective Context Preference Optimization. arXiv:2608.06377.