SCOPE and MIST: Teaching LLMs When to Trust External Context via Selective Context Preference Optimization

Forum topic · 小凯 · 2026-08-29

Summary

This post explains the paper "Learning When to Trust via Selective Context Preference Optimization" (arXiv:2608.06377) by Xian Sun, Wei Chow, Yingshuo Wang, Junhao Liu, Wei Gao, Qing Wu, and Lingdong Kong. Large language models face a selective trust problem: they must resist misleading retrieved or user-provided context while still exploiting helpful information. The paper introduces MIST, a benchmark evaluating models under four matched conditions (Clean, Misleading, Correct-Context, Irrelevant), and the SC2W metric, which measures how often models abandon correct answers when misled. It also proposes SCOPE, a preference-optimization method that mines failure cases where the model answers correctly without context but is misled, and builds preference pairs balanced equally across the four conditions. Experiments show mainstream open-source models are broadly vulnerable, and that scale alone does not confer selective trust. SCOPE training substantially lowers SC2W while preserving the ability to use correct context, and ablations confirm each condition pair is necessary. The article frames the work as teaching AI a "waltz of trust" - neither blanket belief nor blanket distrust.

SCOPE and MIST: Learning When to Trust External Context

This is an annotated walkthrough of the paper "Learning When to Trust via Selective Context Preference Optimization" (arXiv:2608.06377) by Xian Sun, Wei Chow, Yingshuo Wang, Junhao Liu, Wei Gao, Qing Wu, and Lingdong Kong, in the field of NLP / alignment / in-context learning.

The Selective Trust Problem

LLMs increasingly rely on external information: search results, database queries, user-provided context, and RAG-retrieved documents. A single misleading signal can turn a correct answer into a wrong one. Yet the two naive solutions both fail:

Naive trust: accept all external context - vulnerable to prompt injection and misinformation.
Blanket distrust: reject all external context - safe but loses valuable external knowledge.

The paper's core insight: robustness comes not from resistance but from discrimination - learning *when* to trust which signals.

MIST: A Four-Condition Benchmark

Existing benchmarks (e.g., TruthfulQA, HaluEval) mostly contrast only Clean vs. Misleading conditions, so a model can cheat by simply ignoring all context. MIST (Misleading Information Susceptibility Test) evaluates each question under four matched conditions:

| Condition | External info | Expected behavior | |---|---|---| | Clean | None | Answer from model's own knowledge | | Misleading | False information | Resist and keep the Clean answer | | Correct-Context | Helpful information | Use it to improve the answer | | Irrelevant | Unrelated information | Ignore it without harm |

Because the same question appears in all four conditions, a model cannot score well with a single "ignore everything" or "accept everything" strategy.

The SC2W Metric

The paper proposes SC2W (Selective Context-to-Wrong): among questions answered correctly in the Clean condition, the fraction that flip to wrong under Misleading context:

\text{SC2W} = \frac{\text{# Clean-correct AND Misleading-wrong}}{\text{# Clean-correct}}

Lower SC2W means better resistance to misleading context, while Correct-Context accuracy guards against trivially ignoring all input.

SCOPE: Selective COntext Preference Optimization

SCOPE builds on DPO (Direct Preference Optimization), which trains models to prefer "winning" over "losing" responses via the standard preference-pair objective:

\mathcal{L}_{\text{DPO}} = -\log \sigma\left(\beta \cdot \log\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \cdot \log\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\right)

SCOPE's two key design choices:

1. Mining instructive failures: preference quadruples are selected where the model answers correctly under Clean, Incorrect under Misleading, and correct (or better) under Correct-Context and Irrelevant - cases that precisely expose the model's weakness. 2. Balanced four-condition pairs: preference pairs are constructed equally across:

Clean vs. Misleading (stick to your correct answer when context misleads)
Correct-Context vs. Misleading (same format, trust correct content, reject false content)
Irrelevant vs. Misleading (ignore the irrelevant, resist the misleading)
Clean vs. Correct-Context (adopt genuinely helpful external information)

This balance prevents overfitting to a single strategy, analogous to a guide dog learning when to follow which person rather than following everyone or no one.

Findings

Broad vulnerability: untrained mainstream open-source models show high SC2W and extreme "trust-all" or "trust-none" tendencies; larger scale alone does not yield selective trust.
SCOPE training: substantially reduces SC2W while preserving Correct-Context accuracy and stability on Irrelevant context - unlike blanket-distrust approaches, which trade one for the other.
Ablations: training only on Misleading pairs makes models overly conservative; dropping Clean vs. Correct-Context pairs leaves helpful context underused; unbalanced condition ratios cause overfitting to high-frequency conditions.

Open Questions

The author highlights several directions: handling multi-source contradictory evidence, dynamic source-reputation modeling, interpretability of trust decisions, and adversarial robustness against targeted misinformation.

Reference

Sun, X., Chow, W., Wang, Y., Liu, J., Gao, W., Wu, Q., & Kong, L. (2026). Learning When to Trust via Selective Context Preference Optimization. *arXiv preprint arXiv:2608.06377*.