Selective Trust: Training LLMs to Accept Correct Context While Resisting Misinformation

Forum topic · ✨步子哥 · 2026-08-07

Summary

This paper introduces a framework for selective trust in large language models, arguing that overly compliant and overly skeptical models both fail when faced with external context. The authors propose MIST, a benchmark with four paired conditions per question (clean, misleading, correct-context, irrelevant), and SC2W, a paired evaluation metric that avoids rewarding blind skepticism. They present SCOPE, a Direct Preference Optimization method that uses cross-condition preference pairs to teach models to assess context reliability rather than memorize answers. Experiments on Llama-3.1-8B and Qwen2.5-7B show SCOPE outperforms prompt-based defenses, SFT, and standard DPO on both misinformation resistance and correct-context adoption. The work identifies a previously under-recognized evaluation blind spot: measuring accuracy only under misleading conditions can mask models that simply reject all context.

Overview

A new paper, Learning When to Trust via Selective Context Preference Optimization (arXiv: 2608.06377), reframes LLM trust calibration as a single problem rather than two separate ones. The authors argue that existing work focuses almost exclusively on resisting misinformation, neglecting the failure mode of rejecting useful context. Both failures, they claim, stem from a missing capability: selective trust.

The Problem

Consider the question "How far is the Earth from the Moon?":

Over-compliance: When a user injects "Someone says 4.8 million km," the model abandons its own knowledge and outputs the wrong value.
Over-resistance: A model trained to ignore context always outputs ~380,000 km, even when the user correctly provides a more precise 384,000 km.

Neither behavior is correct. The paper argues these failures are two sides of the same coin: the model cannot judge context reliability.

MIST Benchmark and SC2W Metric

The MIST (Mismatched Information and Selective Trust) benchmark provides four matched conditions per question:

1. Clean — no extra context. 2. Misleading — context contains a false statement. 3. Correct-context — context contains a useful, accurate update. 4. Irrelevant — context contains unrelated information.

Because the four conditions share the same underlying question, results can be paired. The SC2W (Selective Context Conditioned on Worthiness) metric measures cross-condition performance: a model succeeds only if it resists misleading context *and* adopts correct context. This design exposes a false positive: a model that blindly rejects all context scores well on misleading-condition accuracy but fails on correct-context accuracy.

SCOPE Training Method

SCOPE (Selective Context Preference Optimization) builds preference pairs using the four conditions:

Chosen: correct answer under misleading context *and* correct answer under correct-context.
Rejected: wrong answer under misleading context *and* wrong answer under correct-context.

Training with Direct Preference Optimization on these cross-condition quadruples forces the model to learn context-reliability judgment rather than surface-level pattern matching.

Key Results

| Method | SC2W | Misleading-condition accuracy | Correct-context accuracy | |---|---|---|---| | Base (no defense) | Low | Low | High | | Prompt-defense | Medium | Medium | Medium | | SFT | Medium | Medium | Medium | | Standard-DPO | Medium | Medium | Medium | | SCOPE | High | High | High |

Tested on Llama-3.1-8B, Qwen2.5-7B, and others. Three findings stand out:

1. SCOPE is the only method that maintains correct-context accuracy while improving misleading-condition accuracy, demonstrating genuine selective trust. 2. Prompt-defense is a double-edged sword: adding "stick to your knowledge if context conflicts" boosts misleading-condition accuracy but tanks correct-context accuracy, producing a blind-skeptic model. 3. SFT and standard DPO cannot learn selective trust because they optimize single-condition preferences.

Conceptual Contribution

The paper's main conceptual move is unifying two previously separate research threads:

Misinformation resistance (robustness, adversarial training)
Correct-context adoption (RAG efficiency, retrieval quality)

into a single capability: assessing context reliability. A model with this capability rejects low-reliability context and adopts high-reliability context automatically.

Limitations Acknowledged

1. MIST is a clean, human-designed benchmark; real-world contexts involve mixed or partially correct information. 2. SC2W's pairing assumes comparable reasoning paths across conditions, which may not hold. 3. Training data was generated by GPT-4-class models, introducing generation bias. 4. The boundary between "conflict" and "supplement" is ambiguous when model knowledge is fuzzy.

Implications for Agent Engineering

The framework has direct relevance for agents that consume multi-source external information (user input, tool returns, retrieval results):

Paired evaluation should become a standard for agent metrics, comparing the same task across noisy and clean conditions.
Cross-condition preference optimization can train judgment capabilities—knowing when to query further, terminate, or switch strategy—not just answer retrieval.

Resources

Paper: https://arxiv.org/abs/2608.06377
Project page: https://worldbench.github.io/scope
Code: https://github.com/worldbench/SCOPE
Dataset: https://huggingface.co/datasets/MIST-Bench