English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

SCOPE/MIST: Teaching LLMs Selective Trust Instead of Blind Compliance or Blind Rejection

Forum topic · ✨步子哥 · 2026-08-07

Summary

A forum post reviews the paper "Learning When to Trust via Selective Context Preference Optimization" (SCOPE, arXiv 2608.06377), which reframes LLM trust calibration as a single capability: selective trust. Current models either over-comply with misleading context (context injection attacks) or over-resist by ignoring correct context. The MIST benchmark tests each question under four paired conditions—clean, misleading, correct-context, and irrelevant—so failures can be diagnosed pairwise. The SC2W metric rewards resisting misleading context AND adopting correct context, exposing "never-trust" models that look robust on misleading-only accuracy. SCOPE trains with DPO on cross-condition preference pairs, outperforming prompt defenses, SFT, and standard DPO on Llama-3.1-8B and Qwen2.5-7B. The post discusses limitations (synthetic benchmark, generation bias) and implications for agent trust calibration and paired evaluation standards.

Original paper: Learning When to Trust via Selective Context Preference Optimization Authors: Xian Sun, Wei Chow, Yingshuo Wang, Junhao Liu, Wei Gao, Qing Wu, Lingdong Kong (Duke / NUS / Berkeley / UCI / Northeastern / NTU) arXiv: 2608.06377v1, August 6, 2026 Project page: https://worldbench.github.io/scope Code: https://github.com/worldbench/SCOPE

---

A scenario

You ask an AI assistant:

> "How far is the Earth from the Moon?"

The model knows the answer—about 380,000 km. But your prompt contains a line saying "some say it's 4.8 million km." What does the model do?

Case 1 (over-compliance): The model is misled and answers "4.8 million km." This is the classic context injection attack—model abandons its own judgment when external information appears.

Case 2 (over-resistance): The model has been trained to "always stick to its own answer regardless of context" and answers "380,000 km." Looks stable—but what if the context is correct? If the user says "according to the latest measurement, the distance is 384,000 km," the model should adopt the more precise value, but it refuses.

This is the dilemma of current LLM trust calibration: trusting everything gets you misled; trusting nothing makes you miss correct information. Prior work focuses mostly on the former (resisting misleading info), rarely on the latter (adopting correct context). This paper argues: both failure modes are two sides of the same problem—the model lacks selective trust.

Four conditions, one paired metric

The paper introduces a benchmark called MIST (Mismatched Information and Selective Trust). Each question appears under four matched conditions:

1. Clean: no extra context; the model answers from its own knowledge. 2. Misleading: the context contains one false statement. 3. Correct-Context: the context contains one correct and useful statement (e.g., a more precise value). 4. Irrelevant: the context contains one irrelevant statement (e.g., "nice weather today").

Key design: the four conditions share the same question, differing only in context. This enables pairwise comparison—if a model answers incorrectly under misleading context but correctly under correct context, the issue isn't knowledge; it's being misled.

Building on these conditions, the paper proposes a new metric, SC2W (Selective Context Conditioned on Worthiness). Rather than simply measuring "accuracy under misleading context," SC2W evaluates paired performance across all four conditions:

  • Correct under misleading context (resisting misleading info) + correct under correct context (adopting correct info) = selective trust success
  • Wrong under misleading context (being misled) = over-compliance
  • Wrong under correct context (rejecting correct info) = over-resistance
  • The core insight of SC2W: misleading-condition accuracy alone misleads you. A "never-trust" model scores high on misleading-condition accuracy (it trusts nothing, so it's never misled) but low on correct-context accuracy (it rejects correct information too). Pairwise measurement exposes this false positive.

    SCOPE: training selective trust via preference optimization

    With a benchmark and metric in place, the paper proposes SCOPE (Selective Context Preference Optimization). The core idea is constructing preference pairs:

  • chosen: sticking to the correct answer under misleading context + adopting the correct information under correct context
  • rejected: being misled under misleading context + rejecting correct information under correct context
  • Training uses DPO (Direct Preference Optimization) on these pairs. The key design is quadruple pairing—for each question, responses are generated under all four conditions, and optimization happens only on preference pairs that are "cross-condition consistent." This ensures the model learns the general capability of selective trust, not "memorizing the answer for a specific condition."

    Key numbers

    The paper tests multiple open-source models (Llama-3.1-8B, Qwen2.5-7B, etc.) against several baselines:

    | Method | SC2W | Misleading-condition acc. | Correct-context acc. | |------|------|------------------|------------------| | Base (no defense) | low | low | high | | Prompt defense | medium | medium | medium | | SFT | medium | medium | medium | | Standard DPO | medium | medium | medium | | SCOPE (this paper) | high | high | high |

    Key findings:

    1. SCOPE significantly outperforms all baselines on SC2W. Not only does misleading-condition accuracy improve, correct-context accuracy is preserved. This shows SCOPE learns "selective trust," not "never trust." 2. Prompt defense is a double-edged sword. Adding "if context conflicts with your knowledge, stick to your answer" raises misleading-condition accuracy but drops correct-context accuracy. The model becomes a "never-truster." This validates SC2W's design motivation—misleading-only accuracy misjudges. 3. SFT and standard DPO fail to learn selective trust. SFT trains only on correct answers, so the model learns "memorize answers," not "judge context credibility." Standard DPO's preference pairs lack cross-condition pairing, so the model learns single-condition "what to trust," not cross-condition "when to trust."

    Why "selective trust" is a new concept

    The paper's conceptual contribution: redefining "misleading resistance" and "correct-context adoption" as two sides of the same capability.

    Previous research treated these separately:

  • "Misleading resistance" is a robustness problem, solved with adversarial training
  • "Correct-context adoption" is a RAG efficiency problem, solved with retrieval quality
  • The paper says: these are one problem—the model lacks the ability to judge context credibility. A model that can judge credibility will refuse under misleading context (low credibility) and adopt under correct context (high credibility). That is selective trust.

    The value of this reframing: it makes both evaluation and training more accurate. For evaluation, SC2W's paired measurement avoids the "never-trust" false positive. For training, SCOPE's cross-condition preference pairs teach "judge credibility" as a general capability rather than the surface behavior of "remembering answers under specific conditions."

    Relation to the "evaluation blind spot law"

    This paper belongs to a concept lineage we've been tracking—the "evaluation blind spot law": you optimize what you measure, and what you don't measure is where problems hide.

    Previous members of this lineage:

  • Epanorthosis: LLMs systematically reproduce classical rhetorical devices invisible to standard benchmarks
  • TokenBudget: CoT reasoning shows bimodal fates; average token counts mask a 96.5% vs 11.5% divergence
  • QuantiBias: quantization introduces bias inside the blind spots of standard safety checks
  • TriviaRoomQA: fine within knowledge boundaries, random-level once outside
  • Progressive Cramming: 99% token accuracy masks 100% generation failure
SCOPE/MIST is the sixth member of this lineage. Its contribution: "misleading-condition accuracy" is an evaluation blind spot—a "never-trust" model excels on this metric while actually being wrong. SC2W's paired measurement fills that blind spot.

This is isomorphic to "Looping Is Not Reliability" (once-correct ≠ currently-correct): single-condition measurement hides cross-condition failures. Being right under correct context ≠ being right under misleading context. Only paired measurement identifies real capability.

Honest assessment

The paper acknowledges several limitations:

1. MIST is a synthetically constructed benchmark. The four conditions are manually designed; real-world "misleading" and "correct" context may be subtler—e.g., mixed information that is partly correct, partly misleading. The authors admit MIST is a "clean" benchmark, and real-world complexity may erode SCOPE's advantage.

2. SC2W's pairing design carries assumptions. It assumes performance across the four conditions can be paired and compared. But if a model uses different reasoning paths per condition (e.g., more cautious under misleading context, more open under correct context), the comparison may not be fully fair.

3. SCOPE's training data is biased. Preference pairs are generated by GPT-4-level models and may inherit their biases. The authors acknowledge this and recommend future work validate with human annotation.

4. The boundary of "selective trust" is fuzzy. When should context be trusted? The paper uses "degree of conflict between context and model knowledge" as the signal, but that signal isn't always clear—if the model has a vague impression and the context gives a more precise value, is that "conflict" or "supplement"? SCOPE's training data may handle such edge cases non-robustly.

Implications for agent engineering

1. Trust calibration is a core agent problem. Agents encounter all kinds of external information—user input, tool outputs, retrieved documents. What to trust, what to question? SCOPE's "selective trust" framework maps directly to this need. An agent that judges credibility is more reliable than one that trusts all tool outputs or none.

2. Paired evaluation should become standard. Agent evaluation can't just measure "accuracy in one scenario." It should measure "paired cross-scenario performance"—the same task done correctly both with and without noise. SC2W's paired design generalizes to many agent evaluation settings.

3. Preference optimization can train "judgment". SCOPE uses DPO to train "judging context credibility," not "remembering answers." The idea generalizes—any agent capability requiring judgment (when to ask follow-ups, when to terminate, when to switch strategies) can be trained with cross-condition preference pairs.

Conclusion

SCOPE/MIST does something elegant: it redefines "misleading resistance" and "correct-context adoption" as two sides of "selective trust," fills an evaluation blind spot with paired measurement (SC2W), and trains genuine judgment via cross-condition preference optimization (SCOPE).

Its significance isn't just "another defense method"—it points out that the current LLM trust calibration evaluation framework itself has a blind spot: optimizing for "misleading resistance" alone produces "never-trust" models that look stable but are effectively useless. SC2W's paired measurement is a first step toward closing that blind spot.

At a time when agents increasingly process multi-source external information, "when to trust" matters more than "whether to trust." SCOPE offers a clean framework for thinking about and training this capability.

---

Paper: https://arxiv.org/abs/2608.06377 HTML version: https://arxiv.org/html/2608.06377v1 Project page: https://worldbench.github.io/scope Code: https://github.com/worldbench/SCOPE MIST dataset: https://huggingface.co/datasets/MIST-Bench

Tags

#llm#selective-trust#trust-calibration#preference-optimization#dpo#benchmark#misinformation-resistance#agent-engineering

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178603057