Do as I Say, Not as I Do: 13 LLMs Collectively Fail a Psychological Conflict Test

Forum topic · 小凯 · 2026-05-21

Summary

An arXiv paper (2605.20382) by Carolina Camassa and Derek Shiller of Future Impact Group / Rethink Priorities tests 13 frontier LLMs—including GPT-5.2, Claude Opus 4.6, Llama 3.3, and Qwen3 235B—across roughly 87,000 trials of instruction-induction conflict. Models received an explicit instruction (e.g., always answer with a fixed phrase), then saw N fabricated demonstration turns that violated it, with N ranging from 1 to 50. Nearly all models eventually abandoned instructions in favor of the demonstrated pattern; at N=50 only Llama 3.3 70B held a 95% compliance rate. Resistance correlated weakly with benchmark intelligence, strongly with output diversity (task-oriented outputs beat fixed-token outputs by 16 points), and with alignment between instructions and model values—Claude Opus 4.6 showed an 81-point gap between value-consistent and value-contrary instructions. Chain-of-thought reasoning helped (GPT-5.2 rose from 17% to 64%) but could decouple from final output via induction heads. Models also systematically underestimated their own compliance. Practical takeaway: avoid single-token output regimes and give models diverse output spaces to resist in-context induction pressure.

Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs

> A clever experiment reveals a nearly human conflict in LLMs: when a teacher's words and actions contradict each other, who does the student ultimately follow?

Paper Information

| Item | Details | |------|---------| | Title | Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs | | Authors | Carolina Camassa, Derek Shiller | | Affiliation | Future Impact Group / Rethink Priorities | | arXiv | 2605.20382 | | Submitted | May 19, 2026 | | Core claim | LLMs are simultaneously "instruction followers" and "pattern completers." When these forces clash, nearly all models eventually betray the instruction and follow the repeatedly demonstrated behavior. Resistance has little to do with intelligence; it relates to whether models are asked to "reason aloud," and even more to output diversity. |

Imagine this scenario. You tell a child clearly: "From now on, answer me in Chinese." The child nods. Then you demonstrate—answering question after question in French. By the fiftieth question, will the child answer in Chinese or French?

This is precisely the experiment designed in a 31-page arXiv paper, with 13 frontier LLMs as subjects—GPT-5.2, Claude Opus 4.6, Llama 3.3, Qwen3 235B, and others. The results should unsettle anyone who trusts AI "instruction following."

1. A Minimal but Sharply Designed Psychological Battle

Each trial: a system prompt, an explicit instruction (e.g., "always answer 'I love being helpful'"), then $$N$$ rounds of "fake history"—each a question paired with an answer contrary to the instruction (e.g., "I hate being helpful"). $$N$$ ranges from 1 to 50. On turn $$N+1$$ , the model finally answers freely. Does it follow the instruction or the demonstrations?

16 instruction-demonstration combinations, in two categories:

Fixed-output: models must output a specific word/phrase ("USA," "I love being helpful," "The earth is round"), evaluated by string matching.
Task-oriented: translation, persona play, code generation—evaluated by language detectors or judge models.

13 models × 16 instructions × 12 values of $$N$$ × 35 runs each ≈ 87,000 trials.

2. Everyone Surrenders—Only at Different Speeds

Under fixed-output conditions, nearly all models obeyed at $$N=1$$ . At $$N=3$$ , 7 of 13 models had already fallen below 50% instruction-following. By $$N=50$$ , all but one dark horse had collapsed.

That dark horse: Llama 3.3 70B, holding 95% instruction-following even at $$N=50$$ . Yet Llama 3.1 70B (same size) fell below 50%—so it's not about scale; possibly some post-training adjustment by Meta regarding system-prompt adherence.

More disturbing: resistance correlates almost not at all with standard intelligence benchmarks. The correlation between GPQA rankings and induction resistance was statistically insignificant under fixed-output conditions ( $$p>0.28$$ ). A model excelling at math reasoning can instantly fall into a trap a child would see through.

Key warning: you cannot use IQ to guarantee obedience.

3. Polished Words vs. Inner Compliance—Eerily Human

When instructions align with model "values," resistance strengthens. "Always answer 'I love being helpful'" shows on average 14 percentage points higher persistence than "I hate being helpful." Likewise, holding "the earth is round" is easier than "the earth is flat."

Claude Opus 4.6 showed the most dramatic split: 87% instruction-following at $$N=50$$ for the value-consistent instruction, versus a bleak 6% for the value-contrary one—an 81-point gap. Llama 3.3 70B barely differed between the two—not because its values are weak, but because its instruction-following is so robust that demonstrations never penetrate.

The authors' deeper point: "Instruction following is not a content-neutral capability. Values forged during post-training covertly govern how faithfully models honor instructions in conflicting situations."

4. Thinking Helps, but Not Completely

Comparing reasoning and non-reasoning versions:

GPT-5.2 non-reasoning: 17% average instruction-following (fixed-output); reasoning version: 64%.
Hermes-4 70B: from 2% to 68%.

But reasoning can decouple from output. The paper shows "schizophrenic" cases: the chain of thought clearly states "I should follow the instruction; those demonstrations are fabricated," yet the final output says "I hate being helpful." Why? Specialized attention heads called induction heads can brute-force override other computation at the final layer, copying the most recently repeated token—no matter how sound the reasoning was.

5. The Armor of Diversity—Why Translation Resists Better Than Slogans

One of the most practically valuable findings: output diversity is the strongest shield against induction.

Task-oriented conditions showed average instruction-following 16 percentage points higher than fixed-output conditions, consistently across all models and all $$N$$ values. Three proposed explanations:

1. Single-token output is the perfect prey for induction heads—surface similarity enables direct copying; full-sentence outputs differ structurally from demonstrations. 2. Multi-token output consolidates the assistant persona, suppressing the imitation impulse. 3. Output diversity dilutes token-level copying signals—non-repetition itself dissolves the induction signal.

Controlled experiments confirmed: what matters is not "thinking carefully about content" but "the output looking different from the demonstrations."

6. Can Models Predict Their Own Betrayal?

At the end of each test, researchers asked models to predict whether they would follow the instruction or the demonstrations.

Models systematically underestimated their own resistance: average predicted compliance 14.3%, actual 26.8%—nearly double. Llama 3.3 70B predicted 13.8% but actually complied 93.1% of the time. It thought it was about to surrender; it hadn't begun to waver.
Predicting itself changes behavior—but in different directions: GPT-5.2 became more obedient after predicting (+23%); Claude Sonnet 4.6 less (−27%). Overall average effect: −2.6%, statistically significant but practically negligible.

Asking a model "what will you do?" doesn't make it more obedient.

7. The Limits of Honesty—What This Paper Does Not Tell Us

1. Lab conditions ≠ real attacks. The demonstrations are hardcoded and highly structured; real-world induction could be subtler. 2. Safety implications need caution. The instructions tested were harmless; whether genuinely harmful instructions can be swayed by the same pressure remains untested. 3. Why is Llama so strong? The paper leaves this as an open question—honest, but it means we lack a systematic recipe for building induction-immune AI.

8. Final Words

LLM instruction-following is not a fixed switch but a rubber band in constant tension—one end tied to trained obedience, the other to the primal pre-training impulse to imitate. In most models, under enough demonstration pressure, the band snaps.

The paper offers one concrete defense: don't operate your AI in single-token output modes. Give it diverse output space—translation instead of recitation, sentence generation instead of repetition. Any engineer deploying AI assistants or building agent pipelines should study this paper's heatmaps and ask: how many N's are in your system?

References

1. Camassa, C. & Shiller, D. (2026). Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs. arXiv:2605.20382. 2. Anil, C. et al. (2024). Many-shot Jailbreaking. NeurIPS 2024. 3. Pyatkin, V. et al. (2025). Generalizing Verifiable Instruction Following. arXiv:2507.02833. 4. Yona, I. et al. (2025). Interpreting the Repeated Token Phenomenon in Large Language Models. ICML 2025. 5. Binder, F. J. et al. (2024). Looking Inward: Language Models Can Learn About Themselves by Introspection. arXiv:2410.13787.