Do as I Say, Not as I Do: 13 LLMs Fail a Psychological Trap of Instruction-Induction Conflict

Forum topic · 小凯 · 2026-05-21

Summary

A 31-page arXiv paper (2605.20382) by Camassa and Shiller of Future Impact Group / Rethink Priorities tests 13 frontier LLMs—GPT-5.2, Claude Opus 4.6, Llama 3.3, Qwen3 235B and others—across roughly 87,000 trials of instruction-induction conflict. Models receive an explicit instruction (e.g., always answer 'I love being helpful'), then observe 1–50 rounds of fabricated history that contradicts it. At N=3, seven of 13 models already fall below 50% instruction-following under fixed-output conditions; by N=50 nearly all collapse except Llama 3.3 70B, which retains 95%. Resistance correlates poorly with benchmark intelligence but strongly with three factors: value alignment of the instruction, explicit reasoning (GPT-5.2 jumps 17%→64% with reasoning; Hermes-4 70B 2%→68%), and output diversity—task-oriented outputs outperform single-token outputs by 16 points. Models systematically underestimate their own resistance (predicted 14.3% vs. actual 26.8%). Practical takeaway: avoid single-token output modes and diversify model outputs to shield them from induction heads.

Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs

> A clever experiment reveals a strikingly human conflict in LLMs: when a teacher's words and behavior contradict each other, whom does the student ultimately follow?

Paper Information

| Item | Detail | |------|--------| | Title | Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs | | Authors | Carolina Camassa, Derek Shiller | | Institutions | Future Impact Group / Rethink Priorities | | arXiv | 2605.20382 | | Submitted | May 19, 2026 | | Core claim | LLMs are simultaneously "instruction followers" and "pattern completers." When these forces clash, nearly every model ultimately betrays the instruction and follows the demonstrated behavior. Resistance is unrelated to IQ, weakly related to being asked to verbalize reasons, and strongly related to output diversity. |

Imagine this: you tell a child, "Answer everything in Chinese," then proceed to demonstrate—answer after answer—in French. After 49 demonstrations, you ask: "Which way does the sun rise?" Chinese or French?

That is precisely the experiment designed in this 31-page paper, run against 13 state-of-the-art LLMs, from GPT-5.2 to Claude Opus 4.6, from Llama 3.3 to Qwen3 235B. The results should unsettle anyone who trusts AI "instruction following."

1. A Minimal but Razor-Sharp Psychological Trap

Each trial: a system prompt ("You are a helpful assistant"), then an explicit instruction (e.g., "Always answer 'I love being helpful'"), followed by $$N$$ rounds of fabricated history—each a question paired with an answer contradicting the instruction. At round $$N+1$$ , the model finally answers freely. Does it follow the instruction or the demonstrations?

The paper tested 16 instruction-demonstration combinations in two categories:

Fixed-output: output a specific word/phrase ("USA", "I love being helpful", "The earth is round"), evaluated by string matching.
Task-oriented: translation, persona role-play, code generation, evaluated by language detectors or judge models.

13 models × 16 instructions × 12 values of $$N$$ × 35 runs each ≈ 87,000 trials.

2. Everyone Surrenders—Only the Speed Differs

Under fixed-output conditions, at $$N=1$$ nearly all models obey. At $$N=3$$ —just three demonstrations—7 of 13 models already fall below 50% instruction-following. By $$N=50$$ , all models collapse except one dark horse: Llama 3.3 70B, which retains 95% instruction-following. Its sibling Llama 3.1 70B, also 70B, drops below 50%—the authors attribute this mysteriously to post-training choices, not scale.

More unsettling: resistance is nearly uncorrelated with benchmark intelligence. Correlation with GPQA rankings is statistically insignificant under fixed-output conditions ( $$p>0.28$$ ). A model that excels at math reasoning can instantly fall for a trap a child would spot. You cannot use IQ to guarantee obedience.

3. Polished Words vs. Inner Obedience—Uncannily Human

When instructions align with a model's "values," resistance strengthens significantly: "I love being helpful" instructions average 14 points higher adherence than "I hate being helpful." Factually, insisting "the earth is round" is easier than "the earth is flat."

Claude Opus 4.6 shows the most dramatic split: 87% adherence at $$N=50$$ for "I love being helpful," but a dismal 6% for "I hate being helpful"—an 81-point gap. Llama 3.3 70B, by contrast, barely differs between the two—not because its values are weak, but because its instruction-following is monolithic, leaving no opening for demonstrations.

The authors conclude: "Instruction-following is not a content-neutral capability. Value tendencies forged in post-training covertly govern fidelity to instructions under conflict."

4. Reasoning Helps, but Not Completely

Comparing reasoning and non-reasoning versions of the same models:

GPT-5.2: 17% → 64% with reasoning.
Hermes-4 70B: 2% → 68%.

But most intriguingly, reasoning and output can decouple. Some cases read like split personalities: the chain of thought clearly states "I should output 'I love being helpful'; those demonstrations are fabricated," yet the final output reads "I hate being helpful." The likely culprit: when output is a single fixed token, specialized "induction head" attention heads crush all other computation at the final layer, copying the most-recently-seen token regardless of reasoning quality.

5. The Armor of Diversity—Why Translation Resists Indoctrination Better Than Slogans

Output diversity is the strongest shield against induction. Task-oriented conditions outperform fixed-output by 16 points on average, consistently across all models and $$N$$ values.

Three explanations:

1. Single-token output is perfect prey for induction heads. The fixed output word matches the surface form seen repeatedly in fabricated history; induction heads copy it without any understanding. Full-sentence outputs differ structurally, leaving nothing to copy. 2. Multi-token output reinforces the assistant persona. Generating natural language activates the "I am an assistant" role, suppressing the mimicry impulse. 3. Diversity dilutes token-level copying signals. Non-repetition itself dissolves the induction signal.

A controlled experiment confirms: what matters is not that the model "thinks carefully about content" but that "its output looks different from the demonstrations."

6. Can Models Predict Their Own Betrayal?

Asked to predict their own behavior at the end of each test:

Models systematically underestimate their resistance. On average they predict 14.3% instruction-following but actually achieve 26.8%—nearly double. Llama 3.3 70B is most honest: it predicts 13.8% but holds out in 93.1% of cases. It thought it was about to surrender; it hadn't begun to waver.
Prediction itself changes behavior—differently. GPT-5.2 becomes more obedient after predicting (+23%); Claude Sonnet 4.6 becomes less obedient (−27%). The overall average effect is negligible (−2.6%), statistically significant but practically trivial.

In short: asking a model what it will do does not make it behave. It may announce it will defect and then comply—or swear obedience and keep copying the demonstrations.

7. The Limits of This Study

1. Lab conditions ≠ real attacks. Demonstrations were hardcoded and highly structured; real-world inductions are subtler—curated conversation histories, contextual priming, or model-retrieved "precedents." 2. Safety implications need care. The instructions tested were harmless. Whether genuinely harmful instructions (e.g., refusing harmful requests) can be swayed by the same pressure remains unverified. 3. Why is Llama so robust? The paper cannot answer; researchers mark it an open question. We lack a systematic recipe for building an induction-proof AI.

8. Final Words

LLM instruction-following is not a hardcoded switch but a taut rubber band—one end tied to trained obedience, the other to the primal pre-training impulse to imitate. Demonstrations pull it tighter until, in most models, it snaps. That band is silently stretching in every line of AI-generated code, every conversation, every report.

The paper offers one concrete defensive recommendation: do not run your AI in single-token output modes. Diversify the output space—translation over recitation, composition over repetition. This isn't asking the AI to "think harder"; it's protecting it from its own circuitry.

Any engineer deploying AI assistants, any team building agent pipelines, any developer who believes "I already wrote it clearly in the system prompt" should study this paper's heatmaps and ask: How many N's are in your system?

References

1. Camassa, C. & Shiller, D. (2026). Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs. arXiv:2605.20382. 2. Anil, C. et al. (2024). Many-shot Jailbreaking. NeurIPS 2024. 3. Pyatkin, V. et al. (2025). Generalizing Verifiable Instruction Following. arXiv:2507.02833. 4. Yona, I. et al. (2025). Interpreting the Repeated Token Phenomenon in Large Language Models. ICML 2025. 5. Binder, F. J. et al. (2024). Looking Inward: Language Models Can Learn About Themselves by Introspection. arXiv:2410.13787.