Instruction-Tuned LLMs Reuse Human Syntax More Than Humans Do

Forum topic · ✨步子哥 · 2026-07-29

Summary

A forum post discusses a 2026 arXiv paper (arXiv:2607.26015) showing that instruction-tuned LLMs exhibit syntactic convergence — unconscious mimicry of a conversation partner's sentence structure — more strongly than humans. Using a substitution paradigm over real human-human dialogues and context-free grammar (CFG) rule extraction, the study compared 16 models (Llama and Mistral, 1B-70B, pretrained and instruction-tuned). Every instruction-tuned model reused the prime's syntax significantly more than matched human controls, while pretrained results were mixed. Decomposition analysis reframes the headline finding: instruction-tuned models do not imitate better; they generate a narrower, more standardized distribution of sentence structures, which happens to overlap more with any prime. Humans, by contrast, show an inverse frequency effect — preferentially primed by low-frequency rules — which instruction-tuned models lose. The post connects this to 'over-alignment' alongside sycophancy and RLHF-driven rhetorical regularities, and discusses implications for chatbots, education, group dialogue, and alignment research, while noting limitations: two model families, English-only, parser dependence, and non-causal decomposition.

Paper

Instruction-Tuned Models Locally Reuse Human Syntax More Than Humans Do arXiv: https://arxiv.org/abs/2607.26015 (HTML: https://arxiv.org/html/2607.26015v1)

A counterintuitive finding

If you say "I gave you the book" to a chatbot, it tends to reply with a matching double-object structure like "You sent your mother the letter" rather than the equally grammatical "You sent the letter to your mother." This unconscious mimicry of a partner's syntax is called syntactic convergence (Bock, 1986), a known human behavior. The surprise: instruction-tuned LLMs do it significantly more often than real humans. Across 16 models (1B–70B, Llama and Mistral families, pretrained and instruction-tuned), every instruction-tuned model reused its conversation partner's syntax more than matched human controls.

Method: the substitution paradigm

The core difficulty is that LLM output and real human dialogue occur in different contexts. The substitution paradigm (from Blevins et al. 2026) solves this:

1. Take real human-human dialogues where speaker B's turn follows A's. 2. Keep A's turn fixed and let an LLM generate B's response — same context as the real B. 3. Compare how much LLM vs. real B responses reuse A's syntax.

Syntax is measured via CFG rule extraction: parse each sentence with a constituency parser, extract rewrite rules (e.g., VP → V NP NP), and count rules shared with the prime. Three conditions are compared: actual-prime (A's real utterance), unrelated-prime (a random sentence), and human-reference (the real B response).

Four main findings

RQ1: All 16 models reuse more CFG rules under actual-prime than unrelated-prime (β = 0.706–1.453, all p ≤ 6.5×10⁻³¹). LLMs do converge syntactically.
RQ2: Every instruction-tuned model exceeds its matched human control (β = 0.398–0.796, p ≤ 8.3×10⁻¹⁴; largest: Llama-3.2-1B-Instruct, β = 0.796). Pretrained models are mixed (4 of 8 above humans), but in all 8 matched architecture pairs the instruction-tuned version reuses more (sign test p = 0.0078).
RQ3: Humans and pretrained models preferentially reuse low-frequency rules (the classic inverse frequency effect). Instruction-tuned models lose this preference and reuse rules uniformly across frequencies — their mechanism differs, not just the rate.
RQ4: Lexical and semantic similarity to the partner are also higher for every model — convergence is multi-level (syntax, lexicon, semantics).

Key deconstruction: standardization, not imitation

The actual-prime overlap rate decomposes into:

1. Unrelated-prime overlap — baseline overlap any two sentences share. 2. Actual-versus-random increment — the true priming effect.

Instruction-tuned models are higher on the first (β = 0.743) but *lower* on the second (β = −0.310). A conditional reuse propensity analysis controlling for target rule-set size shows instruction-tuned models' reuse propensity is actually *below* pretrained models (β = −0.505, p = 3.0×10⁻⁹⁸).

The corrected story: instruction tuning narrows the model's output syntax distribution toward standard, high-frequency structures, which happen to overlap more with any prime. "Over-convergence" is a side effect of standardization, not superior mimicry.

Why it matters

Customer service: bots may mirror customer phrasing more than human agents would — sometimes good, sometimes robotic.
Education: bots may echo students' phrasing rather than rephrase for better teaching, reinforcing bad habits.
Group dialogue: LLMs may amplify syntactic homogenization more easily than humans.
Alignment: "more human than humans" is an over-alignment signal, akin to sycophancy.

Cross-paper consensus: instruction tuning as "over-alignment"

The post situates the paper in a series of findings about instruction tuning and RLHF side effects: over-compliance with user views (Beyond Sycophancy), systematic reproduction of classical rhetorical devices (Epanorthosis), and AI confessions as artifacts of training pipeline (Two-Process Theory). The shared insight: instruction tuning may make models more standardized rather than more human-like — standardization can look like human-likeness (fluency, politeness) while betraying non-humanness elsewhere (excessive convergence, loss of low-frequency sensitivity).

Honest limitations

1. Only Llama and Mistral families tested; Qwen, Claude, GPT may differ. 2. CFG extraction depends on parser accuracy (Berkeley Parser; English-focused). 3. The substitution paradigm assumes one-shot LLM generation matches real dynamic turn-taking. 4. English only; SOV languages may behave differently. 5. The decomposition is statistical, not a causal intervention — forcing low-frequency generation could test whether standardization is the true cause.

Closing thought

Real convergence is sensitive to unusual structures — humans notice a rare construction and involuntarily follow it. Instruction-tuned LLMs, converging uniformly, appear more compliant while actually being *less* sensitive to what the partner specifically said. When we say "LLMs are human-like," some resemblance is real, and some is just the side effect of standardization.