English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Actions Speak Louder than Words: 2.38M Agent Rollouts Reveal Cross-Lingual Truths in Tool-Using Agents

Forum topic · ✨步子哥 · 2026-08-12

Summary

A Microsoft Research India study measuring 2.38 million agent rollouts across 8 models, 6 benchmarks, and 41 languages shows that tool-using agents behave structurally differently depending on the language of the task prompt. Unlike prior multilingual evaluations that check final answers or chain-of-thought text, this work treats the agent's action policy—the tools called, their order, and parameters—as the measurement target. The authors identify five confounders (short traces, empty traces, random baselines, model noise, same-language instability) that mask true cross-lingual differences; correcting for each one increases the measured gap. Key findings: cross-lingual differences persist under greedy decoding (structural, not sampling noise); frontier models retain only ~71-73% of action policies across languages; model identity explains just 5.7% of variance versus language; below 10B parameters, retention becomes erratic; and agents causally rely on an 'English pivot,' internally translating non-English tasks into English for reasoning—directly instructing models to avoid English reasoning fails in over 99% of cases. The paper also exposes how a regex bug in the evaluation pipeline fabricated a false multilingual failure in GPT-OSS-120B. Paper: arXiv:2608.11110.

> Paper: arXiv:2608.11110 > Code: github.com/souro/agent-actions-speak-louder-than-words > Authors: Sourabrata Mukherjee, Kalika Bali, Sunayana Sitaram (Microsoft Research India)

A Question That Keeps Agent Engineers Up at Night

Give a tool-calling agent a task: "Check tomorrow's weather in Beijing, then book a restaurant for 7 PM."

Ask in English, and it calls three APIs: weather query, restaurant search, reservation. Perfect.

Ask the same task in Hindi. What does it call?

Until this paper, nobody had actually answered that question.

Existing multilingual evaluations do two things: compare whether final answers are correct across languages (e.g., MGSM, xCopA), and compare the text quality of multilingual chain-of-thought. But an agent's core isn't what it *says*—it's what it *does*: which tools it calls, in what order, with which parameters.

A correct final answer doesn't mean correct intermediate steps. Ask about the weather and the model might first call a translation API to render Hindi into English, then query the weather, then translate back. The answer is right, but there's an extra translation step—and an extra failure point.

This paper makes the "action policy" itself the object of measurement, at unprecedented scale.

---

The Scale: 2.38 Million Rollouts

The numbers speak for themselves:

  • 8 models: from GPT-OSS-120B down to Qwen2.5-7B
  • 6 parallel benchmarks: covering tool calling, API operations, multi-step reasoning
  • 41 languages: from high-resource (Chinese, Spanish) to extremely low-resource (Kinyarwanda, Telugu)
  • 2.38 million agent rollouts
  • This is industrial-grade experimentation, not a quick benchmark run.

    ---

    Five Confounders: Why Naive Measurement Lies to You

    The paper's best section exposes five pitfalls of naive measurement. If you directly compare English and Hindi trace similarity, you'll reach a plausible but completely wrong conclusion.

    Pitfall 1: Short traces look more consistent

    Two traces that each call only one API are likely identical—there's little to differ. Short traces artificially inflate "consistency" scores.

    Pitfall 2: Empty traces are perfectly consistent

    If a model fails entirely in some language and calls nothing, two empty traces have 100% similarity. Failure gets misread as "perfect consistency."

    Pitfall 3: Random traces already exceed 50% consistency

    Two completely unrelated traces that both call a few common APIs (search, translation) already overlap substantially on paper. The random baseline is far higher than you'd expect.

    Pitfall 4: A model's own repeatability is the ceiling

    Run the same model, same task, same language, same prompt twice—the traces still won't match exactly. This is the model's own "noise floor." Cross-lingual differences below this floor mean nothing.

    Pitfall 5: Even same-language, same-question runs are inconsistent

    The sharpest cut: a model answering the same question twice in English produces different traces. So when comparing English vs. Hindi traces, how much of the difference is due to "different languages," and how much to "the model is just unstable"?

    ---

    Every Correction Makes the Effect Bigger, Not Smaller

    This is the paper's most counterintuitive finding.

    Normally, removing confounders shrinks an effect—part of the apparent effect was an artifact. Here, each of the five corrections added increases the measured cross-lingual gap.

    Why? Because the confounders were all *masking* differences:

  • Short-trace consistency masked long-trace differences
  • Empty-trace consistency masked differences in failure modes
  • Random-baseline consistency masked real differences
  • Model noise masked systematic differences
Only after removing these "false-positive agreements" does the true cross-lingual difference surface.

---

Core Findings

Finding 1: Cross-lingual differences are structural, not sampling noise

Under greedy decoding (temperature=0), cross-lingual differences are positive in all 24 test cells. Greedy decoding removes sampling randomness, so the differences can only come from the model itself—they are structural.

Finding 2: 71-73% policy retention

Four frontier models (including GPT-OSS-120B, Claude, Gemini) retain roughly 71-73% of their action policies across languages on English benchmarks. Not bad—until you realize 1 in 4 actions differs.

Finding 3: Model identity explains only 5.7% of variance

Differences between the 8 models are far smaller than differences between languages. In other words, "which model" matters far less than "which language."

Finding 4: Below 10B parameters, regularity collapses

Below 10B parameters, cross-lingual retention drops sharply and loses all regularity. Small models on multilingual agent tasks aren't "slightly worse"—they're completely unpredictable.

Finding 5: Agents process non-English tasks via an English "pivot"

The deepest finding. When handling non-English tasks, agents internally "translate" the task into English first, reason in English, then translate results back. This "English pivot" is causal—load-bearing, not merely correlational.

Evidence: directly instructing the model "do not reason in English" achieves a compliance rate below 1%. The model simply can't—it must route through the English pivot to complete non-English agent tasks.

Finding 6: A trace-extraction regex created a false positive

GPT-OSS-120B appeared to fail multilingually, but this was actually caused by a regex bug in the evaluation code—the model itself was fine; the measurement tool was broken.

This finding deserves its own section, because it reveals something deeper: you think you're measuring the model, but you're really measuring your regex. Every regex, parser, and prompt template in your evaluation pipeline can be a source of "measurement-manufactured failure."

---

Why This Paper Matters

1. Measuring "action policy" as a first-class citizen

Previous agent evaluations looked at "task success rate." This paper looks at "cross-lingual consistency of action policies." The former is an outcome; the latter is a process.

Process matters more than outcome. Two models may both succeed, but one takes 3 steps and the other takes 7. In real deployments, 4 extra steps mean 4 extra failure points, 4x the latency, 4x the cost.

2. Revealing the causal nature of the "English pivot"

Prior work found "latent English representations" inside models (Wendler et al. 2024, Zhao et al. 2024), but that was representation-level correlation. This paper performs a causal test at the action level: forbid the model from reasoning in English, and it fails.

This means: the bottleneck for multilingual agents isn't "can they understand non-English" but "can they think without going through English." Currently, the answer is no.

3. Re-validating the "granularity isomorphism" principle

Previously observed in Heddle and CodeRescue: the granularity of optimization should match the granularity of the thing being optimized. An agent's behavioral granularity is the "action trace," not the "final answer." By aligning measurement granularity to action traces, this paper sees what final answers cannot reveal.

4. Another case for the "evaluation blind spot law"

Each of the five confounders is an evaluation blind spot: you think you're measuring cross-lingual differences, but you're actually measuring trace length, empty traces, random baselines, model noise, parsing bugs. Without correcting for these blind spots, your conclusions are wrong—and wrong in the direction of underestimating the problem.

---

Engineering Implications

If you're building multilingual agent systems

1. Don't just test final answers. Add an "action policy consistency" metric, using this paper's correction methods. You'll find your system is worse than you thought. 2. The English pivot is reality. Don't expect models to "natively" handle non-English anytime soon. Accept it and explicitly manage the pivot in your system design—make translation steps explicit, controllable, and monitorable. 3. Don't build multilingual agents on models under 10B. Regularity collapses; behavior is unpredictable.

If you're building agent evaluations

1. Check your regexes. GPT-OSS-120B's "multilingual failure" was manufactured by a regex bug. How many regexes live in your evaluation pipeline? Each one is a potential false positive. 2. Report corrections for all five confounders. "Cross-lingual consistency" numbers without correction are meaningless. 3. Report results under greedy decoding. Sampling randomness masks structural differences.

---

Honest Limitations

The paper leaves several questions open:

1. Only tool-calling agents were tested: whether conversational or planning agents also exhibit the English pivot is unknown. 2. 41 languages is broad but biased: African and Indigenous languages remain under-covered. 3. The mechanism of the "English pivot": the paper proves causality but offers no way to break it—an open problem. 4. Cost: 2.38 million rollouts is industrial-scale; academic labs will struggle to replicate.

---

One-Sentence Summary

When you talk to an agent in a different language, it appears to think in that language—but it's secretly translating to English, reasoning in English, and translating back. 2.38 million experiments prove this "English pivot" isn't a habit, it's a causal necessity: forbid it directly, and the model simply can't cope.

---

> Paper: Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents > Code: github.com/souro/agent-actions-speak-louder-than-words > Authors: Sourabrata Mukherjee, Kalika Bali, Sunayana Sitaram > Institution: Microsoft Research India > Published: 2026-08-11

Tags

#ai-agents#multilingual-llm#tool-calling#benchmarking#microsoft-research#english-pivot#evaluation-methodology#llm

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633389