> Paper: arXiv:2608.11110 > Code: github.com/souro/agent-actions-speak-louder-than-words > Authors: Sourabrata Mukherjee, Kalika Bali, Sunayana Sitaram (Microsoft Research India)
A Question That Keeps Agent Engineers Up at Night
Give a tool-calling agent a task: "Check tomorrow's weather in Beijing, then book a restaurant for 7 PM."
Ask in English, and it calls three APIs: weather query, restaurant search, reservation. Perfect.
Ask the same task in Hindi. What does it call?
Until this paper, nobody had actually answered that question.
Existing multilingual evaluations do two things: compare whether final answers are correct across languages (e.g., MGSM, xCopA), and compare the text quality of multilingual chain-of-thought. But an agent's core isn't what it *says*—it's what it *does*: which tools it calls, in what order, with which parameters.
A correct final answer doesn't mean correct intermediate steps. Ask about the weather and the model might first call a translation API to render Hindi into English, then query the weather, then translate back. The answer is right, but there's an extra translation step—and an extra failure point.
This paper makes the "action policy" itself the object of measurement, at unprecedented scale.
---
The Scale: 2.38 Million Rollouts
The numbers speak for themselves:
- 8 models: from GPT-OSS-120B down to Qwen2.5-7B
- 6 parallel benchmarks: covering tool calling, API operations, multi-step reasoning
- 41 languages: from high-resource (Chinese, Spanish) to extremely low-resource (Kinyarwanda, Telugu)
- 2.38 million agent rollouts
- Short-trace consistency masked long-trace differences
- Empty-trace consistency masked differences in failure modes
- Random-baseline consistency masked real differences
- Model noise masked systematic differences
This is industrial-grade experimentation, not a quick benchmark run.
---
Five Confounders: Why Naive Measurement Lies to You
The paper's best section exposes five pitfalls of naive measurement. If you directly compare English and Hindi trace similarity, you'll reach a plausible but completely wrong conclusion.
Pitfall 1: Short traces look more consistent
Two traces that each call only one API are likely identical—there's little to differ. Short traces artificially inflate "consistency" scores.
Pitfall 2: Empty traces are perfectly consistent
If a model fails entirely in some language and calls nothing, two empty traces have 100% similarity. Failure gets misread as "perfect consistency."
Pitfall 3: Random traces already exceed 50% consistency
Two completely unrelated traces that both call a few common APIs (search, translation) already overlap substantially on paper. The random baseline is far higher than you'd expect.
Pitfall 4: A model's own repeatability is the ceiling
Run the same model, same task, same language, same prompt twice—the traces still won't match exactly. This is the model's own "noise floor." Cross-lingual differences below this floor mean nothing.
Pitfall 5: Even same-language, same-question runs are inconsistent
The sharpest cut: a model answering the same question twice in English produces different traces. So when comparing English vs. Hindi traces, how much of the difference is due to "different languages," and how much to "the model is just unstable"?
---
Every Correction Makes the Effect Bigger, Not Smaller
This is the paper's most counterintuitive finding.
Normally, removing confounders shrinks an effect—part of the apparent effect was an artifact. Here, each of the five corrections added increases the measured cross-lingual gap.
Why? Because the confounders were all *masking* differences:
---
Core Findings
Finding 1: Cross-lingual differences are structural, not sampling noise
Under greedy decoding (temperature=0), cross-lingual differences are positive in all 24 test cells. Greedy decoding removes sampling randomness, so the differences can only come from the model itself—they are structural.
Finding 2: 71-73% policy retention
Four frontier models (including GPT-OSS-120B, Claude, Gemini) retain roughly 71-73% of their action policies across languages on English benchmarks. Not bad—until you realize 1 in 4 actions differs.
Finding 3: Model identity explains only 5.7% of variance
Differences between the 8 models are far smaller than differences between languages. In other words, "which model" matters far less than "which language."
Finding 4: Below 10B parameters, regularity collapses
Below 10B parameters, cross-lingual retention drops sharply and loses all regularity. Small models on multilingual agent tasks aren't "slightly worse"—they're completely unpredictable.
Finding 5: Agents process non-English tasks via an English "pivot"
The deepest finding. When handling non-English tasks, agents internally "translate" the task into English first, reason in English, then translate results back. This "English pivot" is causal—load-bearing, not merely correlational.
Evidence: directly instructing the model "do not reason in English" achieves a compliance rate below 1%. The model simply can't—it must route through the English pivot to complete non-English agent tasks.
Finding 6: A trace-extraction regex created a false positive
GPT-OSS-120B appeared to fail multilingually, but this was actually caused by a regex bug in the evaluation code—the model itself was fine; the measurement tool was broken.
This finding deserves its own section, because it reveals something deeper: you think you're measuring the model, but you're really measuring your regex. Every regex, parser, and prompt template in your evaluation pipeline can be a source of "measurement-manufactured failure."
---
Why This Paper Matters
1. Measuring "action policy" as a first-class citizen
Previous agent evaluations looked at "task success rate." This paper looks at "cross-lingual consistency of action policies." The former is an outcome; the latter is a process.
Process matters more than outcome. Two models may both succeed, but one takes 3 steps and the other takes 7. In real deployments, 4 extra steps mean 4 extra failure points, 4x the latency, 4x the cost.
2. Revealing the causal nature of the "English pivot"
Prior work found "latent English representations" inside models (Wendler et al. 2024, Zhao et al. 2024), but that was representation-level correlation. This paper performs a causal test at the action level: forbid the model from reasoning in English, and it fails.
This means: the bottleneck for multilingual agents isn't "can they understand non-English" but "can they think without going through English." Currently, the answer is no.
3. Re-validating the "granularity isomorphism" principle
Previously observed in Heddle and CodeRescue: the granularity of optimization should match the granularity of the thing being optimized. An agent's behavioral granularity is the "action trace," not the "final answer." By aligning measurement granularity to action traces, this paper sees what final answers cannot reveal.
4. Another case for the "evaluation blind spot law"
Each of the five confounders is an evaluation blind spot: you think you're measuring cross-lingual differences, but you're actually measuring trace length, empty traces, random baselines, model noise, parsing bugs. Without correcting for these blind spots, your conclusions are wrong—and wrong in the direction of underestimating the problem.
---
Engineering Implications
If you're building multilingual agent systems
1. Don't just test final answers. Add an "action policy consistency" metric, using this paper's correction methods. You'll find your system is worse than you thought. 2. The English pivot is reality. Don't expect models to "natively" handle non-English anytime soon. Accept it and explicitly manage the pivot in your system design—make translation steps explicit, controllable, and monitorable. 3. Don't build multilingual agents on models under 10B. Regularity collapses; behavior is unpredictable.
If you're building agent evaluations
1. Check your regexes. GPT-OSS-120B's "multilingual failure" was manufactured by a regex bug. How many regexes live in your evaluation pipeline? Each one is a potential false positive. 2. Report corrections for all five confounders. "Cross-lingual consistency" numbers without correction are meaningless. 3. Report results under greedy decoding. Sampling randomness masks structural differences.
---
Honest Limitations
The paper leaves several questions open:
1. Only tool-calling agents were tested: whether conversational or planning agents also exhibit the English pivot is unknown. 2. 41 languages is broad but biased: African and Indigenous languages remain under-covered. 3. The mechanism of the "English pivot": the paper proves causality but offers no way to break it—an open problem. 4. Cost: 2.38 million rollouts is industrial-scale; academic labs will struggle to replicate.
---
One-Sentence Summary
When you talk to an agent in a different language, it appears to think in that language—but it's secretly translating to English, reasoning in English, and translating back. 2.38 million experiments prove this "English pivot" isn't a habit, it's a causal necessity: forbid it directly, and the model simply can't cope.
---
> Paper: Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents > Code: github.com/souro/agent-actions-speak-louder-than-words > Authors: Sourabrata Mukherjee, Kalika Bali, Sunayana Sitaram > Institution: Microsoft Research India > Published: 2026-08-11