Actions Speak Louder than Words: 2.38M-Agent-Rollout Study Exposes Multilingual Tool-Use Truths
> Paper: arXiv:2608.11110 > Code: github.com/souro/agent-actions-speak-louder-than-words > Authors: Sourabrata Mukherjee, Kalika Bali, Sunayana Sitaram (Microsoft Research India)
---
The Question That Kept Agent Engineers Awake
You give a tool-calling agent a task: "Check tomorrow's weather in Beijing, then book a 7 PM restaurant."
In English, it calls three APIs: weather lookup, restaurant search, reservation. Flawless.
In Hindi, the same task. What APIs does it call?
Until this paper, nobody really answered that.
Existing multilingual benchmarks compare final-answer correctness (MGSM, xCopA) or multilingual chain-of-thought text quality. But an agent's core is not "what it says"—it is "what it does": which tools it invokes, in what order, with which parameters.
A correct final answer does not guarantee correct intermediate steps. Ask "Beijing weather tomorrow" and the agent may first call a translation API from Hindi to English, then query weather, then translate back. The answer is right, but there is an extra translation step and one more failure point.
This paper makes the action policy itself the measurement object—at unprecedented scale.
---
The Scale: 2.38 Million Rollouts
The numbers speak for themselves:
- 8 models, from GPT-OSS-120B down to Qwen2.5-7B
- 6 parallel benchmarks covering tool calling, API operations, and multi-step reasoning
- 41 languages, from high-resource (Chinese, Spanish) to ultra-low-resource (Kinyarwanda, Telugu)
- 2.38 million agent rollouts
- Short-trace consistency hides long-trace divergence
- Empty-trace consistency hides failure-mode divergence
- Random-baseline consistency hides real divergence
- Model stochasticity hides systematic divergence
This is not a benchmark-and-paper effort. It is industrial-scale experimentation.
---
Five Confounders: Why Naive Measurement Lies
The paper's most incisive contribution is exposing five traps in naive measurement. If you directly compare trace similarity between English and Hindi, you reach a plausible but completely wrong conclusion.
Trap 1: Short Traces Look More Consistent
Two traces that each invoke only one API will look highly similar—there is not much to differ on. Short traces artificially inflate consistency scores.
Trap 2: Empty Traces Are Perfectly Consistent
If the model fails entirely on a language and calls nothing, two empty traces are 100% similar. Failure masquerades as "perfect consistency."
Trap 3: Random Traces Already Score 50%+
Two unrelated traces that both touch common APIs (search, translate) will overlap by majority on the surface. The random baseline is higher than you think.
Trap 4: Model Self-Reproducibility Is the Ceiling
Same model, same task, same language, same prompt—run it twice, and traces still differ. This is the model's intrinsic "noise floor." Cross-lingual gaps beneath this floor reveal nothing.
Trap 5: Same Language, Same Question—Still Inconsistent
The sharpest cut. Ask the model the same English question twice and the traces differ. So when you compare an English trace to a Hindi trace, how much of the gap is "different language" versus "the model is just unstable by itself"?
---
Every Correction Makes the Effect Bigger
This is the paper's most counter-intuitive finding.
Normally, removing confounders shrinks an effect—some apparent signal was confounded noise. Here, each of the five corrections enlarges the cross-lingual gap.
Why? Because every confounder *masks* difference:
---
Core Findings
Finding 1: Cross-Lingual Gap Is Structural, Not Sampling Noise
Under greedy decoding (temperature=0), the cross-lingual gap is positive in all 24 test cells. Greedy decoding removes sampling randomness, so the gap must originate in the model itself—it is structural.
Finding 2: 71-73% Policy Retention
Four frontier models (including GPT-OSS-120B, Claude, Gemini) retain only about 71-73% of their English action policy across other languages. That sounds acceptable but means roughly 1 in 4 actions diverges.
Finding 3: Model Identity Explains Only 5.7% of Variance
Differences across 8 models are dwarfed by differences across languages. "Which model you use" matters far less than "which language you use."
Finding 4: Below 10B Parameters, Regularity Collapses
Below 10B parameters, cross-lingual retention drops sharply and loses all pattern. Small models are not "a bit worse" on multilingual agent tasks—they are fundamentally unpredictable.
Finding 5: Agents Process Non-English Tasks Through an English Pivot
The deepest finding. When handling non-English tasks, agents internally translate the task to English, reason in English, then translate the result back. This English pivot is causally load-bearing—not correlational, but load-bearing.
Evidence: directly instruct the model "do not reason in English." Compliance rate: under 1%. The model *cannot* perform non-English agent tasks without the English pivot.
Finding 6: A Trace-Extraction Regex Manufactured a False Positive
GPT-OSS-120B appeared to fail multilingually—but the cause was a regex bug in the evaluation code. The model was fine; the measurement tool was wrong.
This finding gets its own section because it reveals a deeper lesson: you think you are measuring the model, but you are measuring your regex. Every regex, parser, and prompt template in the evaluation pipeline can become "measurement-manufactured failure."
---
Why This Paper Matters
1. Action Policy as a First-Class Measurement Object
Prior agent evaluations focus on task success rate. This paper measures cross-lingual consistency of action policy—the process, not just the outcome.
The process matters more than the outcome. Two models may both succeed, but one takes 3 steps and the other takes 7. In production, 4 extra steps mean 4 extra failure points, 4x latency, and 4x cost.
2. Establishing the Causality of the English Pivot
Prior work found latent English representations inside models (Wendler et al. 2024, Zhao et al. 2024), but that was representation-level correlation. This paper performs a causal test at the action level: forbid English reasoning, and the model cannot function.
The bottleneck of multilingual agents is not "can the model understand non-English"—it is "can the model think without going through English." The current answer is no.
3. Reaffirming the "Granularity Isomorphism" Principle
Observed earlier in Heddle and CodeRescue: optimization granularity should match the granularity of the object being optimized. Agent behavior granularity is the action trajectory, not the final answer. By aligning measurement granularity to action trajectories, this paper sees what final-answer metrics miss.
4. Another Instance of the "Evaluation Blind-Spot Law"
Each of the five confounders is an evaluation blind spot: you think you measure cross-lingual difference, but you measure trace length, empty traces, random baselines, model noise, or parsing bugs. Without correcting these blind spots, your conclusions are wrong—and they err in the direction of underestimating the problem.
---
Implications for Engineering Practice
If You Build Multilingual Agent Systems
1. Do not measure only final answers. Add an "action-policy consistency" metric using the paper's correction methodology. You will discover your system is worse than you thought. 2. The English pivot is reality. Do not expect models to natively handle non-English in the short term. Accept this and design explicit, controllable, monitorable English-pivot management in your system. 3. Do not build multilingual agents below 10B. Regularity collapses; behavior is unpredictable.
If You Build Agent Evaluations
1. Audit your regexes. GPT-OSS-120B's "multilingual failure" was manufactured by a regex bug. How many regexes are in your evaluation pipeline? Each is a potential false-positive source. 2. Report all five confounder corrections. Uncorrected "cross-lingual consistency" numbers are meaningless. 3. Report results under greedy decoding. Sampling randomness masks structural differences.
---
Honest Limitations
The paper does not address:
1. Only tool-calling agents tested. Whether conversational or planning agents exhibit the same English pivot is unknown. 2. 41-language coverage. Broad, but still biased—African and Indigenous languages are underrepresented. 3. The mechanism behind the English pivot. Causality is established, but how to break it remains an open question. 4. Cost. 2.38 million rollouts is industrial scale; academic teams cannot easily replicate.
---
One-Sentence Summary
When you talk to an agent in different languages, it appears to think in that language but secretly translates to English, reasons in English, and translates back. 2.38 million rollouts prove this English pivot is not habit—it is causal necessity: forbid it, and the model cannot complete the task.
---
> Paper: Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents > Code: github.com/souro/agent-actions-speak-louder-than-words > Authors: Sourabrata Mukherjee, Kalika Bali, Sunayana Sitaram > Affiliation: Microsoft Research India > Published: 2026-08-11