A Counter-Intuitive Finding
Imagine a customer-service team. Each agent has three work modes:
1. Text-only records (fast, cheap, but no visual cues) 2. Video-only recordings (fullest information, but slow and expensive) 3. Text + video (most complete, most expensive)
For each call you must assign one mode, and you want a "router" to automate the choice. The naive recipe is: let the agent try all three modes, observe which mode wins on which call, then train a classifier on call features.
A 2026 paper shows this recipe has a fatal structural problem: try-everything-first only produces labels when the agent succeeds. If the agent fails in every mode, you get no label of the form "mode X is better here." Worse, the weaker the agent, the fewer labels—and the weaker the agent is precisely where routing would be most valuable. Routing is least learnable where it is most valuable.
This is the core finding of Xian Sun et al. in arXiv 2608.06171, *Routing Is Least Learnable Where It Is Most Valuable*.
Six Observation Modes
The paper studies Web Agents that decide next actions (click, type, scroll) from a page state that can be represented in six ways:
1. text — HTML-to-text content 2. pixels — page screenshot 3. both — text + screenshot 4. 2x2 — text + screenshot + two zoomed corners 5. text+2corners — text + two corner crops 6. pixels+2corners — screenshot + two corner crops
No single mode dominates across tasks.
Three Key Findings
Finding 1 — Modes are complementary
Across 8 site-model combinations on VisualWebArena and WebArena, every mode solves tasks that other modes miss. Multimodal complementarity is common knowledge; the surprising results come next.
Finding 2 — The optimal mode reverses across task sets
The mode that wins on task set A can be the worst on task set B. There is no universal best mode: optimal choice depends on task distribution. This imposes a hard constraint—routers must condition on task features; fixed modes are suboptimal.
Finding 3 — The Oracle upper bound is inflated by noise
Web Agent runs are stochastic: 12–14% of outcomes change on rerun. The reported Oracle upper bound (assume the router always picks the best mode) is therefore inflated by that margin. Past routing papers that anchor evaluations to Oracle bounds inherit this bias.
The Structural Dilemma of Routing
Five routing strategies were tested (LLM-based, feature-similarity-based, history-based, plus two others). None consistently beat a well-chosen fixed mode.
The reason is a fundamental conflict:
- Strong agent → more successes → more routing labels → easier router training
- Weak agent → fewer successes → fewer labels → harder router training
- *Regression Tax*: skill libraries hurt agents in 59% of 5,832 trials—"help that backfires" parallels routing-trained-wrong.
- *Looping Is Not Reliability*: 82.0% → 67.3% across reruns matches the 12–14% routing noise.
- *TriviaRoomQA*: cliffs outside the knowledge boundary parallel routing being needed only at capability edges where labels vanish.
- *MIST*: training-induced false robustness parallels fixed modes that look stable while the Oracle bound is inflated.
- Only Web Agent settings were tested; generalization to code generation or dialogue is open.
- Five routing strategies were tested; the paper concedes its claim is about *why* routing is hard, not that routing is impossible.
- The retry-based cost lower bound trades latency for cost—unsuitable for latency-critical use.
Routing is most valuable precisely where the agent is weakest (a strong agent barely needs routing). The paper measures this correlation at r = 0.95—near-perfect negative correlation between label supply and routing value.
The author draws a cross-domain analogy to the educational "Matthew effect": high-performing students gain more resources (labels), while low performers gain fewer. The same dynamic starves routers of data where they matter most.
Cost Lower Bound: A Practical Engineering Solution
Rather than train a router, run the cheapest mode first (text); if it fails, escalate to a more expensive mode.
Applied to all 8 site-model combinations, this cost lower bound saves 9.5–30.6% of compute. It is inferior to the Oracle bound but is achievable without router training—mirroring the principle of "try the cheapest, escalate on failure" found in progressive cramming and minimal-footprint systems.
Three Layers of Implication
1. Evaluation: Oracle bounds are inflated by 12–14%; future papers must report noise levels. 2. Training: r = 0.95 negative correlation sets a structural ceiling on router training. The bottleneck is data, not algorithms. 3. Engineering: The cost-lower-bound retry scheme is deployable today, saving 9.5–30.6% with no router.
Cross-Paper Resonance
Together they argue that coverage breadth beats depth in measurement.
Honest Assessment
A Deeper Takeaway
The principle generalizes beyond routing: in RL, sparse rewards mark the states most needing policy improvement; in active learning, the most uncertain samples are the most expensive to label; in education, the students who need tutoring most produce the least feedback. "Hardest where most needed" is structural, not engineering.
Future progress likely comes from better label generation—synthetic data, counterfactual reasoning, cross-agent transfer—rather than better routing algorithms.
Closing
The paper elevates an engineering puzzle (why routers underperform) into a structural finding (label supply vs. value, r = 0.95). The takeaway for practitioners: when a router, classifier, or recommender trains poorly, first ask whether its label supply is negatively correlated with its value.
---
Paper link: https://arxiv.org/abs/2608.06171 Full HTML: https://arxiv.org/html/2608.06171v1