Routing Is Least Learnable Where It Is Most Valuable: Upper and Lower Bounds of Web Agent Observation Modes
A Counterintuitive Finding
Imagine managing a customer support team where each agent can work in three modes: reading text transcripts only (fast, cheap, but misses nuance), watching video only (most information, but slow and expensive), or both (most complete, but costliest). Your job is to train a "router" that picks a mode per call.
Intuitively, you'd have the agents try all three modes, see which works where, and train a classifier to select modes. But a paper from August 2026 shows this approach has a fatal structural flaw.
The problem is in "have the agents try all modes first." If the agent is weak and fails in every mode, you never get the labels saying "this mode works better on this kind of task." Router training labels only exist when the agent succeeds. The weaker the agent, the fewer the labels—and the weaker the agent, the more routing is needed.
The hardest part of routing to learn is exactly where routing is most valuable.
This is the core finding of Xian Sun et al.'s arXiv paper 2608.06171, *Routing Is Least Learnable Where It Is Most Valuable*.
Six Observation Modes
The paper studies observation-mode selection for Web Agents, which decide actions (click, type, scroll) based on page state. Six modes are compared:
1. text – page text (HTML-to-text) 2. pixels – screenshot 3. both – text + screenshot 4. 2x2 – text + screenshot + two zoomed corner crops 5. text+2corners – text + two corner crops 6. pixels+2corners – screenshot + two corner crops
Each has trade-offs: text is fast but lacks visual info; pixels is slow but complete; both is fullest but costliest. No mode is optimal on all tasks.
Three Key Findings
1. Modes are complementary
Testing all six modes on 8 site-model combinations in VisualWebArena and WebArena, each mode solves tasks the others miss. Modes are not substitutes; they cover each other's blind spots.
2. The optimal mode reverses across task sets
The best mode on task set A can be the worst on task set B. Across the 8 site-model combinations, the optimal mode choice reverses. There is no universally best mode—the right choice depends on the task distribution, so a fixed mode is always suboptimal, and a router must identify task features.
3. Oracle upper bounds are inflated by noise
Web Agent runs are stochastic—the paper measures 12–14% of outcomes flip on re-runs. This means reported Oracle upper bounds (perfect routing) are inflated: previously reported Oracle bounds may be overstated by 12–14%, undermining evaluations that use them as reference points.
The Structural Dilemma of Routing
The paper tests 5 routing strategies (LLM-based, feature-similarity-based, historical success-rate-based). None consistently beats a fixed, well-chosen mode.
The core contradiction: routing supervision signals only arise when the agent succeeds. When the agent fails, you cannot tell whether the mode was wrong or the agent simply cannot do the task. This creates a feedback loop:
- Stronger agent → more successes → more routing labels → easier router training
- Weaker agent → fewer successes → fewer labels → harder router training
- Regression Tax: skill libraries can make agents worse—tools meant to help can hurt, as with badly trained routers.
- Looping Is Not Reliability: correctness is not an absorbing state (82.0% → 67.3%); routing's 12–14% noise is the same phenomenon—past success ≠ current success.
- TriviaRoomQA: models degrade sharply beyond knowledge boundaries; routing is needed exactly beyond the agent's competence—where it also cannot learn.
- MIST: robustness training can create false robustness; a fixed mode looks "stable," but Oracle bounds hide 12–14% inflation.
And routing is most valuable precisely where the agent is weakest—strong agents don't need routing at all. The paper measures r = 0.95, a near-perfect negative correlation between label supply and routing's value.
This mirrors the "Matthew effect" in education: strong students get more feedback resources, weak students get fewer. The places that most need routing are where routing is hardest to learn.
Cost Lower Bound: A Practical Engineering Solution
Despite the difficulty of router training, the paper offers a deployable cost lower bound strategy: run each task first with the cheapest mode (text); if it succeeds, stop; if it fails, escalate to a costlier mode.
Across all 8 site-model combinations this saves 9.5–30.6% of cost. It falls short of the Oracle bound, but it is achievable and requires no router training—the engineering principle of "try the cheapest thing first, escalate only on failure."
Implications at Three Levels
1. Evaluation: The 12–14% noise inflation means prior Oracle upper bounds are overstated. Future evaluations must report noise levels alongside bounds. 2. Training: The r=0.95 negative correlation imposes a structural ceiling on router training—a data problem, not an algorithm problem. 3. Engineering: The cost-lower-bound escalation delivers 9.5–30.6% savings with no training, at the price of added latency in failure cases (worst case runs the task twice).
Cross-Paper Resonance
Honest Assessment
Limitations: (1) only Web Agent scenarios were tested, though the structural contradiction is likely general; (2) the 5 routing strategies are weak, so the paper's contribution is diagnosing *why* routers are hard to train, not proving routers are useless; (3) the escalation scheme sacrifices latency in the failure path.
A Deeper Insight
The most memorable takeaway is a general principle: the supply of supervised labels and the value of supervised learning can be negatively correlated.
The same appears in RL (rewards are sparse where policy improvement is most needed), active learning (the most uncertain samples are the most expensive to label), and education (weak students most need tutoring but generate the least feedback). "Hardest to learn where most needed" is a structural predicament, not an engineering bug.
The authors offer no solution, but a clear diagnosis: the ceiling on router training is at the data layer, not the algorithm layer. Future breakthroughs may come from smarter label generation—synthetic data, counterfactual reasoning, or cross-agent transfer—rather than better routing algorithms.
Conclusion
This paper elegantly elevates an engineering problem (routers train poorly) into a structural finding (label supply is anti-correlated with routing value). Next time a router, classifier, or recommender trains badly, ask: are its label supply and its value negatively correlated? If yes, the problem is not the algorithm—it's the structure.
Paper: https://arxiv.org/abs/2608.06171 HTML full text: https://arxiv.org/html/2608.06171v1