English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Routing Is Least Learnable Where It Is Most Valuable: Upper and Lower Bounds of Web Agent Observation Modes

Forum topic · ✨步子哥 · 2026-08-09

Summary

A 2026 arXiv paper (2608.06171) by Xian Sun et al. reveals a structural dilemma in routing for Web Agents choosing among six observation modes (text, pixels, both, 2x2, text+2corners, pixels+2corners). Across 8 site-model combinations on VisualWebArena and WebArena, the authors show: (1) modes are complementary—each solves tasks others miss; (2) the optimal mode reverses across task sets, so no universal best mode exists; (3) Oracle upper bounds are inflated by 12–14% run-to-run noise. Most critically, routing supervision labels are only produced when the agent succeeds, so label supply is negatively correlated (r=0.95) with routing's value—the hardest-to-learn cases are exactly where routing helps most. No tested routing strategy beats a fixed well-chosen mode. The paper proposes a practical cost-lower-bound strategy: run the cheapest mode (text) first, escalate to costlier modes on failure, yielding 9.5–30.6% cost savings without router training. The core insight—that label supply and learning value can be anti-correlated—applies broadly to any learning setting where supervision signals arise only on success.

Routing Is Least Learnable Where It Is Most Valuable: Upper and Lower Bounds of Web Agent Observation Modes

A Counterintuitive Finding

Imagine managing a customer support team where each agent can work in three modes: reading text transcripts only (fast, cheap, but misses nuance), watching video only (most information, but slow and expensive), or both (most complete, but costliest). Your job is to train a "router" that picks a mode per call.

Intuitively, you'd have the agents try all three modes, see which works where, and train a classifier to select modes. But a paper from August 2026 shows this approach has a fatal structural flaw.

The problem is in "have the agents try all modes first." If the agent is weak and fails in every mode, you never get the labels saying "this mode works better on this kind of task." Router training labels only exist when the agent succeeds. The weaker the agent, the fewer the labels—and the weaker the agent, the more routing is needed.

The hardest part of routing to learn is exactly where routing is most valuable.

This is the core finding of Xian Sun et al.'s arXiv paper 2608.06171, *Routing Is Least Learnable Where It Is Most Valuable*.

Six Observation Modes

The paper studies observation-mode selection for Web Agents, which decide actions (click, type, scroll) based on page state. Six modes are compared:

1. text – page text (HTML-to-text) 2. pixels – screenshot 3. both – text + screenshot 4. 2x2 – text + screenshot + two zoomed corner crops 5. text+2corners – text + two corner crops 6. pixels+2corners – screenshot + two corner crops

Each has trade-offs: text is fast but lacks visual info; pixels is slow but complete; both is fullest but costliest. No mode is optimal on all tasks.

Three Key Findings

1. Modes are complementary

Testing all six modes on 8 site-model combinations in VisualWebArena and WebArena, each mode solves tasks the others miss. Modes are not substitutes; they cover each other's blind spots.

2. The optimal mode reverses across task sets

The best mode on task set A can be the worst on task set B. Across the 8 site-model combinations, the optimal mode choice reverses. There is no universally best mode—the right choice depends on the task distribution, so a fixed mode is always suboptimal, and a router must identify task features.

3. Oracle upper bounds are inflated by noise

Web Agent runs are stochastic—the paper measures 12–14% of outcomes flip on re-runs. This means reported Oracle upper bounds (perfect routing) are inflated: previously reported Oracle bounds may be overstated by 12–14%, undermining evaluations that use them as reference points.

The Structural Dilemma of Routing

The paper tests 5 routing strategies (LLM-based, feature-similarity-based, historical success-rate-based). None consistently beats a fixed, well-chosen mode.

The core contradiction: routing supervision signals only arise when the agent succeeds. When the agent fails, you cannot tell whether the mode was wrong or the agent simply cannot do the task. This creates a feedback loop:

  • Stronger agent → more successes → more routing labels → easier router training
  • Weaker agent → fewer successes → fewer labels → harder router training
  • And routing is most valuable precisely where the agent is weakest—strong agents don't need routing at all. The paper measures r = 0.95, a near-perfect negative correlation between label supply and routing's value.

    This mirrors the "Matthew effect" in education: strong students get more feedback resources, weak students get fewer. The places that most need routing are where routing is hardest to learn.

    Cost Lower Bound: A Practical Engineering Solution

    Despite the difficulty of router training, the paper offers a deployable cost lower bound strategy: run each task first with the cheapest mode (text); if it succeeds, stop; if it fails, escalate to a costlier mode.

    Across all 8 site-model combinations this saves 9.5–30.6% of cost. It falls short of the Oracle bound, but it is achievable and requires no router training—the engineering principle of "try the cheapest thing first, escalate only on failure."

    Implications at Three Levels

    1. Evaluation: The 12–14% noise inflation means prior Oracle upper bounds are overstated. Future evaluations must report noise levels alongside bounds. 2. Training: The r=0.95 negative correlation imposes a structural ceiling on router training—a data problem, not an algorithm problem. 3. Engineering: The cost-lower-bound escalation delivers 9.5–30.6% savings with no training, at the price of added latency in failure cases (worst case runs the task twice).

    Cross-Paper Resonance

  • Regression Tax: skill libraries can make agents worse—tools meant to help can hurt, as with badly trained routers.
  • Looping Is Not Reliability: correctness is not an absorbing state (82.0% → 67.3%); routing's 12–14% noise is the same phenomenon—past success ≠ current success.
  • TriviaRoomQA: models degrade sharply beyond knowledge boundaries; routing is needed exactly beyond the agent's competence—where it also cannot learn.
  • MIST: robustness training can create false robustness; a fixed mode looks "stable," but Oracle bounds hide 12–14% inflation.
All four papers point to one theme: measurement coverage matters more than measurement depth.

Honest Assessment

Limitations: (1) only Web Agent scenarios were tested, though the structural contradiction is likely general; (2) the 5 routing strategies are weak, so the paper's contribution is diagnosing *why* routers are hard to train, not proving routers are useless; (3) the escalation scheme sacrifices latency in the failure path.

A Deeper Insight

The most memorable takeaway is a general principle: the supply of supervised labels and the value of supervised learning can be negatively correlated.

The same appears in RL (rewards are sparse where policy improvement is most needed), active learning (the most uncertain samples are the most expensive to label), and education (weak students most need tutoring but generate the least feedback). "Hardest to learn where most needed" is a structural predicament, not an engineering bug.

The authors offer no solution, but a clear diagnosis: the ceiling on router training is at the data layer, not the algorithm layer. Future breakthroughs may come from smarter label generation—synthetic data, counterfactual reasoning, or cross-agent transfer—rather than better routing algorithms.

Conclusion

This paper elegantly elevates an engineering problem (routers train poorly) into a structural finding (label supply is anti-correlated with routing value). Next time a router, classifier, or recommender trains badly, ask: are its label supply and its value negatively correlated? If yes, the problem is not the algorithm—it's the structure.

Paper: https://arxiv.org/abs/2608.06171 HTML full text: https://arxiv.org/html/2608.06171v1

Tags

#web-agents#routing#observation-modes#llm-evaluation#oracle-bounds#supervised-learning#cost-optimization#structural-dilemma

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178603088