An Embarrassing Discovery
You train a World Action Model (WAM) that generates actions, predicts future states, and scores each candidate action. The promise of test-time scaling sounds perfect: spend more compute at inference, generate N candidate actions, pick the best one with a value function, and performance goes up.
But a team of researchers from CMU, MIT, Harvard, and Stanford decided to seriously audit this: what does the extra compute actually buy?
They ran experiments with Cosmos Policy on RoboCasa. The results were striking:
- A deployable value selector performed 1.66 percentage points worse than random selection
- Even a hindsight oracle (god's-eye view) gained only 1.40 pp, with a confidence interval crossing zero
- In 82% of states, all candidate actions produced completely identical outcomes—there was no room to select at all
- The deployment path was only 1.15x faster (target was 1.5x)
- Even excluding future-image decoding, only 1.25x
- Fix decision thresholds and repeat-consistency gates before seeing data
- Use power analysis to decide whether full experiments are worth running
- Honestly report failures of gates rather than quietly swapping experiments
- Do candidates truly differ?
- Did nominal savings become actual speedup?
- Is the evaluation path reliable under extra compute?
In other words: you spent 8x the compute on search, only to find that in most cases all 8 candidate actions end the same way—and your value function performs worse than a coin flip.
Not a "Bad Selector" Problem—There's Nothing to Select
The paper (arXiv: 2610.41970) introduces a concept everyone had overlooked: selection headroom.
Imagine ordering at a restaurant where the menu shows 8 dishes and you hire a food critic to pick the best. But what if all 8 dishes are just different platings of the same dish? No critic can work magic.
That's exactly the situation with Cosmos Policy on RoboCasa. The researchers executed all 8 candidate actions at each state and found that in 82% of states, all candidates either all succeeded or all failed. The candidates were geometrically distinct (action-vector dispersion was normal), but they made no difference in task outcomes.
The Ceiling of a God's-Eye View
To confirm the problem isn't a weak selector but an absence of options, the researchers built a hindsight oracle: it knows the actual outcome of every candidate and always picks a successful one.
The result? The oracle beat random selection by only 1.40 percentage points, and the episode-level bootstrap confidence interval was [-0.25, +3.40]—crossing zero.
Even with a perfect selector, this configuration offers almost nothing to gain. The selection headroom itself is this small.
A finer analysis found: of 263 states, only 48 had any outcome differences between candidates, and of those 48, only 20 were "stably separable" (one candidate succeeded in both repeats while another failed both times). But the label-shuffle null hypothesis expected about 14.8 stably separable states—the observed 20 is only slightly above chance.
47% Fewer FLOPs, Only 15% Faster
The second audit dimension is equally deflating.
Since selection doesn't help, why not save compute? Token pruning cut 47.9% of per-forward FLOPs—which sounds impressive.
But the researchers did something few people do: they measured end-to-end latency directly.
Results:
Why? Of a total query time of 1241.8 ms, the 5 DiT forward passes take 530.7 ms (prunable), but VAE encoding (238.0 ms) and future-image decoding (434.4 ms) are irreducible. You halved the prunable portion, but that portion is only a small fraction of the total.
It's like widening one highway segment from 2 to 4 lanes when the real bottleneck is the toll booth and the exit ramp.
Power Gate: Don't Run Experiments That Can't Detect Effects
Even more admirable: before running joint-allocation experiments, the researchers set a pre-registered power gate—estimate noise on a small sample, compute the required sample size for the pre-registered effect size. If noise is too high relative to the effect, don't run the full experiment.
The joint-allocation effect's systematic component was 0.0045 while the mean per-state noise was 0.0066—the signal was smaller than the noise. They honestly reported "inconclusive" rather than forcing an interpretation of what might be pure noise.
The Most Sinister Finding: Corruption Rates Correlate with Compute
The third audit dimension is a genuine ghost story.
During experiments, the researchers discovered a silent framebuffer corruption bug in the RoboCasa/robosuite simulator:
Mechanism
When the environment performs a hard reset, the simulator creates a new instance, but the old instance lingers awaiting garbage collection. When the old instance is GC'd, its OpenGL context is released—while a different context is active. Releasing the old context silently corrupts the current environment's framebuffer.
Subsequent renders return corrupted observations: frozen frames or noise. No errors, no exceptions.
Corruption Rates Scale with Inference Compute
The chilling part:
| Inference mode | Corruption rate | |---|---| | Direct policy (1 forward pass) | 0.67% | | Direct-mode best-of-N (N=8) | 0/40 | | Planning mode (multiple future-state + value calls) | up to 65% |
The more compute you spend, the higher the probability of corruption.
This means: if you see "improvement" in planning mode, you can't tell whether it's real improvement or measurement error from rendering corruption. More compute, less trustworthy measurement.
Official Numbers Unaffected
The researchers carefully verified that the officially reported 67.1% direct-policy figure is essentially unaffected (corruption effect ~+0.25 pp, upper bound +0.6 pp, and the direction understates success). But the problem: the bug only erupts at scale on compute-intensive planning paths, which official benchmarks don't test.
The researchers released a framebuffer sentinel detector, a fix patch, and reproduction scripts.
Premise-Chain Auditing: Don't Judge by Success Rate
The paper's biggest methodological contribution is the premise-first audit framework.
The Problem with the Traditional Approach
Most papers evaluate test-time scaling by running it and checking whether success rate improves. But that can't distinguish three scenarios: 1. Selection signal exists and the selector captures it → real gain 2. No selection signal, but simulator bugs make some episodes "luckily" succeed → false gain 3. Selection signal exists but isn't captured—compute wasted → no gain
Identical success rates, completely different causes.
The Premise Chain
The researchers decompose test-time scaling into three independent premises:
1. Decision-signal premise: do candidate actions differ in exploitable outcomes? 2. Efficiency-realization premise: do nominal FLOPs savings translate into end-to-end speedup? 3. Evaluation-integrity premise: does the evaluation path remain reliable under extra compute?
Each premise is audited independently, without back-inferring from final success rates.
Pre-Registration: Commit Before Experiments
Another highlight is the pre-registered methodology:
When the hindsight oracle's episode-level CI crossed zero, they reported "weak, inconclusive evidence" rather than arguing the positive point estimate. When the joint-allocation power gate failed, they ran no full factorial and reported "inconclusive rather than evidence of no effect."
Lessons for Engineering Practice
1. Measure Selection Headroom Before Training a Selector
If you're deploying best-of-N planning, the first step isn't training a better value function—it's checking whether candidates differ in outcomes at all. Use a hindsight oracle. If headroom is tiny, invest in candidate diversity instead.
2. Always Measure End-to-End Latency
FLOPs savings are the easiest metric to self-deceive with. 47.9% fewer FLOPs yielded only 15% actual speedup. Measure full-query wall-clock time before deployment.
3. Check Simulator Reliability on Compute-Intensive Paths
Most simulator bugs don't surface on low-compute paths. Before claiming any test-time-scaling gains, verify the evaluation path is reliable under extra compute. The framebuffer sentinel idea generalizes to other simulators.
4. Pre-Register Your Experiments
Fix decision criteria before seeing data. "Inconclusive" is actionable information: you need a larger sample or lower noise, not a cleverer explanation.
Where This Paper Sits
The paper is not saying "test-time scaling is useless." The authors explicitly scope conclusions to Cosmos Policy on RoboCasa's tested configuration.
But it provides an audit methodology applicable to any test-time-scaling scenario:
---
Core contributions: premise-chain audit methodology + three negative results + a rendering-corruption detector and fix
Applicability: any WAM/Agent system using test-time scaling—audit the three premises before claiming "compute for performance."