EnvOS: Auditing Computer-Use Agents When the World Moves On (arXiv 2610.40246)

Forum topic · 小凯 · 2026-10-31

Summary

EnvOS (QpiAI, arXiv:2610.40246, accepted at NeurIPS 2026 workshops AABA4ET and CL4FMAgents) addresses a blind spot in computer-use agent evaluation: the world changes while an agent works. Static benchmarks score only the final state, conflating three failure sources—agent error, faulty environment perturbations, and grading mistakes. EnvOS formalizes perturbations as ρ = (δ, τ, c, g): a state rewrite, a trigger condition, a disclosure channel, and a once-only guard, then certifies evaluation artifacts via five checks (intervention validity, observation consistency, feasibility, behavioral discrimination, scorer robustness). Execution pre-freezes instances and hashes 29 repo paths with SHA-256; validity is checked before outcomes are counted, with independent re-derivation from archived evidence. Experiments on a synthetic workplace testbed show dramatic effects: in a clinic-booking task, Claude Opus 5 succeeded 0/16 (no reminder), 2/16 (reminder only), and 15/16 (live value shown). In an insurance-approval task with the formula payout = min(claim − deductible, current limit), results diverged by model. Blind archive challenges found zero corrupted records accepted as clean. The paper's core thesis: evaluation of agents in dynamic environments must be timestamped, evidence-based, and recursively audited—especially before such environments feed RLVR reward pipelines.

This post is a detailed Chinese-language review of the paper EnvOS: Auditing Computer-Use Agents When the World Moves On (Kadu, Gopal, Singh, Jain, Krishnan — QpiAI, Bangalore; arXiv:2610.40246; accepted at NeurIPS 2026 Workshops AABA4ET & CL4FMAgents). Structured summary below; original Chinese text preserved in spirit.

The Problem: A Photo vs. a Recording

The author opens with an analogy: you book a clinic follow-up at 2 PM (Room 3, Building A); the hospital silently moves it to Building B at 2:30; you confirm at 3:00 and fail. Whose fault is it? Most agent evaluations assume the world is frozen from the start of an episode and grade only the terminal state—like judging a crime scene from a single photo. In dynamic environments, a "failure" label hides three distinct suspects:

1. The agent — saw the new value but didn't use it (a true failure). 2. The environment — the perturbation misfired, fired early, twice, or damaged unrelated records (agent wrongly blamed). 3. The grader — the change fired correctly but the scoring script judged a correct behavior as wrong.

EnvOS splits this into intervention validity (did the scheduled state transition actually occur, on time, without side effects?) and evaluation validity (did the agent actually receive the promised information, and is the verdict backed by archived evidence?). This matters acutely for RLVR: a wrong checker verdict becomes a poisoned reward that trains the model badly.

Experiment 1: The Relocating Clinic

Setup: the agent reads email, checks a calendar, opens a booking portal showing location v0; a later qualifying request silently rewrites the authoritative record to v1 (old page still visitable, no notification). Timing is crucial: the agent must see the old value *before* the change.

Three conditions (same backend change in all three):

| Condition | What the review step shows | |---|---| | OFF-A | Nothing extra | | CUE-A | A "please re-check" reminder + link, no new value | | LIVE-A | Same reminder + link, new location displayed |

Results (Claude Opus 5, 16 instances, 48-step budget): OFF-A 0/16, CUE-A 2/16, LIVE-A 15/16. Paired sign test LIVE vs. CUE: 13 vs. 0 pairs, p = 0.000244, surviving paper-wide Holm correction (p = 0.00293). Notably, in CUE-A the agent clicked the re-check link in only 1 of 16 trajectories.

Experiment 2: The Midnight-Revised Insurance Policy

Task formula: payout = min(claim − deductible, current limit), where the limit changes after the agent reads the old value.

Claude Opus 5: 10/10 in both OFF-A and ON-A — it proactively re-checked the policy record in every OFF-A run; the panel effect was untestable at ceiling.
Claude Sonnet 5: OFF-A 1/10, ON-A 9/10 (local p = 0.00781, not significant after paper-wide Holm, p = 0.0781).
Gemini 3.1 Pro Preview: OFF-A 0/10, ON-A 5/10 (p = 0.0625 — no detected effect; different calling stack, so no vendor effect can be isolated).

Conclusion: whether an agent "notices change" is a joint function of model, task, and interface — no single leaderboard is valid.

The EnvOS Pipeline

A perturbation is ρ = (δ, τ, c, g): state rewrite operator, trigger condition, disclosure channel, and a once-only guard. Key ontology rule: "the page was served," "the model was shown," and "the model understood and used it" are three independent events; only the first two are auditable.

1. Construction — start from an app-agnostic phenomenon ("a value the agent saw is superseded before it acts"), then instantiate it. 2. Certification — five gates: intervention validity, observation consistency, feasibility (per frozen instance), behavioral discrimination (a deliberately flawed policy must be caught), and scorer robustness. Outcomes are pass / violation / monitoring error, and "monitoring error" never counts as pass. 3. Controlled execution + evidence-based evaluation — instances, budgets, invariants, and analysis rules are pre-frozen; a release manifest SHA-256 hashes 29 repo paths, container images, generators, callers, and model IDs. A second, independently implemented codebase re-derives validity and outcomes from archived evidence; under-determined archives are marked "pending."

Evaluation ordering: validity-first counting — an episode enters analysis only if the perturbation fired and promised observations were delivered; otherwise the failure belongs to the exam, not the examinee. Conversely, a valid but unsuccessful run (including budget exhaustion) is simply a failure — no retakes. Outcome scoring checks five registered invariants (correct target state, single submission, no collateral damage, required post-submission actions, report consistency), each against the authoritative state at action time, not at episode end.

Auditing the Auditor

Three "archive challenge" groups corrupt archived evidence to test the pipeline:

Sealed challenge: 28 blind-built cases — scorer hit 28/28 expected verdicts.
Rules-only challenge: a GPT-6 session with only rule excerpts and the archive schema designed 56 cases; of 36 admitted, all 10 clean controls were accepted and none of the 26 corrupted cases passed as clean (24 caught, 2 no-verdict, 0 leaks).
External challenge: a Gemini CLI agent generated 56 cases; the pipeline caught 25 of 39 corrupted ones (14/20 insurance, 11/19 clinic) versus 4 for a naive log-check baseline.

Since these programmatically generated environments and audited auto-scorers are natural RLVR training venues, every scorer blind spot would become a model's bad habit — hence the recursive audit.

Honest Boundaries

Only two workflows on a synthetic 14-service testbed; no evidence of portability to arbitrary production apps.
"Understanding" is unobservable; only service and delivery of information are auditable.
Independent re-derivation is same-team second implementation, not third-party audit.
Challenge coverage has gaps; missed cases are logged, and a dated erratum is preserved in the appendix.
Exploratory phenomenon families (appendix S1) are excluded from main conclusions.

Takeaways

1. The world is alive; evaluation needs timestamps. Every action should be judged against the world at that moment, not at episode end. 2. Interface is capability. 0/16 → 2/16 → 15/16 is one model facing three ways the world presents information. 3. Check the exam room before judging the examinee — and vice versa: valid-but-incomplete is still a failure. 4. Audit is recursive — graders must themselves be audited with blind-built cases. 5. Rigor is a result: pre-registration, hash manifests, dated revisions, and abandoned under-counted blocks make every number reproducible. 6. Only audited evaluations deserve to be RLVR training venues — otherwise every scorer blind spot becomes a trained-in flaw.

Reference

Kadu, A., Gopal, A., Singh, N., Jain, A., & Krishnan, A. (2026). *EnvOS: Auditing Computer-Use Agents When the World Moves On*. arXiv:2610.40246. https://arxiv.org/abs/2610.40246