OS-Themis: A Multi-Agent Critic Framework for Reliable GUI Agent Reward Modeling

> "Justice must not only be done, but must be seen to be done." — English legal maxim

This post introduces OS-Themis, a scalable critic framework for generalist GUI rewards developed by researchers from USTC, Shanghai AI Laboratory, and NVIDIA (arXiv:2603.19191). Named after the Greek goddess of justice, it replaces the single, opaque "judge" with a structured panel of specialized AI agents.

The Problem: Who Grades the Grader?

GUI agents (AI that operates phones/computers via screenshots and clicks) rely on reward signals for reinforcement learning. Existing approaches all have flaws:

• Rule-based rewards: precise but brittle; cannot cover unseen scenarios and invite reward hacking.
• Trained reward models: require expensive human annotation and degrade on out-of-distribution tasks.
• LLM-as-a-Judge: flexible and zero-shot, but suffers from:
• Context loss — only the last steps of a long trajectory are effectively considered
• Evidence dilution — many irrelevant successes mask one critical failure (e.g., 48 of 50 steps go well, but step 23's wrong address sinks the task)
• Low signal-to-noise ratio

Evidence dilution is especially dangerous: wrong signals get amplified by RL, reinforcing the very mistakes that caused failure.

The Architecture: A Courtroom of Four Agents

OS-Themis coordinates four specialized agents in two stages:

| Agent | Role | Analogy | |-------|------|---------| | Selector | Extracts key milestones from long trajectories | Prosecutor: find key evidence | | Verifier | Validates each milestone using before/after screenshots | Forensic expert: verify evidence | | Reviewer | Audits the whole evidence chain, flags gaps, triggers iterative refinement | Defense attorney: challenge completeness | | Judge | Issues the final binary verdict from the full deliberation | Judge: final ruling |

Stage 1 — Milestone Verification Module (MVM): the Selector condenses a 50-step trajectory into a handful of milestones (subgoal, key step, importance rationale), and the Verifier checks each one against visual evidence.

Stage 2 — Verdict Calibration Module (VCM): the Reviewer audits for missing milestones or lax success criteria, requesting refinement iteratively; the Judge then rules based on the entire deliberation process — not just a black-box score.

This design addresses context loss (milestones carry full context), evidence dilution (auditing ensures key failures aren't ignored), and explainability (a complete evidence chain for every verdict).

Results

• OGRBench (OmniGUIRewardBench): the first cross-platform ORM benchmark spanning mobile (Android), web, and desktop.
• vs. baselines: +18.8% accuracy vs. DigiRL, +7.7% vs. ZeroGUI; +29.6%/+5.1% precision; +16.9%/+13.0% recall; +26.2%/+13.4% F1 (relative improvements reported in the paper).
• Online RL: using OS-Themis as the reward signal improves agent performance on AndroidWorld by 10.3%.
• Data filtering: validating/filtering trajectories boosts Qwen3-VL self-training by 6.9%.
• Ablations: removing any of the four agents hurts performance — Selector (no milestones), Verifier (inaccurate scoring), Reviewer (over-optimistic due to evidence dilution), Judge (over-conservative).

Key Takeaways

• Multi-agent collaboration over monolithic judging: specialized agents (extract, verify, audit, judge) outperform a single generalist evaluator.
• Process supervision over outcome supervision: analyzing which milestones succeeded or failed teaches agents more than a binary task-level score.
• Interpretability matters: evidence-backed verdicts are invaluable for debugging and for high-stakes domains like healthcare and finance.

Limitations and Open Questions

• RL scaling: large-scale online RL remains limited by infrastructure.
• Reward granularity: rewards are currently binary; milestone-level reward shaping is future work.
• Ethical risks: semantic reward hacking via VLM reasoning loopholes, bias propagation from pretrained VLMs, and privacy concerns around high-fidelity screenshots all require careful mitigation and human-in-the-loop safeguards.

References

1. Li, Z., Wu, Z., Zhao, Y., et al. (2026). *OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards*. arXiv:2603.19191. 2. Wang, Z., et al. (2025). *DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning*. arXiv preprint. 3. Zhang, C., et al. (2025). *ZeroGUI: Zero-Shot Graphical User Interface Agents via Visual Imitation*. arXiv preprint. 4. Rawles, C., et al. (2024). *AndroidWorld: A Dynamic Benchmark for Environment-Centric Android Agents*. arXiv preprint. 5. Ouyang, L., et al. (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022.