OpenForgeRL: Training Harness-Native AI Agents in Real Environments

> *"Teaching an AI to use tools is not about making it memorize an operations manual — it's about letting it learn to steer from the real driver's seat."*

This post is an English walkthrough of the paper "OpenForgeRL: Train Harness-native Agents in Any Environment" (arXiv:2607.21557), popular on zhichai.net for its relevance to the OpenClaw community.

The Problem: The Harness-Training Gap

Modern AI agents — Claude Code, Codex, OpenClaw, ZeroClaw — rely on a complex inference harness handling multi-turn reasoning, tool use, state management, and error handling. Yet current training approaches (SFT on expert demonstrations, or standard RL like PPO/GRPO) struggle with harness complexity:

1. Stateful reasoning is hard to express — multi-turn harness interactions don't fit the simple "state → action → reward" loop 2. Multi-process coordination is difficult — model inference, tool execution, and environment interaction run in parallel 3. Environment dependencies — each harness has unique API, filesystem, and network requirements

The result: agents trained in simplified simulators flounder in real harnesses — misusing tools, losing track of multi-step plans, and failing to recover from errors.

Core Insight: Decouple Training from Inference

OpenForgeRL's answer is elegant: train directly inside the real harness. Two components make this possible:

1. Lightweight Proxy

A proxy sits between the harness and the model:

• Intercepts all model call requests, records request-response pairs and reward signals
• Converts them into standard RL formats (trajectories, rewards, advantages)
• Feeds standard RL codebases (e.g., veRL)
• The harness doesn't know training is happening; the RL framework doesn't need to know the harness's complexity. Both can evolve independently.

2. Kubernetes Orchestrator

Each rollout gets an isolated remote container with the full harness environment, supporting GUI/browser environments, custom filesystems, GPU inference, and auto-scaling.

Training Loop

1. Harness receives a task, calls the model through the proxy 2. Proxy logs the full interaction and computes rewards 3. Standard RL library (PPO/GRPO) updates model weights 4. Updated weights participate in the next inference round

Key Results

Tool-use agents (OpenForgeClaw, on OpenClaw / ZeroClaw harnesses):

• ClawEval pass^3: 31.7
• ClawEval pass@3: 55.9
• QwenClawBench: 33.7
• Achieved with only hundreds to thousands of tasks — high sample efficiency

GUI agents (OpenForgeGUI):

• OSWorld-Verified: 37.7 — matching or exceeding models several times larger
• Online-Mind2Web: 63.0
• WebVoyager: 72.3

Comparisons: Agents trained with OpenForgeRL outperform similarly-sized open-source baselines on nearly all benchmarks, including SFT agents and RL agents trained in simplified environments.

Analysis Highlights

How RL changes agent behavior:

• Self-verification: agents actively test intermediate results (e.g., running tests after code generation)
• Tool coverage: agents select the right tool for the task rather than defaulting to familiar ones
• Plan completion: agents persist through multi-step plans

Persistent weakness — error recovery: Even after RL, agents often get stuck when tool calls fail. Causes: many failure modes, complex recovery strategies, and sparse reward signals. Possible fixes: dense reward shaping, curriculum learning, symbolic planning.

Harness design principles:

• Provide sufficient but not redundant structure (ZeroClaw gives too few learning signals; richer harnesses like Codex help)
• Make error messages clear and actionable
• Maintain observability of intermediate states

Why It Matters

OpenForgeRL represents a shift from training in perfect simulated environments to cultivating reliable agents in the real world. For the open-source community, OpenClaw's inclusion alongside Claude Code and Codex validates open harnesses as serious research platforms. The key takeaway: training environment quality can matter as much as — or more than — model scale.

References

• Yu, X., et al. (2026). OpenForgeRL: Train Harness-native Agents in Any Environment. *arXiv preprint* arXiv:2607.21557.
• Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
• Sheng, G., et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. *NeurIPS 2024*.
• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. *NeurIPS 2022*.