OpenForgeRL: Training Harness-Native AI Agents in Real Environments

*Translation and analysis of a Chinese forum post explaining the paper "OpenForgeRL: Train Harness-native Agents in Any Environment" (arXiv:2607.21557).*

> "Teaching AI to use tools isn't about having it memorize an operations manual—it's about letting it learn to steer from the real driver's seat."

The Problem: The "Cockpit Dilemma" of Agent Training

Modern AI agents—Claude Code, Codex, OpenClaw—rely on complex inference harnesses that drive multi-turn reasoning, tool use, and external system access. These harnesses act like a car's cockpit: input/output interfaces, state display, task planning, error backtracking, and conversation history.

Yet current training methods largely train agents in simplified "simulators" and hope they generalize to real harnesses. The result: agents learn "theory" but falter in real environments—mis-calling tools, losing track of multi-step plans, and failing to recover from errors.

Existing open RL infrastructure (veRL, OpenRLHF, TRL) struggles because:

• Stateful reasoning is hard to express: multi-turn harness interactions involve complex state changes beyond simple state→action→reward loops
• Multi-process coordination is difficult: harnesses run model inference, tool execution, and environment interaction in parallel
• Environment dependencies: each harness has unique API, filesystem, and network requirements

OpenForgeRL's Architecture: Two Core Innovations

1. Lightweight Proxy — the "Translator" Between Harness and RL

A proxy intercepts all model calls from the harness, forwards them, records request-response pairs plus reward signals, and converts them into standard RL formats (trajectories, rewards, advantages).

• The harness doesn't need to know training is happening—it just runs normally
• The RL framework doesn't need to understand harness complexity—it just receives standard-format data
• The two sides are decoupled and can evolve independently

2. Kubernetes Orchestrator — an Isolated "Training Ground" per Rollout

Each training rollout runs in its own Docker container with the full harness environment and agent code, supporting GPU-accelerated inference and automatic scaling. Different rollouts remain isolated from one another.

Training Loop

1. The harness receives tasks and calls the model through the proxy 2. The proxy logs the full interaction history 3. Logged interactions are converted to RL training data with reward signals 4. A standard RL codebase (e.g., veRL) runs PPO/GRPO updates 5. Updated weights rejoin the inference loop until convergence

Experimental Results

OpenForgeClaw (tool-use agents on OpenClaw / ZeroClaw)

• ClawEval pass^3: 31.7
• ClawEval pass@3: 55.9
• QwenClawBench: 33.7

Notably, these results were achieved with "only hundreds to thousands of tasks," demonstrating high sample efficiency.

OpenForgeGUI (browser-based GUI agents)

• OSWorld-Verified: 37.7
• Online-Mind2Web: 63.0
• WebVoyager: 72.3

The paper notes that 37.7 on OSWorld "matches or exceeds models many times larger" in GUI settings—evidence that training environment realism can matter more than model scale.

Harness Choice Matters

Comparing ZeroClaw (minimal), OpenClaw (standard), and Codex (feature-rich):

• Some harnesses are easier to learn from because they provide richer learning signals
• RL improves reliability: more self-verification, broader tool coverage, better multi-step plan completion
• Error recovery remains a weakness: agents still get stuck after unexpected failures, likely because errors are diverse, recovery strategies are complex, and reward signals are sparse

Baselines

Agents trained with OpenForgeRL "outperform similar-size open-source baselines on nearly all benchmarks," including SFT-only agents and agents trained in simplified environments.

Key Analysis and Design Principles

How RL changes behavior: SFT teaches imitation; RL teaches evaluation of expected returns. Post-RL agents check intermediate results more often, choose tools based on task needs, and persist through multi-step plans.

Harness design principles: 1. Provide sufficient but not redundant structure (ZeroClaw too bare, Codex potentially overwhelming, OpenClaw "just right") 2. Error messages should be clear and actionable 3. Maintain observability of intermediate states

Possible fixes for error recovery: dense reward shaping for reasonable recovery attempts, curriculum learning from simple to complex failures, and symbolic planning for higher-level recovery strategies.

Broader Implications

• Paradigm shift: from generic pretraining toward environment-specific post-training—train agents where they will actually live
• Open-source ecosystem: OpenClaw's inclusion alongside commercial harnesses signals that open harnesses are serious research platforms; OpenForgeRL gives the community a tool to compete with closed alternatives
• Caveats: capability gaps between institutions, potential misuse of agents trained in real environments, and employment impacts all warrant attention

Conclusion

OpenForgeRL's core idea is simple: let AI learn from the real driver's seat rather than from classroom slides. Its elegant machinery—proxy-based decoupling and Kubernetes-orchestrated rollouts—makes any interceptable harness trainable. The results show that a mid-sized model trained in a realistic harness can beat larger models trained in simulations. Real intelligence, the post concludes, isn't memorized—it grows from interaction with the world.

References

• Yu, X., Peng, B., Xu, R., Zou, H., Wu, Q., Cheng, H., Yao, W., Singh, N., Yu, Z., & Gao, J. (2026). OpenForgeRL: Train Harness-native Agents in Any Environment. *arXiv preprint* arXiv:2607.21557.
• Schulman, J., et al. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
• Sheng, G., et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. *NeurIPS 2024*.