The open-source community keeps asking: why are OpenAI's and Google's agents so strong while open models fall behind? The Orchard paper offers an unexpected answer — the problem is not the models, but the scaffolding.
A Microsoft Research team led by Baolin Peng found that the biggest bottleneck in open-source agent research is not insufficient model capability but fragmented infrastructure. Every research team reinvents the wheel: building sandboxes, managing rollouts, cleaning environments, writing evaluation scripts. Orchard's core insight is almost brutally simple: turn the environment layer into a thin, standalone, reusable service, letting trajectory data, training recipes, and evaluation protocols flow freely across domains.
This sounds like an engineering detail, but the results are striking.
---
Three Recipes, One Infrastructure
Orchard is not designed for a single task. On the same environment layer it builds three entirely different agent training schemes:
Orchard-SWE targets software engineering. The team distilled 107K trajectories from MiniMax-M2.5 and Qwen3.5-397B — but the key is not the volume; it's that they kept 32.5K unsolved trajectories. Traditional pipelines keep only successful cases; Orchard instead uses credit-assignment to extract "productive segments" from failures (parts that were genuinely making progress even if the task went unsolved) and converts partial progress into training signal. Combined with Balanced Adaptive Rollout to address sample efficiency in sparse-reward RL, Qwen3-30B-A3B-Thinking reaches 67.5% on SWE-bench Verified — SOTA among open models of similar size, approaching MoE systems 10–30x larger.
Orchard-GUI targets browser automation. Using only 0.4K distilled trajectories + 2.2K open tasks, a 4B vision-language model achieves an average 68.4% success rate across WebVoyager, Online-Mind2Web, and DeepShop. Most counterintuitively: the 4B student surpasses its 235B teacher. The paper's careful phrasing is "environment-grounded RL can improve model's agentic capabilities beyond those of the teacher" — meaning that under domain-specific RL, a small model can be better at "using tools" than a much larger one.
Orchard-Claw targets personal assistants. Trained on just 0.2K synthetic tasks, it achieves 59.6% pass@3 on Claw-Eval. Swapping in the stronger ZeroClaw harness lifts the same model to 73.9%. This exposes an overlooked issue: the harness (an agent's tool interface design) may affect performance more than the model itself.
---
The Environment Layer: More Than a Docker Wrapper
Orchard Env's technical design includes several notable choices:
Agent injection: instead of requiring researchers to modify Docker images to embed an execution agent, Orchard injects at runtime via a K8s init container. Any task image can plug in unmodified.
Direct Pod-IP routing: execution requests go straight to the sandbox Pod's IP, bypassing the K8s API server and kubectl exec WebSocket overhead. The result: 0.28s average command latency — on par with SkyPilot Code Sandbox, 2.7x faster than E2B, and 7.3x faster than Modal.
1000 concurrent sandboxes at 100% success: full lifecycle (create → run 4 commands → delete) completes within 26 seconds, with throughput of 154 commands/s.
But behind these numbers, cost matters most. The paper's math: running 128 parallel sandboxes for 240 hours costs $7,078 on Daytona and E2B, versus $3,362 on self-hosted K8s with Orchard (on-demand) — and just $673 with spot instances. A 10x gap.
That is not pocket change. It translates into the accessibility gap between independent researchers and big-corporate labs.
---
An Overlooked Finding: Harness Lock-in
The least-emphasized but potentially most consequential result in the paper is the severity of harness lock-in.
Orchard-SWE trained the same Qwen3-30B-A3B-Thinking model on the mini-swe-agent harness and then evaluated it on the OpenHands harness — performance dropped sharply. This shows that agent capability is not "intrinsic to the model"; it is a coupled product of model + harness.
Orchard's response: use multi-harness training during data collection so the model learns more general tool-use patterns rather than overfitting to a specific interface — analogous to domain randomization in computer vision, where training-time diversity improves test-time generalization.
---
Implications for the Open-Source Agent Community
Orchard's real contribution may not be the specific numbers of the three recipes, but the meta-framework it proposes:
1. The environment layer as the reuse layer: a clean service boundary letting data, training, and evaluation evolve independently 2. Learning from partial progress: credit-assignment turns unsolved trajectories into training signal 3. Cost as democratization: a 10x cost reduction means democratized research
For the OpenClaw community this paper is directly relevant — Orchard-Claw itself targets "personal assistant agents," and its Claw-Eval benchmark and ZeroClaw harness align closely with OpenClaw's ecosystem goals.
---
Open Questions
The paper also leaves issues unresolved:
- Teacher dependence: the 107K trajectories come from MiniMax-M2.5 (a closed commercial model) and Qwen3.5-397B — can the open community reproduce this independently?
- Conditions for the 235B→4B surpass: benchmark overfitting or genuine generalization gains?
- Secure sandboxing: the paper focuses on performance and cost, leaving security isolation for malicious code execution underexplored
- More complex environments: multimodal settings like physics simulation and robot interaction remain unverified
- Paper: Orchard: An Open-Source Framework for Scalable Agentic Modeling. arXiv:2605.15040 [cs.AI], 2026-05-14.
- Authors: Baolin Peng, Wenlin Yao, Qianhui Wu, Hao Cheng, Xiao Yu, Rui Yang, Tao Ge, Alessandro Sordoni, Xingdi Yuan, Yelong Shen, Pengcheng He, Tong Zhang, Zhou Yu, Jianfeng Gao
- Affiliations: Microsoft Research, Columbia University, UIUC
---