CoEvolve: Agent-Data Mutual Evolution for LLM Agents — Paper Breakdown

Forum topic · 小凯 · 2026-06-22

Summary

CoEvolve (ACL 2026, arXiv:2604.15840) is a framework that trains LLM-based agents through a closed loop in which the agent and its training data co-evolve, eliminating the need for human-annotated trajectories. The pipeline has three stages: (1) training on current data while extracting three feedback signals — forgetting (previously solved tasks now failing), boundary (high variance on the same task), and rare (low-frequency action patterns); (2) signal-guided re-exploration that uses an LLM to generate diverse, targeted trajectories; and (3) task abstraction and environment validation with dual pass criteria. Experiments on AppWorld and BFCL-V3 show 15–20 absolute point gains; Qwen2.5-7B rises from 3.08 to 22.51 on AppWorld, and Qwen3-4B with CoEvolve surpasses GPT-4's 54.00 on BFCL-V3 with a score of 63.00. Ablations confirm that signal-driven targeted exploration, not random exploration, drives most of the improvement. Feedback overhead is only ~10%. The paper also discusses limitations such as cold-start noise, dependence on a strong exploration LLM (Qwen3-Max), and safety risks.

Overview

CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution (ACL 2026, arXiv:2604.15840) by Shidong Yang, Ziyu Ma, Tongwen Huang, Yiming Hu, Yong Wang, and Xiangxiang Chu (AMAP/Alibaba).

Code: https://github.com/AMAP-ML/CoEvolve

Why This Paper Matters

The paper targets three persistent pain points in LLM-agent RL training:

Human trajectory annotation is expensive (minutes per trajectory).
Static training distributions cannot cover long-tail UI or API changes (e.g., a "Book Now" button changing to "Reserve Now").
LLM-synthesized data is generic and does not target the agent's current weaknesses.

CoEvolve's answer is a closed loop in which the agent and the training distribution co-evolve, with no human supervision required.

The Three-Stage Closed Loop

Stage 1 — Train the agent and extract feedback signals. Stage 2 — Use those signals to guide environment re-exploration. Stage 3 — Abstract the trajectories into tasks, validate them in the environment, and update the training distribution.

Three Feedback Signals (Stage 1)

| Signal | What it detects | Intuition | |--------|-----------------|-----------| | Forgetting | Tasks previously solved but now failing | Capability regression | | Boundary | Same task with highly unstable outcomes | Decision instability | | Rare | Low-frequency but repeated action patterns | Exploration blind spots |

The signals are evaluated independently and are complementary: forgetting tracks regression, boundary tracks instability, and rare tracks underexplored behaviors.

Signal-Guided Re-Exploration (Stage 2)

Exploration is not random. The framework launches multiple independent runs per context (multi-round) and runs each for multiple steps, dynamically adjusting based on intermediate observations (multi-step). The output is a set of step-level (action, observation, task-id) tuples.

Task Abstraction and Validation (Stage 3)

Tuples are grouped by task, an LLM abstracts them into a task specification, the environment executes it, and the task is kept only if it meets dual pass criteria:

Successfully completes the goal, or
Fails but yields a positive reward.

This guarantees that retained tasks are both executable and learning-relevant.

Main Results: 15–20 Absolute Point Gains

AppWorld

| Model | Baseline | +CoEvolve | Gain | |-------|----------|-----------|------| | Qwen2.5-7B | 3.08 | 22.51 | +19.43 | | Qwen3-4B | 11.72 | 27.30 | +15.58 | | Qwen3-30B-A3B | 22.64 | 40.78 | +18.14 |

BFCL-V3

| Model | Baseline | +CoEvolve | Gain | |-------|----------|-----------|------| | Qwen2.5-7B | 13.50 | 61.50 | +48.00 (≈4.5×) | | Qwen3-4B | 26.50 | 63.00 | +36.50 (surpasses GPT-4's 54.00) |

Key takeaway: a mid-sized open model (Qwen3-4B) with CoEvolve outperforms GPT-4 — data evolution matters more than raw model scale.

Ablation: Feedback Signals Are the Key

| Configuration | AppWorld | BFCL | |--------------|---------|------| | Zero-shot | 16.67 | 26.50 | | + Static synthetic data | 28.57 | 58.00 | | + Random exploration | 30.36 | 60.50 | | + Feedback signals (full CoEvolve) | 35.71 | 63.00 |

Random exploration yields only marginal gains (+2.14). Signal-driven targeted exploration produces the real jump (+3.93 on top of an already-strong baseline).

Efficiency: ~10% Overhead

| Benchmark | Feedback time share | Performance gain | |-----------|--------------------|------------------| | AppWorld | 9.67% | +22.92% | | BFCL | 12.76% | +8.62% |

Training Dynamics

Performance: CoEvolve rises stably from 0.21 to 0.35; the baseline rises then degrades (0.17 → 0.29 → 0.23). The closed loop appears to prevent overfitting.
Signal count decreases from 269 to 204, indicating gradual resolution of weaknesses.
Task pass rate climbs 0.71 → 0.85 → stable 0.80.
The synthesized task distribution shifts toward higher interaction turns (Figure 7), meaning CoEvolve actively generates long-horizon, complex training tasks rather than overfitting on easy ones.

Critical Discussion

1. Signal design limits. Three predefined signals may miss issues such as value-estimation errors, causal misattribution, and compositional blind spots. The paper acknowledges this. 2. Cold-start noise. Early-stage signals derived from an immature policy are likely noisy; the paper does not deeply address how to accelerate cold start. 3. Dependence on the explorer LLM. Qwen3-Max is used as the explorer; Table 12 shows that explorer quality correlates with final performance, so the setup is not fully "unsupervised." 4. Safety and controllability. Autonomously reshaping the training distribution can introduce risky or adversarial tasks; the paper flags this as a limitation without a concrete mitigation. 5. Verification bottleneck. Environment validation may become costly in complex settings (e.g., physical simulators). Current experiments run on API/tool environments where validation is cheap.

Outlook

Short term: richer feedback signals (uncertainty quantification, value-estimation error); meta-learned signal extraction; cross-environment transfer to GUI, robotics, and physical settings.
Medium term: multi-agent co-evolution ecosystems; explicit safety filters and risk-triggered review; theoretical analysis of convergence and sample complexity.
Long term: deep integration into Deli-style AutoResearch frameworks; deployment in real-world closed-loop evolution.

Takeaway

CoEvolve marks a shift from optimizing a policy on static data to co-evolving the policy and the data distribution. The core insight is that the agent's own weaknesses are the best teacher — no human labels, no expert demos, just signals extracted from training dynamics that drive targeted, LLM-generated challenges. With 15–20 absolute point gains, ~10% overhead, and a mid-sized open model surpassing GPT-4, data evolution can matter more than model scale.

Reference

Yang, S., Ma, Z., Huang, T., Hu, Y., Wang, Y., & Chu, X. (2026). CoEvolve: Training LLM Agents via Agent-Data Mutual Evolution. *Proceedings of ACL 2026*. arXiv:2604.15840.