Agent0 and Agent0-VL: Zero-Data Self-Evolving AI Agents with Tool-Integrated Reasoning

Forum topic · ✨步子哥 · 2025-11-26

Summary

Agent0 is a self-evolving agent framework that trains large language models without human-annotated data. It co-evolves two agents from a single base model (e.g., Qwen3-8B-Base): a Curriculum Agent that generates increasingly difficult frontier tasks, and an Executor Agent that solves them via tool-integrated reasoning with a code interpreter. Rewards rely on self-consistency uncertainty (peak near p=0.5), tool-usage incentives, and repetition penalties, optimized with GRPO plus an Ambiguity-Dynamic Policy Optimization (ADPO) scheme that scales advantages by pseudo-label confidence. Iterative training yields monotonic gains: Qwen3-8B average math benchmark scores rise from 49.2 to 58.2, and AIME24/25 roughly doubles. Agent0-VL extends the paradigm to vision-language reasoning with a unified Solver-Verifier architecture, process-level rewards, and confidence-gated Self-Repair that injects PATCH-style corrections. It improves MathVista from 67.8 to 75.6 and HallusionBench from 65.0 to 72.9, and as a process reward model adds ~7.3% gains to other models. The papers (arXiv:2511.16043, arXiv:2511.19900) suggest a shift from data-driven to self-evolving AI, while noting risks of distribution shift and tool unreliability.

Overview

This post is an in-depth Chinese-language analysis of two related papers: Agent0 (arXiv:2511.16043v1) and Agent0-VL (arXiv:2511.19900v1), which describe frameworks for self-evolving AI agents trained from zero human-annotated data — meaning no human supervision labels, though the models still learn through self-generated tasks, self-evaluation, and self-correction.

Key points

The data dependency problem: Current LLM agents rely on massive human-labeled datasets (e.g., behind OpenAI's Deep Research or Google's Agentic Coding), creating a scalability bottleneck tied to the pace of human knowledge production. RLHF paradigms cap AI potential accordingly.
Dual-agent co-evolution in Agent0: From one base model (Qwen3-8B-Base), two agents emerge:
A Curriculum Agent that proposes increasingly hard "frontier tasks". Its reward peaks when the Executor's self-consistency is near $p \to 0.5$ over $$k=10$$ samples — i.e., tasks of just the right difficulty.
An Executor Agent trained with GRPO to solve these tasks, using majority voting to produce pseudo-labels $\tilde{y}$ instead of human ground truth.
Tool integration: The Executor can pause reasoning to run Python in a sandbox (multi-turn rollout), mixing intuitive language-model reasoning with precise computation. The Curriculum Agent gets a tool-usage reward $R_{tool}(x; \pi_\phi) = \gamma \cdot \min(N_{tool}(y), C)$ with cap $$C=4$$ , encouraging tasks that genuinely require tools.
Compound reward and ADPO: The Curriculum reward is $R_C(x_i) = R_{format}(x_i) \cdot \max(0, (\lambda_{unc}R_{unc} + \lambda_{tool}R_{tool}) - R_{rep})$ , where the uncertainty reward is a peaked inverted-V at $$p=0.5$$ , and a repetition penalty $R_{rep}(x_i) = \lambda_{rep} \frac{|C_k|}{B}$ enforces task diversity. For the Executor, ADPO (Ambiguity-Dynamic Policy Optimization) scales advantages by pseudo-label confidence and widens trust-region clipping for high-ambiguity tasks, letting rare creative reasoning paths escape suppression by static PPO clipping.
Evidence for the virtuous cycle:
Fixing a generation-1 Executor against a generation-3 Curriculum drops pass rate 64% → 51% while tool calls rise 1.65 → 2.60 (difficulty scales with tool reliance).
Qwen3-8B math average climbs 55.1 → 56.5 → 58.2 across iterations with no plateau.
Ablations: removing curriculum training costs 9.3%, tool reward 7.2%, repetition penalty 5.4%.

Agent0-VL: vision-language extension

Purely textual self-evaluation fails on geometry and chart reasoning (linguistic shortcuts, hallucinated spatial relations). Agent0-VL unifies a Solver and Verifier in a single model switched by a role indicator $m \in \{S, V\}$ .
The Verifier emits structured per-step feedback (score, confidence, critique, tool_check) and can re-run tools to cross-check — a process-level reward: $r^{(t)}_{proc} = \lambda_{tool} \cdot r_{tool,t} + score_t \cdot conf_t - \beta_{div} D_{KL}(\pi^V_\theta \parallel \pi^E_\theta)$ .
Confidence-gated Self-Repair: when Verifier confidence falls below $\tau_c=0.7$ , a sigmoid gate $g_t = \sigma(\kappa(\tau_c - conf_t))$ opens and a minimal PATCH-style correction is injected into the Solver's reasoning chain. A showcased case: the model corrects a misidentified blind-spot quadrant and lands on the right answer (8.13) with no human intervention.

Reported results

Math (Agent0): average across 7 datasets (MATH, GSM8K, etc.) 49.2 → 58.2 (+18%); AIME24/25 13.9 → 28.0.
General reasoning: 42.1 on SuperGPQA/MMLU-Pro-style suites, beating Socratic-Zero which relies on the OpenAI API.
Vision-language (Agent0-VL): MathVista 67.8 → 75.6; HallusionBench 65.0 → 72.9. As a process reward model scoring other models, it adds ~7.3% average gains.
Iteration curves show monotonic improvement over three rounds with no ceiling observed.

Discussion and outlook

The author frames this as a paradigm shift from "data alchemy" to "self-evolving biology": rather than compressing human knowledge, agents could explore it, with near-zero marginal scaling cost and interpretable tool-call traces. Open challenges include distribution shift (self-generated data may cause mode collapse; mitigated by repetition penalties and dynamic clipping but not eliminated) and tool reliability (models might exploit interpreter bugs). Future multimodal work could integrate 3D renderers and physics simulators into the tool loop.

References cited in the post

1. Xia et al., *Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning*, arXiv:2511.16043v1, 2025. 2. Xia et al., *Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning*, arXiv:2511.19900v1, 2025. 3. Shao et al., *DeepSeekMath*, arXiv:2402.03300 (origin of GRPO). 4. Qu et al., *Vision-Zero*, arXiv:2503.10433. 5. Zhang et al., *Qwen3 Technical Report*, arXiv:2504.00212.

*Note: The figures, scores, and mechanisms above are reported by the post's author based on the cited papers and have not been independently verified.*