Agent0 and Agent0-VL: Self-Evolving AI Agents from Zero Data via Tool-Integrated Reasoning

Forum topic · ✨步子哥 · 2025-11-26

Summary

Agent0 is a self-evolving agent framework that removes the dependency on human-annotated data by letting two agents—born from the same base model (e.g., Qwen3-8B-Base)—co-evolve. A Curriculum Agent generates increasingly difficult 'frontier tasks' using an executor's self-consistency uncertainty as a difficulty signal, while an Executor Agent solves these tasks via reinforcement learning (GRPO) with majority-voting pseudo-labels and an integrated Python code interpreter for tool-augmented hybrid reasoning. Reward shaping includes uncertainty, tool-usage, and repetition-penalty terms, plus ADPO (Ambiguity-Dynamic Policy Optimization) to handle pseudo-label noise. Across 10 benchmarks, Agent0 lifts Qwen3-8B-Base from 49.2 to 58.2 average on math reasoning and nearly doubles AIME24 scores (13.9 to 28.0). Its successor Agent0-VL extends the paradigm to vision-language reasoning with a unified Solver–Verifier model, process-level rewards, tool-verified critique, and confidence-gated Self-Repair, improving MathVista from 67.8 to 75.6 and HallusionBench from 65.0 to 72.9. The papers (arXiv:2511.16043, arXiv:2511.19900) suggest a shift from data-driven training toward scalable, tool-grounded self-improvement.

Overview

This post introduces Agent0 (arXiv:2511.16043) and its multimodal extension Agent0-VL (arXiv:2511.19900), two frameworks for self-evolving LLM agents that improve without any human-annotated data by leveraging tool-integrated reasoning.

Key points

Motivation: Current LLM agents depend on massive human-labeled corpora (RLHF-style), creating a scalability bottleneck and capping AI at the pace of human knowledge updates. "Zero data" here means no human supervision labels—models generate, evaluate, and correct their own tasks.
Dual-agent co-evolution in Agent0: From one base model (Qwen3-8B-Base), two agents emerge:
A Curriculum Agent that proposes progressively harder "frontier tasks," rewarded when the executor's self-consistency approaches $p \to 0.5$ (maximum uncertainty = ideal difficulty).
An Executor Agent trained with GRPO to solve these tasks, using majority voting pseudo-labels $\tilde{y}$ instead of ground truth.
Together they form a "virtuous cycle": better tool use forces harder tasks, which in turn forces deeper tool mastery.
Tool integration: The executor embeds a Python interpreter into multi-turn rollouts, mixing linguistic intuition with exact computation (hybrid reasoning). The curriculum agent receives a tool-usage reward $R_{tool}(x; \pi_\phi) = \gamma \cdot \min(N_{tool}(y), C)$ with cap $$C=4$$ to encourage but not overuse tools.
Evidence of the upward spiral:
Fixed first-generation executors drop from 64% to 51% pass rate on third-generation tasks (difficulty grows), while average tool calls rise from 1.65 to 2.60.
Qwen3-8B math averages rise monotonically: 55.1 → 56.5 → 58.2 across iterations.
Ablations: removing curriculum training (-9.3%), tool reward (-7.2%), or repetition penalty (-5.4%) each hurts performance.
Algorithmic details: The curriculum reward combines format, uncertainty ( $R_{unc} = 1 - 2|p - 0.5|$ ), tool, and repetition terms. The executor uses ADPO (Ambiguity-Dynamic Policy Optimization), scaling advantages by confidence $$s(x)$$ and modulating trust-region clipping so low-consistency (ambiguous) samples produce gentler updates while rare creative reasoning paths can escape static clipping.
Agent0-VL for vision-language reasoning: A single model plays both Solver and Verifier roles (via role indicators $m \in \{S, V\}$ ). The Verifier produces structured step-level critiques with scores, confidence, and optional tool cross-checks (process-level rewards); Self-Repair injects PATCH-style JSON corrections into the Solver's reasoning when Verifier confidence exceeds a gate $\tau_c = 0.7$ .
Results:
Agent0: Qwen3-8B-Base average across 7 math benchmarks jumps from 49.2 to 58.2 (+18%); AIME24/25 doubles from 13.9 to 28.0. On general reasoning (SuperGPQA, MMLU-Pro) it scores 42.1, beating API-dependent Socratic-Zero.
Agent0-VL: MathVista 67.8 → 75.6; HallusionBench 65.0 → 72.9. As a process reward model it yields ~7.3% average gains for other models.
Iteration curves show monotonic improvement with no plateau over three rounds.
Outlook and risks: The paradigm shifts AI from "data alchemy" to self-directed evolution—unlimited scalability and interpretable tool traces—but faces distribution-shift/collapse risks and tool-reliability exploits (e.g., learning to abuse interpreter bugs). Future work may integrate 3D renderers and physics simulators as tools.

References cited in the post

1. Xia et al., *Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning*, arXiv:2511.16043v1, 2025. 2. Xia et al., *Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoning*, arXiv:2511.19900v1, 2025. 3. Shao et al., *DeepSeekMath*, arXiv:2402.03300, 2024 (origin of GRPO). 4. Qu et al., *Vision-Zero*, arXiv:2503.10433, 2025 (baseline). 5. Qwen3 Technical Report, arXiv:2504.00212, 2025 (base models).