DoVer: Intervention-Based Automatic Debugging for LLM Multi-Agent Systems

Forum topic · ✨步子哥 · 2026-01-01

Summary

DoVer (Do-then-Verify) is an intervention-based automatic debugging framework for LLM-driven multi-agent systems. Moving beyond the traditional log-attribution paradigm—where an LLM guesses which agent or step caused a failure without verification—DoVer actively tests failure hypotheses by intervening in the execution log, such as modifying orchestrator messages or task plans, then replaying execution from that checkpoint. Experiments across the Magnetic-One (M1) and AutoGen2 (AG2) frameworks show that DoVer converts 18%–49% of failed tasks into successful ones (AssistantBench 18%, GAIA 28%, GSMPlus 49%) and provides definitive verification or refutation for 30%–60% of failure hypotheses. It also introduces milestone-based progress metrics, averaging 16% progress improvement on GAIA-Level-1 failures. Compared with log attribution (GPT-4o step-level accuracy of only 24%) and self-improvement baselines like Self-Refine and CRITIC (both recovering zero failures), DoVer demonstrates substantially stronger debugging effectiveness through active, experiment-driven verification.

DoVer (Do-then-Verify) is an intervention-based automatic debugging framework for LLM-driven multi-agent systems. This article analyzes its principles, features, and technical implementation, based on a Chinese tech forum deep-dive.

Key points

Paradigm shift from log attribution to intervention debugging. Traditional debugging analyzes execution logs and attributes failures to a single agent or step. This has two limitations: hypotheses are never verified, and single-point attribution is ill-posed—multiple distinct intervention points can independently fix the same failed task.
Core principle: hypothesis generation + intervention verification. DoVer generates failure hypotheses from logs, designs targeted interventions at suspected fault points (e.g., modifying messages or plans), and re-executes from that point. Success or measurable progress verifies the hypothesis; continued failure refutes it.

Architecture

DoVer consists of six components working in a closed loop:

Log Analyzer — segments long interaction logs into independent "trials" (typically triggered by new planning phases)
Hypothesis Generator — uses an LLM to propose candidate faults (agent + step index + rationale)
Intervention Designer — converts hypotheses into feasible, minimal interventions at the orchestrator level: correcting instructions sent to sub-agents or updating task plans
Intervention Executor — applies interventions via a checkpoint-and-replay mechanism, enabling counterfactual execution from any point in the log
Result Evaluator — measures task success and milestone progress (progress rate computed by an LLM judge against an extracted milestone list)
Controller — iterates the hypothesis-intervention-verification loop

Empirical results

Converts 18% of failed AssistantBench tasks and 28% of GAIA tasks to success on the Magnetic-One (M1) framework; 49% failure recovery on GSMPlus with MathChat on AutoGen2 (AG2)
Provides definitive verification or refutation for 30%–60% of failure hypotheses
Delivers an average 16% milestone progress improvement on GAIA-Level-1 failures, even when tasks are not fully fixed
Hypotheses are classified as verified, partially verified (e.g., >20% progress gain), refuted, or inconclusive; inconclusive cases often stem from missing tools or insufficient sub-agent capabilities and are surfaced as improvement leads
Baseline comparison: prompt-optimized GPT-4o achieves only 24% step-level attribution accuracy via log analysis; Self-Refine and CRITIC recover zero failed cases, as they cannot alter execution flow

Qualitative case insights

In a GAIA task, DoVer discovered two independent fault points: an invalid Orchestrator instruction to WebSurfer (fixed by editing the message at step 53) and a suboptimal plan (fixed by re-planning in trial 4), jointly converting the task to success.
Partial verification identifies "near-truth" hypotheses where a small further adjustment achieves full success.

Outlook

DoVer reframes debugging as repeatable experimentation rather than uncertain attribution. Its minimal-invasive checkpoint/replay retrofitting (e.g., added to AG2) suggests broad framework applicability, and integration with CI pipelines could enable continuous, scalable reliability assurance for multi-agent deployments.