ARIS: An Open-Source Framework for AI Research While You Sleep, Using Adversarial Review to Prevent Self-Deception

Forum topic · 小凯 · 2026-06-07

Summary

ARIS (Autonomous Research via Adversarial Multi-Agent Collaboration), developed by a Shanghai Jiao Tong University team (Ruofeng Yang, Yongcan Li, Shuai Li), is an open-source framework addressing the core failure mode of long-horizon AI research agents: 'plausible unsupported success' — agents produce fluent, convincing papers whose claims are fabricated or over-generalized, a problem the authors call hallucinated credibility. ARIS's core thesis is that any single-agent long-horizon task is unreliable, and cross-model adversarial review (Proposer vs. Reviewer from different model families) is the minimal configuration to break self-play blind spots. The framework features a three-layer decoupled architecture (execution, orchestration, assurance) with 65+ portable Markdown skills, five end-to-end workflows covering idea discovery through paper writing and rebuttal, a three-stage claim-integrity audit (experiment integrity, result-to-claim mapping, paper-claim audit), a five-round scientific editing pipeline, a persistent research wiki knowledge graph for spiral learning across sessions, and four adjustable effort levels. Documented overnight runs (~8 hours, 20+ GPU experiments, 4 review-revision cycles) improved reviewer scores from 5.0 to 7.5 while pruning unsupported claims. Unlike single-agent systems such as AI Scientist, ARIS engineers trustworthiness as a first-class system component.

A team at Shanghai Jiao Tong University (Ruofeng Yang, Yongcan Li, Shuai Li) has released ARIS (Autonomous Research via Adversarial Multi-Agent Collaboration), an open-source framework that lets AI agents conduct full research workflows — from idea discovery to paper writing to rebuttal — while addressing the field's most overlooked risk: agents that produce plausible but unsupported results.

Key points

Core problem — "hallucinated credibility": Long-horizon agents rarely crash; instead, they produce fluent papers with fabricated or over-generalized claims. Failures are not from lack of capability but from excessive persuasiveness that human reviewers and other agents struggle to detect.
Strict hypothesis: Any single-agent long-horizon task is unreliable. Cross-model adversarial review (Executor vs. Reviewer from different model families) is the minimal configuration to break self-play blind spots.
Three-layer architecture: Execution (65+ skills, MCP bridges, research wiki), Orchestration (5 end-to-end workflows: Idea Discovery, Experiment Bridge, Auto Review Loop, Paper Writing, Rebuttal), and Assurance (3-stage audit + 5-round scientific editing + manuscript QA).
Reviewer independence protocol: Executors provide only file paths and review goals — never summaries. Reviewers read artifacts directly. Fresh-context (new-thread) reviewers prevent narrative contamination; repository-level access enables deep code audits.
Three-stage claim integrity audit: (1) Experiment-integrity audit catches phantom results, self-normalized scores, model-derived reference labels, dead-code inflation, and scope inflation; (2) Result-to-claim mapping produces a Claim Ledger (supported / partially supported / invalidated); (3) A zero-context reviewer cross-checks every quantitative claim in the LaTeX source against raw result files.
Five-round editing pipeline: clutter removal, active voice, sentence structure, terminology consistency, and — critically — numerical consistency checks between paper text and tables/figures/result files.
Persistent research wiki: A knowledge graph of Papers, Ideas, Experiments, and Claims with eight relation types (e.g., extends, contradicts, tested_by, invalidates), enabling spiral learning so agents don't repeat failed directions across sessions.
65+ portable skills: Pure Markdown SKILL.md files work natively on Claude Code, Codex CLI, and Cursor, with community adaptations — no vendor lock-in.
Four effort levels (lite ≈0.4×, balanced 1×, max ≈2.5×, beast ≈5–8×) scale coverage, not reviewer reasoning effort (Codex reviewers always run at xhigh).

Documented overnight run

An ~8-hour run launched 20+ GPU experiments and completed 4 review-revision cycles, raising internal reviewer scores from 5.0 to 7.5. Notably, unsupported claims were actively pruned during audits — validating the core thesis that without adversarial review, those claims would have survived into the final paper.

Comparison with existing work

Unlike single-agent pipelines such as AI Scientist (Sakana) or Agent Laboratory, ARIS is built on multi-agent adversarial collaboration: one agent does the work, another actively probes its weaknesses. The design philosophy — decoupling execution, orchestration, and assurance — extends beyond research to any long-horizon, high-stakes, verification-requiring agent scenario (legal review, medical diagnosis support, financial reporting).

Limitations

High compute cost (overnight runs require many GPU experiments and 6+ model calls)
Dependence on high-capability (often closed-source) reviewer models
The "rescue agent" escalation has no higher-level arbitration if it also fails
Three-stage audits reduce but cannot eliminate correlated reviewer errors, since reviewers are themselves LLMs
Current validation is mainly in ML/CS; other domains need community-contributed skills (30+ exist)

Reference

Yang, R., Li, Y., & Li, S. (2026). *ARIS: Autonomous Research via Adversarial Multi-Agent Collaboration*. Shanghai Jiao Tong University. arXiv:2605.03042.

Project: https://github.com/wanshuiyin/Auto-claude-code-research-in-sleep