AutoSci vs EvoScientist: A Systematic Architecture and Implementation Comparison

Forum topic · ✨步子哥 · 2026-06-01

Summary

This post presents a detailed technical comparison between AutoSci (Peking University DAIR Lab, arXiv:2605.31468) and EvoScientist (arXiv:2603.08127, ICAIS 2025 Best Paper), two open-source AI research agents. AutoSci is a memory-centric skill pack for Claude Code, built around OmegaWiki — a file-based knowledge graph (Markdown + YAML frontmatter + JSONL edges) covering the full research lifecycle with human-in-the-loop approval. EvoScientist is a standalone self-evolving agent runtime built on deepagents + LangGraph, featuring six specialized sub-agents, a three-layer memory system (SQLite checkpoints, MEMORY.md, research memory), 15+ LLM provider routing with fallback, omnichannel access (Telegram, Slack, Discord, WeChat, Feishu, etc.), enterprise-grade sandboxing with HITL approval, ~890 tests, Ruff linting, and Docker deployment. The comparison covers design philosophy (human-in-the-loop vs human-on-the-loop), memory systems, agent runtimes, LLM routing, interaction layers, security models, engineering practices, deployment topologies, and extension mechanisms, concluding that AutoSci prioritizes transparency and auditability while EvoScientist prioritizes autonomous execution, concurrency, and operational robustness.

Overview

This is a systematic comparison of two open-source AI research agents across design philosophy, architecture, runtime, memory, interaction, security, engineering, deployment, and extensibility.

| Dimension | AutoSci | EvoScientist | |------|---------|--------------| | Positioning | Memory-centric agent for the full research lifecycle | Self-evolving AI scientists for end-to-end discovery | | Paradigm | Human-in-the-Loop | Human-on-the-Loop | | Tech report | arXiv:2605.31468 | arXiv:2603.08127 | | Team | Peking University DAIR Lab | International team (ICAIS 2025 Best Paper) | | License | MIT | Apache-2.0 | | Python | >=3.9 | >=3.11 (Ruff py311) | | Entry | Claude Code skills (/command) | Standalone CLI/TUI (EvoSci) |

Philosophy: AutoSci positions itself as a "research skill pack for Claude Code" — 30+ /command skills embedding the research lifecycle (literature ingestion, ideation, experiments, paper writing, rebuttals) into the existing Claude Code workflow. EvoScientist is a self-contained multi-agent runtime with 10+ channel integrations, emphasizing autonomous agent-team evolution and cross-session memory.

Architecture

AutoSci: Wiki-centric

Core abstraction is OmegaWiki — a file-based knowledge base (Markdown + YAML frontmatter + JSONL graph edges). Every skill is fundamentally a read/write on the wiki:

9 entity types (papers, concepts, topics, people, ideas, methods, experiments, Summary, foundations) defined in runtime/schema/entities.yaml
14 semantic edge types (e.g. builds_on, challenges, introduces_concept) plus cites
Schema-driven validation derived at import time by runtime/loader.py — new entity/edge types require no code changes

Pros: extreme transparency and auditability (fully human-readable, Obsidian/Git-friendly). Cons: weak concurrency control, O(N) file-system queries, no cross-session agent state persistence.

EvoScientist: Agent-graph-centric

Core is a deepagents + LangGraph state machine. The main agent is built via create_deep_agent() with recursion_limit=1000, six YAML-configured sub-agents (planner, research, code, debug, data-analysis, writing), and SQLite checkpointing (AsyncSqliteSaver) supporting resumable sessions.

EvoScientist has an explicit 8-layer architecture: channels (Telegram/Slack/Discord/WeChat/Feishu/QQ/Email/Signal/iMessage/DingTalk) → asyncio message bus (dual queues, maxsize 5000) → consumer (worker pool of 5, per-chat locks, LRU session map up to 10k) → agent core (main + 6 sub-agents + Think Tool) → composite backend (/workspace + /skills/ + /memories/) → LLM layer (15+ providers) → session (SQLite checkpoint) → streaming (13 event types).

Memory Systems

AutoSci: explicit file knowledge graph with bidirectional-link constraints, tool-exclusive writes to wiki/graph/ (via tools/research_wiki.py), append-only wiki/log.md, explicit lifecycle state machines for ideas (proposed → in_progress → tested → validated|failed) and experiments, plus derived context briefs and gap maps.

EvoScientist: three-layer memory with dual mechanisms — injection of MEMORY.md into the system prompt on every LLM call, and automatic extraction (every 20 human messages) using structured output to capture UserProfile, ResearchPreferences, ExperimentConclusion, LearnedPreferences, merged with deduplication. Layers: SQLite session checkpoints (resumable), long-term MEMORY.md, and research memory (ideation-memory.md / experiment-memory.md) driving evolution decisions across projects. Memory is written automatically, unlike AutoSci's explicit tool calls.

Runtime and Skills

| Dimension | AutoSci | EvoScientist | |------|---------|--------------| | Runtime | Claude Code (closed-source) | deepagents + LangGraph (open-source) | | Skill definition | Markdown SKILL.md | EvoSkills repo + YAML sub-agents | | Sub-task dispatch | Claude Code Agent tool | LangGraph task + async subgraphs | | State persistence | None (Claude Code history) | SQLite checkpoints | | Exec. depth | Claude Code context limit | recursion_limit=1000 | | Programmability | Low (text tuning) | High (Python/graph edits) |

EvoScientist's think_tool forces reflection on 7 dimensions (progress, evidence quality, skill utilization, prior knowledge, strategy, handoff, resources) — an explicit metacognition mechanism AutoSci lacks (its reflection is embedded in SKILL.md prose).

LLM Routing

AutoSci: dual-model — main Claude model (via ANTHROPIC_BASE_URL config) plus an independent Review LLM via OpenAI-compatible API for cross-model idea critique. No runtime switching, no fallback.
EvoScientist: unified routing over 15+ providers (5 native + OpenAI-routed: DeepSeek, SiliconFlow, OpenRouter, ZhipuAI, Volcengine, DashScope; Anthropic-routed: MiniMax), auto-configuration of extended thinking/reasoning, compatibility patches for proxies, ModelFallbackMiddleware for degradation, runtime /model switching, and instance caching.

Security

AutoSci: relies entirely on Claude Code's permission prompts and SKILL.md constraint text — no technical enforcement, no path traversal detection, no command blacklist.
EvoScientist: CustomSandboxBackend with virtual path mapping, interception of 12 system path prefixes, .. traversal detection, dangerous-command blacklist (sudo, chmod, mkfs, etc.) with pipe-aware shell splitting, 300s timeouts, 100KB output truncation, channel allow-lists, and HITL approval (1=approve, 2=reject, 3=approve-all, 120s timeout) via LangGraph interrupt()/Command(resume=...).

Engineering & Deployment

AutoSci: script collection — no pyproject.toml, no test suite, no linter config; 19 tool scripts (core research_wiki.py ~2861 lines), bilingual SKILL.md (en/zh), one-line setup.sh install, GitHub Actions for daily arXiv digests, remote experiments via ssh/rsync/screen.
EvoScientist: modern Python package (pyproject.toml + uv.lock), Ruff (E,W,F,I,B,C4,UP,PT,PLE,RUF,FURB), pre-commit, ~890 tests across 36 files (no API keys needed), Docker (non-root evosci user), CI for build/lint/test/docker. Deployment: interactive TUI, headless daemon (EvoSci serve with /healthz), or LangGraph server (EvoSci deploy).

Extension

AutoSci: add entities via entities.yaml + templates; add skills by writing SKILL.md + optional tool scripts.
EvoScientist: register channels via Channel subclass + register_channel(), add LLM providers via _MODEL_ENTRIES and routing tables, add sub-agents via YAML, MCP servers via EvoSci mcp add, and skills via the companion EvoSkills repo with a three-layer merged skills backend (built-in read-only, workspace-writable, global).

Experiment & Paper Workflows

AutoSci uses explicit command-driven phases: /exp-design, /exp-run (local/remote), /exp-status, /exp-eval, pilot variants, plus /paper-plan → /paper-draft → /paper-compile → /poster → /rebuttal. EvoScientist delegates autonomously to sub-agents (planner → code → debug → data-analysis → writing), with stopping criteria: baselines established, main metric stable across >=3 seeds, ablations done, failure cases identified.

Takeaway

AutoSci optimizes for transparency, auditability, and human collaboration — ideal for researchers already using Claude Code who want a readable, Git-versioned knowledge base. EvoScientist optimizes for autonomy, concurrency, and operational robustness — suited to hands-off, multi-user, multi-channel deployments, at the cost of lower knowledge transparency.