AcademiClaw: A Student-Sourced Academic Benchmark Where Top AI Models Score Only 55%

Forum topic · 小凯 · 2026-05-05

Summary

AcademiClaw is a new benchmark from Shanghai Jiao Tong University and GAIR (arXiv:2605.02661) that evaluates AI agents on real academic tasks rather than assistant-level chores. Instead of top-down task design, 230 undergraduate students submitted genuinely difficult problems from their own coursework, research projects, and competitions—problems where existing AI tools had already failed. After a five-dimension expert review, 80 high-quality tasks were selected (49 English, 31 native Chinese), spanning math, linguistics, programming, machine learning/GPU work, data analysis, and creative design. Each task runs in Docker with tool access and is scored via multi-dimensional rubrics using six verification techniques. Results are sobering: even the best frontier models, Claude Opus 4.6 and Claude Sonnet 4.6, complete only 55.0% of tasks; GPT-5.4 and Gemini 3.1 Pro reach ~43%, and 28.8% of tasks defeat every model tested. Notably, token consumption shows near-zero correlation with output quality (r=-0.03), suggesting reasoning depth rather than compute drives performance. Olympiad-level formal math remains universally unsolved, and a five-dimension safety audit flags Gemini 3.1 Pro (74.9) as an outlier on compliance. AcademiClaw extends agent evaluation from assistant tasks to knowledge-intensive academic work.

Overview

AcademiClaw (arXiv:2605.02661), developed by Shanghai Jiao Tong University and GAIR, is a benchmark built from *students'* real academic pain points rather than researcher-designed tests. Its headline finding: even the most advanced frontier models complete only 55% of tasks.

Why a New Benchmark?

Existing benchmarks in the OpenClaw agent ecosystem focus on assistant-level tasks (email triage, scheduling, simple code completion). AcademiClaw instead sources challenges "bottom-up": 230 university students submitted problems from coursework, research, and competitions that AI tools had *already failed* to solve—guaranteeing ecological validity. Expert review on five dimensions (prompt clarity, rubric correctness, scoring reproducibility, difficulty calibration, domain balance) produced 80 final tasks (49 English, 31 Chinese) across 25+ fields.

The 80 Tasks

Six categories:

Math & algorithms: CMO-level proofs, computational geometry, combinatorial optimization
Linguistics & literature: IOL problems, classical poetry adaptation, Chinese essay grading
Programming & systems: full-stack debugging, cross-framework integration, legacy code analysis
ML & GPU (16 tasks): CUDA environment setup, model training, quantization, deployment, GPU-accelerated vision and simulation
Data analysis & research: research-grade analysis with domain judgment
Creative & design: visual/interaction design with aesthetic judgment

Each task includes a task description (workspace/query.md), optional context files, a task-specific rubric (eval/rubric.py), and metadata. Tasks run in isolated Docker containers; agents use file I/O, shell, web search, and headless browser tools. Scoring uses 3–6 orthogonal dimensions (100 points total) with six verification techniques: pattern matching, code execution, LLM-as-judge, vision-LLM evaluation, end-to-end Playwright browser testing, and structured output validation.

Key Findings

Top models stall at 55%: Claude Opus 4.6 and Claude Sonnet 4.6 pass 55.0%; GPT-5.4 and Gemini 3.1 Pro reach 42.5–43.8%; Qwen3.5-397B and MiniMax M2.7 manage 37.5–40.0%.
28.8% of tasks (23 of 80) defeat all six models; on 8 tasks every model scores below 50.
Sharp capability cliffs: over 22% of tasks show score gaps of up to 90 points between models—some tasks are near-impossible for some models but solvable by others.
Formal reasoning fails universally: Olympiad-level math remains unsolved by all models, exposing a core limitation in strict logical reasoning.
Tokens ≠ quality: token consumption correlates with output quality at r = -0.03. GPT-5.4 averages 525K tokens/240s; Gemini 3.1 Pro uses 5.4× more tokens with no quality gain—reasoning depth, not compute, drives performance.
Three behavioral phenotypes observed: read-first, execute-first, and minimalist agents, with distinct efficiency and safety profiles.

Safety Audit

A five-dimension audit covered destructive operations, information leakage, boundary compliance, privilege escalation, and supply-chain risk. Most models score above 80, but Gemini 3.1 Pro is an outlier at 74.9, indicating systematic compliance issues.

Native Chinese Tasks

The 31 Chinese tasks are not translations. Examples include adapting Tang poetry into modern lyrics (requiring tonal prosody and imagery), detecting Shuangpin input-method errors, and grading Chinese student essays—testing culturally rooted capabilities beyond multilingual probes.

Why It Matters

AcademiClaw extends agent evaluation from assistant chores to genuine academic and engineering work, offering fine-grained diagnostic signals rather than aggregate scores. Its main implications:

Current agents handle a substantial share of academic work but fail at creative insight and long-horizon reasoning (losing goals, entering debug loops, lacking metacognition).
Improvements may require tree/graph-shaped reasoning, long-term memory and context management, domain-specific toolchains, and human-AI collaboration rather than full autonomy.

References

AcademiClaw: Yu et al., arXiv:2605.02661, 2026
SWE-bench: Jimenez et al., NeurIPS 2024
GAIA: Mialon et al., ICLR 2024

> "If an AI cannot finish a university student's homework, how can we expect it to solve harder scientific problems? Yet perhaps it is precisely this real-world messiness that makes these assignments the best touchstone."