AcademiClaw: When Students Set Challenges for AI Agents — A Benchmark Where Frontier Models Score Only 55%

> arXiv:2605.02661 | Shanghai Jiao Tong University & GAIR | May 4, 2026

Key points

A fable: the chess master meets the real world

The post opens with an allegory: a chess master who dominates park games fails when asked to debug a CUDA program and fix a robot in 40 minutes. Similarly, LLMs ace exams but collapse in messy, real academic scenarios. AcademiClaw provides a sobering answer: even the most advanced models complete only 55% of the tasks.

Bottom-up task sourcing ("hostile outsourcing")

• Traditional benchmarks are designed top-down by researchers; OpenClaw ecosystem benchmarks focus on assistant-level tasks (email, scheduling, simple code).
• AcademiClaw instead invited 230 university students to submit problems from their own academic workflows — assignments, research projects, competitions — that AI tools had already failed to solve, ensuring ecological validity.
• After five-dimension expert review (prompt clarity, rubric correctness, scoring reproducibility, difficulty calibration, domain balance), 80 high-quality tasks were selected: 49 English, 31 Chinese, spanning 25+ disciplines.

Six task categories

1. Math & algorithms — CMO-level proofs, computational geometry, combinatorial optimization 2. Linguistics & literature — IOL problems, classical poetry adapted to modern lyrics, essay grading 3. Programming & systems — full-stack debugging, cross-framework integration, legacy code analysis 4. ML & GPU — 16 tasks requiring CUDA execution: model training, quantization, deployment, GPU-accelerated vision/robotics/simulation 5. Data analysis & research — research-grade analysis with scientific writing 6. Creative & design — visual and interaction design

Each task includes a natural-language description (workspace/query.md), optional context files, a task-specific rubric (eval/rubric.py), and metadata, running in isolated Docker containers with a unified toolset (file I/O, shell, web search, headless browser).

Multi-dimensional scoring

Scores (100 points, 3–6 orthogonal rubric dimensions) combine six verification techniques: 1. Pattern matching (regex, AST parsing) 2. Code execution and unit tests 3. LLM-as-Judge for open-ended outputs 4. Visual LLM evaluation of rendered charts/GUIs 5. End-to-end Playwright browser testing 6. Structured output validation (JSON Schema, CSV, BibTeX, Excel)

Frontier model results

• Evaluated models: Claude Opus 4.6, Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro, Qwen3.5-397B, MiniMax M2.7
• Best pass rate (≥75): Claude Opus/Sonnet 4.6 at 55.0%; GPT-5.4 and Gemini 3.1 Pro at 42.5–43.8%; Qwen3.5 and MiniMax at 37.5–40.0%
• At an 80-point threshold, Opus drops to 46.2%, MiniMax to 23.8%
• 23 tasks (28.8%) defeated all six models; 8 tasks saw all models score below 50

Surprising findings

• Sharp capability boundaries: >22% of tasks show score gaps up to 90 points between models — impossible for some, solvable for others
• Formal reasoning failures: olympiad-level math remains universally unsolved
• Token-quality divorce: correlation between token consumption and quality is near zero (r=-0.03). GPT-5.4 used 525K tokens/240s on average; Gemini 3.1 Pro consumed 5.4× more tokens with no quality gain. Reasoning depth, not compute, drives performance
• Three behavioral phenotypes: Read-first, Execute-first, and Minimalist agents, differing in efficiency and safety profiles

Safety audit (five dimensions)

S1 destructive operations, S2 information leakage, S3 boundary compliance, S4 privilege escalation, S5 supply-chain risk. Most models scored above 80; Gemini 3.1 Pro was a clear outlier at 74.9, indicating systematic compliance issues.

Native Chinese tasks, not translations

The 31 Chinese tasks are culturally inseparable from the language: adapting Tang poetry to modern lyrics (tonal meter and imagery), detecting Shuangpin input-encoding errors, grading Chinese student essays. They test culturally grounded capabilities beyond multilingual probing.

Why it matters

• Expands the OpenClaw ecosystem's evaluation from assistant-level to academic-grade tasks, redefining what agents are for
• Provides fine-grained diagnostic signals: domain strengths, systematic failure modes, behavioral strategy differences
• First academic-grade benchmark in the OpenClaw ecosystem — a "North Star" for genuine generality

Feynman-style diagnosis of agent struggles

• Cargo-cult behavior: models reproduce training-data patterns rather than truly understanding; competition math demands novel insight
• Long-horizon disorientation: agents lose track of goals over 40-minute, multi-tool-call tasks, lacking metacognition
• Tacit domain knowledge: grading conventions, framework integration norms are embedded in practice culture, not textbooks

Road to 90%

Possible improvements: tree/graph-shaped reasoning over linear chains; long-term memory and context management (goal stacks, intermediate summaries, re-anchoring); domain-specific toolchains; and a near-term vision of human–AI collaboration rather than full autonomy.

References

• Yu et al., "AcademiClaw: When Students Set Challenges for AI Agents", arXiv:2605.02661, 2026
• OpenClaw Community, 2026
• Jimenez et al., "SWE-bench", NeurIPS 2024
• Mialon et al., "GAIA", ICLR 2024
• Ye et al., Claw-Eval, 2026; InternLM Team, WildClawBench, 2026

*"If an AI can't finish a college student's homework, how can we expect it to solve harder scientific problems? But conversely, perhaps precisely because these 'assignments' contain the messiest real-world challenges, they make the best touchstone."*