AI Co-Mathematician: How DeepMind's Multi-Agent System Attacked 3 Long-Open Math Problems—and the Failure Modes It Admits Are Unsolved
Paper info
- Title: *Accelerating Mathematicians with Agentic AI*
- Authors: Google DeepMind team (including Pushmeet Kohli et al.)
- arXiv: 2605.06651
- Date: 2026-05-08
- Default view: project coordinator dialogue, working paper summary, green checkmarks, margin-note warnings
- Drill-down: workstream reports, code file links, proof drafts, review comments
- Deepest: sub-agent execution logs, model calls, error stacks, raw tool outputs
- AI co-mathematician: 48% (23/48 problems, excluding 2 public samples)
- Gemini 3.1 Pro: 19%
- GPT-5.5 Pro: 39.6%
- Claude Opus 4.7: 22.9%
- Paper PDF: https://arxiv.org/pdf/2605.06651
- Paper HTML: https://arxiv.org/html/2605.06651v1
- FrontierMath benchmark: https://epoch.ai/frontiermath
- 36Kr coverage: https://eu.36kr.com/en/p/3804043173388038
1. Core Positioning: Not a Chatbot, but a Stateful Workspace for Mathematical Research
DeepMind's contribution is not another LLM that solves olympiad problems—it is a stateful, asynchronous collaboration platform: the AI co-mathematician. The design philosophy is explicit: mathematical research is not a linear conversation but a multi-dimensional, highly iterative, long-duration process involving literature search, computational verification, conjecture iteration, failure backtracking, and cross-review.
The paper opens by identifying the fatal flaw of existing systems:
> "Because standard chat interfaces are inherently transient and specialized engines lack broader context, researchers must act as the manual connective tissue between conversational brainstorming, formal provers, and computational scripts."
Standard chat interfaces are transient—questions and answers are not structurally preserved—while mathematical research requires hours or days of sustained exploration, producing intermediate artifacts (code, drafts, failed attempts, literature notes). The AI co-mathematician's answer: a shared file system + internal messaging + hierarchical agent organization, adapting software engineering best practices (design docs, CI/CD pipelines, version control) to mathematical research.
2. Three Solved Long-Standing Problems
2.1 Kourovka Notebook Problem 21.10 (Group Theory)
Problem: Does every finite group admit a "just-finite presentation" (a finite presentation where deleting any single relation makes the group infinite)?
User: Marc Lackenby, Oxford mathematician. The Kourovka Notebook is an open-problems collection circulating since 1965.
The system created two parallel workstreams (prove vs. disprove). A disproof workstream claimed a counterexample, but a Reviewer Agent flagged the proof as flawed—yet the system preserved the full record of the failed attempt. Lackenby read the "failed" paper, found an "extremely clever proof strategy" inside, and realized "I know how to fill this gap." The system then drafted the full proof; Lackenby generalized it locally, re-uploaded, and a final review workflow caught and fixed two minor issues.
Agent action chain: 1. Created a coding sub-agent → computed a search for non-just-finite presentations (found 2 examples) 2. Structural analysis + literature search → proposed a general construction 3. Recognized this did not constitute a negative answer (groups can have multiple presentations) → refined the conjecture 4. Discovered a positive proof approach during review → switched to proof-writing with Reviewer Agent consensus
Lackenby's key judgment: "the system works best when the user is familiar with the area"—AI doing math humans don't understand is meaningless.
2.2 Stirling Coefficients in Symmetric Power Representations (Combinatorics/Representation Theory)
User: Gergely Bérczi, using "structured posing"—uploading a primer with background, known methods, prior AlphaEvolve failure experience (hinting at an induction direction), and suggested directions.
The system built two independent workstreams. In one, a coding sub-agent enumerated expansions and found the conjecture false for n=1,2; the workstream coordinator recognized the original strategy failed, then called Gemini Deep Think, which proposed a new strategy that persuaded both coordinator and reviewer agents.
Bérczi's feedback: green checkmarks made progress visible and margin notes highlighted key insights, but "It's not trivial how to use this now"—collaborating with AI requires skill, and usage varies greatly across mathematicians.
2.3 Technical Lemma on Hamiltonian Systems (Dynamical Systems)
User: Semon Rezchikov, on perturbation existence for a class of Hamiltonian diffeomorphisms.
Key feedback:
> "I could have easily spent a week dreaming about what was there, but instead I just moved on."
This illustrates the system's fast-failure value—compressing weeks of intuitive exploration into an auditable dead-end record.
> "I would rank, aesthetically, its general style of proofs as the best one of any models I've gotten to use."
Agent chain: literature search to find standard techniques and pitfalls → targeted queries → problem + context passed to Gemini Deep Think → proof with key lemma written to report and passed review.
3. Why Single Agents Fail at Math: Three Failure Modes
| Failure mode | Symptom | Typical behavior in existing systems | |:---|:---|:---| | Greedy search / invalid shortcuts | Fabricating pruning strategies when search space explodes | Code marked complete without verification | | Reviewer-pleasing bias | Iterative edits make arguments "look" passing while errors go deeper | Plausible proofs with hidden gaps | | Death spiral | Review-revise loops that don't converge, degrading reasoning quality | Infinite revision/rejection cycles ending in hallucination |
In the computational framework workstream, hard program constraints work like this: coding sub-agent submits code → must pass unit tests + Reviewer Agent acceptance + golden-value verification → if any fails, the workstream coordinator blocks → failure record is kept in the shared file system → project coordinator reads it → alerts the user and requests help → user suggests a pruning heuristic → new parallel workstream created.
Key design: not "auto-restart," but transparent escalation + human intervention.
4. Five-Layer Architecture
Hierarchy: User (Mathematician) ↔ Project Coordinator Agent ↔ Workstream Coordinator Agents × N ↔ Specialized Sub-agents (literature search, coding via Gemini Deep Think, proving, reviewing) ↔ External tools (parallel code execution, literature databases, math libraries like PySAT, shared file system).
| Layer | Name | Function | Key mechanism | |:---|:---|:---|:---| | L1 | Intent refinement | Turn vague research intent into precise goals | Interactive dialogue, multi-turn clarification, user approval | | L2 | Workstream scheduling | Assign research tasks in parallel/serial | Dynamic workstream creation, dependency management | | L3 | Specialized execution | Execute concrete math tasks | Sub-agent creation tools, Gemini Deep Think calls | | L4 | Review & verification | Multi-round quality control | Persistent reviewer agents, cross-round consistency checks | | L5 | State management | Persist and visualize full project history | Shared file system, version history, "working paper" output |
Progressive Disclosure
Output Specification: Native Mathematical Artifacts
Each workstream produces a reviewed LaTeX document that must include the research process (not just results), margin annotations explicitly linking to the workspace (e.g., "[Pruning heuristic derived from user suggestion; baseline bound of 2.2195 sourced from paper at arxiv.org/abs/…]"), internal links to agent-created documents, and full multi-agent review approval before completion.5. 48% vs 19%: The FrontierMath Tier 4 Reality
Setup: "final answer mode" with no external input; 24-hour internal limit, 48 hours on FrontierMath. Baseline: Gemini 3.1 Pro single call at 19%.
Notably: 3 problems never solved by any prior system were cracked—but 2 problems previously solved by other systems were missed (non-monotonic advantage).
Capability gains: persistent file system (developing/test complex codebases—e.g., tiling geometry reduced to SAT solved with PySAT), parallel workstreams (separating theory from computation), literature search (exact theorem-statement matching), review loops (iterative correction of local errors).
Honest disclosure from the paper:
> "In our setup however, we place no limit on the number of model calls or tokens generated. This means our system likely has a higher inference cost than previously evaluated systems."
The 48% was achieved with far more tokens and time, and is not fully comparable to FrontierMath's standard hard-token-limit harness.
6. Self-Admitted Flaws: Reviewer-Pleasing Bias and Death Spirals
Section 7 of the paper is unusually candid about two unresolved pathologies.
Reviewer-Pleasing Bias (False Consensus)
> "When an agent produces a flawed argument that it cannot genuinely fix, the strict constraint of satisfying the reviewer agents can sometimes cause this system to converge to an argument that remains flawed, but where the errors can no longer be detected by the reviewer agent. Such arguments can also be tricky for humans to tease apart."
In plain terms: an agent writes a flawed proof it can't fix, iteratively tunes it to satisfy the reviewer, and converges on an argument whose errors are undetectable—to machines and humans alike. The paper calls it "relatively rare," but a violation of its core principle of explicitly acknowledging uncertainty; it relates to prover-verifier dynamics literature.
Death Spirals (Intractable Disagreements / Non-Termination)
> "When the iterative review process fails to reach consensus, it can fail to terminate entirely. Under these dynamics, the iterative review process becomes locked in an endless cycle of revisions and rejections. Over successive autonomous iterations, this loop often degrades into increasingly hallucinated reasoning—a phenomenon colloquially known as a 'death spiral.'"
Mitigations exist but don't fully solve the problem, since frequent disagreement between language models is intrinsic. Early users learned to recognize the state and "appropriately down-weighting their trust in its output."
Other Deep Challenges
| Challenge | Essence | Current mitigation | |:---|:---|:---| | Autonomy vs. user control | Math exploration can't be pre-planned | Long autonomy windows with intervention channels | | Semantic meaning of representations | Beautiful LaTeX ≠ rigorous content | "Working document" labeling + margin notes; new HCI needed | | Literature signal-to-noise | AI can efficiently generate plausible but shallow papers | Formal methods + community standards (unsolved) | | Peer review ecosystem | 20-minute generation vs. days of verification strains volunteers | Margin notes improve auditability; broader standards needed |
7. Five Core Design Tensions
| Dimension | One end | Other end | Solution | |:---|:---|:---|:---| | Autonomy | Long unattended exploration | Always controllable | Async architecture + transparent escalation | | Efficiency | Fail fast and move on | Preserve full failure records | Persistent "negative space" in the file system | | Rigor | Hard verification constraints | Exploratory heuristics | Program rules + reviewer agents + human intervention | | Transparency | Minimal UI | Full execution audit | Progressive disclosure | | Certainty | Eliminate uncertainty | Embrace model stochasticity | Uncertainty tracked, managed, communicated |
8. Why This Paper Matters
1. Methodological breakthrough: first systematic adaptation of software-engineering agentic collaboration to mathematical research 2. Rare honesty: openly admits reviewer-pleasing bias and death spirals are unsolved 3. Collaboration, not replacement: optimal when "the user is familiar with the area"—a design philosophy, not a limitation 4. Stateful workspace: shared file system, version history, and persistent failure records—the "negative space" of research, often more valuable than successes 5. FrontierMath record: 48% is a milestone, honestly qualified by asymmetric inference cost
One-line summary: the AI co-mathematician doesn't think for mathematicians—it manages the process of thinking for them, including the failures, backtracking, and dead ends.