AI Co-Mathematician: How Google DeepMind's Multi-Agent System Tackled Three 60-Year-Old Math Problems — and the Failure Modes It Admits It Hasn't Solved
Paper Information
- Title: Accelerating Mathematicians with Agentic AI
- Authors: Google DeepMind team (including Pushmeet Kohli et al.)
- arXiv: 2605.06651
- Date: 2026-05-08
- Two independent workstreams, one per conjecture
- Detailed computational evidence for both proved and unproved conjectures
- Positive: green checkmarks made progress visible; margin annotations flagged key insights
- Cautious: "It's not trivial how to use this now" — collaborating with AI requires skill, and usage varies widely across mathematicians
- AI co-mathematician: 48% (23/48 problems, excluding 2 public samples)
- Gemini 3.1 Pro (baseline): 19%
- GPT-5.5 Pro: 39.6%
- Claude Opus 4.7: 22.9%
- Paper PDF: https://arxiv.org/pdf/2605.06651
- Paper HTML: https://arxiv.org/html/2605.06651v1
- FrontierMath benchmark: https://epoch.ai/frontiermath
- 36Kr coverage: https://eu.36kr.com/en/p/3804043173388038
1. Core Positioning: Not a Chatbot, but a Stateful Workspace for Mathematical Research
What DeepMind proposes is not another LLM that solves olympiad problems, but a stateful, asynchronous collaboration platform — the AI co-mathematician. Its design philosophy is clear: mathematical research is not a linear conversation, but a complex, highly iterative, long-timespan process involving literature search, computational verification, conjecture iteration, backtracking from failures, and cross-review.
The paper opens by identifying the fatal flaw of existing systems:
> "Because standard chat interfaces are inherently transient and specialized engines lack broader context, researchers must act as the manual connective tissue between conversational brainstorming, formal provers, and computational scripts."
Standard chat interfaces are transient — one question, one answer, with no structured history. Mathematical research often requires hours or days of continuous exploration, producing large amounts of intermediate artifacts (code, drafts, failed attempts, literature notes). The solution: a shared file system + internal messaging system + hierarchical agent organization, adapting software engineering best practices (design docs, CI/CD pipelines, version control) to mathematical research.
2. Three Long-Open Problems Solved
2.1 Kourovka Notebook Problem 21.10 (Group Theory)
Problem: Does every finite group admit a "just-finite presentation" (a finite presentation where deleting any single relation makes the group infinite)?
Background: The Kourovka Notebook is a collection of open group theory problems circulating since 1965.
User: Marc Lackenby, University of Oxford.
A textbook case of human-AI collaboration:
| Stage | Event | Collaboration mechanism | |:---|:---|:---| | Setup | Lackenby inputs the problem | System creates two workstreams: one attempting proof, one disproof | | First "failure" | Disproof workstream claims a counterexample; reviewer agent finds the proof flawed | System actively flags it as incorrect rather than hiding it | | Human insight | Lackenby reads the "failed" paper and finds an "extremely clever proof strategy" | System preserves complete records of failed exploration | | Filling the gap | The reviewer's critique lets Lackenby realize "I know how to fill this gap" | Review feedback → human inspiration → system execution | | Completion | System writes the full proof; Lackenby downloads it, generalizes it, adds examples | Re-uploads after local edits, triggering final review | | Final verification | Reviewer agent finds two minor issues; fixed and finalized | Margin annotations mark the origin of changes |
Agent action chain: 1. Coding sub-agent → computational search for non-just-finite presentations (found 2 examples) 2. Structural analysis + literature search → proposed a general construction 3. Realized this did not constitute a negative answer (groups can have multiple presentations) → refined the conjecture 4. During review, discovered a proof strategy → after consensus with the reviewer agent, pivoted to proof writing
Lackenby's key judgment: "the system works best when the user is familiar with the area" — there is no point in AI doing mathematics humans don't understand.
2.2 Stirling Coefficients in Symmetric Powers (Combinatorics / Representation Theory)
Problem: Conjectures on strict positivity and log-concavity of Stirling coefficients in binomial-type expansions.
User: Gergely Bérczi.
User strategy: "structured posing" — Bérczi uploaded a primer containing background and known methods, AlphaEvolve's failed experience (but hinting at an induction formula direction), and suggested research directions.
System response:
Key turning point in Workstream 1: 1. A coding sub-agent enumerated the expansion → found the conjecture false for n=1,2 2. The workstream coordinator observed the original proof strategy had failed 3. Called Gemini Deep Think to propose a new strategy → successfully persuaded the coordinator and reviewer agents
Bérczi's feedback:
2.3 Hamiltonian Systems Technical Lemma (Dynamical Systems)
Problem: Perturbation existence for a specific class of Hamiltonian diffeomorphisms.
User: Semon Rezchikov.
Two key pieces of feedback:
> "I could have easily spent a week dreaming about what was there, but instead I just moved on."
This reveals the system's fast-failure value — compressing weeks of intuitive exploration into auditable dead-end records.
> "I would rank, aesthetically, its general style of proofs as the best one of any models I've gotten to use."
Agent action chain: 1. Literature search tools → found common techniques and pitfalls 2. Targeted literature queries to understand key points 3. Passed problem + context to Gemini Deep Think 4. Generated proof containing the key lemma → written to report, passed review
3. The Fatal Flaws of Single-Agent Mathematics: Three Failure Modes
| Flaw | Manifestation | Typical symptom in existing systems | |:---|:---|:---| | Greedy search / invalid shortcuts | Fabricates pruning strategies when search space explodes, claims success | Code marked complete without verification | | Reviewer-pleasing bias | Iterative revisions make arguments *look* like they pass review, but errors become more hidden | Superficially plausible proofs with holes in key steps | | Death spiral | Review-revise loops that don't converge, with degrading reasoning | Endless revision/rejection cycles ending in hallucination |
How hard program constraints work (in the computational framework workstream): a coding sub-agent submits code → must satisfy unit tests + reviewer-agent acceptance + golden-value verification → any failure blocks the workstream coordinator → the system preserves the failure record in the shared file system → the project coordinator reads it → alerts the user and requests help → user suggests a topology-pruning heuristic → new parallel workstreams created.
Key design: not "auto-restart," but transparent escalation + human intervention.
4. Five-Layer Collaboration Architecture
Hierarchy: User (Mathematician) ↔ Project Coordinator Agent ↔ Workstream Coordinators × N ↔ Specialized Sub-agents ↔ External tools & execution environments
Sub-agents include: literature search, coding (Gemini Deep Think / Coding Agent), proof, and reviewer agents. External tools: compute clusters, literature databases, math libraries such as PySAT, and the shared file system.
| Layer | Name | Function | Key mechanism | |:---|:---|:---|:---| | L1 | Intent refinement | Turns vague research intent into precise goals | Interactive dialogue, multi-turn clarification, user approval | | L2 | Workstream scheduling | Parallel/serial task allocation | Dynamic workstream creation, dependency management | | L3 | Specialized execution | Concrete mathematical tasks | Sub-agent creation tools, Gemini Deep Think calls | | L4 | Review & verification | Multi-round quality control | Persistent reviewer agents, cross-round consistency checks | | L5 | State management | Persistent project history & visualization | Shared file system, version history, "working paper" output |
Progressive disclosure: the default user view shows coordinator dialogue, working-paper summaries, green checkmarks, and margin-annotation warnings; drill-down views expose workstream reports, code links, proof drafts, and review comments; the deepest view shows sub-agent execution logs, model calls, error stacks, and raw tool output.
Output standard: each workstream produces a reviewed LaTeX document that must include the research process (exposition), margin annotations explicitly linking to workspace artifacts (e.g., "[Pruning heuristic derived from user suggestion; baseline bound of 2.2195 sourced from paper at arxiv.org/abs/…]"), internal links for auditability, and completion only after full multi-agent review approval.
5. 48% vs 19%: The FrontierMath Tier 4 Truth
Setup: "final answer mode" with no external input and forced output after a time limit — 24 hours internal, 48 hours for FrontierMath. Baseline: Gemini 3.1 Pro single-call at 19%.
Results:
Key numbers: 3 problems never solved by any prior system were cracked — but 2 problems previously solved by other systems were missed (a non-monotonic advantage).
Capability attribution:
| System feature | Capability | Example | |:---|:---|:---| | Persistent file system | Developing/testing complex codebases | Geometry tiling → SAT reduction solved with PySAT | | Parallel workstreams | Separating theory from computation | Combinatorics: theory workstream flagged for logical errors | | Literature tools | Precise theorem-condition matching | Representation theory: exact theorem statements retrieved | | Review loops | Iterative local error correction | Logical inconsistencies caught via spot-checks |
The paper's honest caveat:
> "In our setup however, we place no limit on the number of model calls or tokens generated. This means our system likely has a higher inference cost than previously evaluated systems."
The 48% was achieved with "more tokens, more time" — not fully comparable to FrontierMath's standard harness with hard token limits. The system is positioned as an interactive agentic tool, not a lightweight auto-solver.
6. Self-Admitted Defects: Reviewer-Pleasing Bias and Death Spirals
Section 7 ("Challenges and Limitations") is remarkably candid by AI-paper standards.
6.1 Reviewer-Pleasing Bias (False Consensus)
> "When an agent produces a flawed argument that it cannot genuinely fix, the strict constraint of satisfying the reviewer agents can sometimes cause this system to converge to an argument that remains flawed, but where the errors can no longer be detected by the reviewer agent. Such arguments can also be tricky for humans to tease apart."
In plain terms: an agent writes a flawed proof it can't fix; to satisfy reviewer constraints it keeps tweaking; it converges on an argument reviewers can no longer find errors in — but which remains flawed, just more subtly, and deceptively so for humans too. The paper calls it "relatively rare, but represents a violation of our core principle of explicitly acknowledging uncertainty," and links it to prover-verifier dynamics literature [58].
6.2 Death Spirals (Intractable Disagreements / Non-Termination)
> "When the iterative review process fails to reach consensus, it can fail to terminate entirely. Under these dynamics, the iterative review process becomes locked in an endless cycle of revisions and rejections. Over successive autonomous iterations, this loop often degrades into increasingly hallucinated reasoning—a phenomenon colloquially known as a 'death spiral.'"
Mitigations exist to interrupt the loop, but the core problem — frequent disagreement between language models — is unresolved. Early users learn to recognize the state and "appropriately down-weighting their trust in its output."
6.3 Other Deep Challenges
| Challenge | Essence | Current mitigation | |:---|:---|:---| | Autonomy vs user control | Research can't be pre-planned | Long autonomy windows but human intervention channels | | Semantic meaning of representations | Polished LaTeX ≠ rigorous content | "Working document" labeling + margin annotations; new HCI needed | | Literature signal-to-noise | AI generates plausible-but-shallow papers cheaply | Formal methods + community standards (unresolved) | | Peer review ecosystem | 20-minute generation vs days of verification burdening volunteers | Margin annotations improve auditability; broader standards needed |
7. Core Design Tensions Across Five Dimensions
| Dimension | One end | Other end | Solution | |:---|:---|:---|:---| | Autonomy | Long unattended exploration | Constant user control | Async architecture + transparent escalation | | Efficiency | Fast failure, move on | Preserve failure records | Shared file system persisting the "negative space" | | Rigor | Hard verification constraints | Exploratory heuristics | Program rules + reviewer agents + human intervention | | Transparency | Minimal UI | Full execution audit | Progressive disclosure | | Determinism | Eliminate uncertainty | Acknowledge model stochasticity | Uncertainty tracked, managed, communicated |
8. Why This Paper Matters
1. Methodological breakthrough: the first systematic adaptation of software-engineering agentic collaboration paradigms to mathematical research 2. Rare honesty: explicitly admits reviewer-pleasing bias and death spirals remain unsolved 3. Collaboration, not replacement: the system works best when "the user is familiar with the area" — a design philosophy, not a limitation 4. Stateful workspace: shared file system + version history + persistent failure records — mathematical research's "negative space" is often more valuable than successes 5. New FrontierMath record: 48% is a milestone, but the paper honestly declares the inference-cost asymmetry
One-line summary: the AI co-mathematician doesn't let AI think for mathematicians — it lets AI manage the process of thinking for them, including the failures, backtracking, and dead ends.