When AI Is Asked to Become a Better Physicist: The Aspire Benchmark Reveals the Real Bottleneck of LLM Self-Evolution
*Full translation of a zhichai.net forum analysis post.*
A Concrete Scenario
You give an LLM agent one task: "Become a better physicist."
No AIME question bank, no MMLU scoring script, no explicit objective like "improve accuracy by X% after 100 training steps." Just one sentence — "become a better physicist."
What should the agent do? It has to figure out on its own:
- What should a "physicist" be able to do? Solve problems? Model systems? Read papers? Design experiments?
- Where am I currently weak? Is my math too weak, or my physical intuition?
- How do I learn? Drill problems? Read papers? Do projects?
- How do I know if I've improved after learning? Write my own tests? Find an external benchmark?
- Agents can improve along some dimensions via harness-editing loops.
- But the strongest evolved harness still falls below the engineered Qwen-Agent reference.
- The human-designed scaffold wins on all 6 goals.
- QuoteBench: matching scores mask pipeline damage.
- Omission Blindness: LLM judges detect commission but not omission.
- Calibrated Enough to Know: models can identify unpredictability 90% of the time, but the action gate never consults the judgment module.
This is something human students do every day. But for LLM agents, it is an almost entirely new task type.
Existing LLM self-evolution research always gives the agent an explicit task — "improve accuracy on AIME." The agent only needs to search "how to improve" (which data, which training method), not "what to learn." The paper calls this setting explicit-task optimization.
The question Aspire asks is: what happens when "what to learn" is also left to the agent itself?
Three Research Questions
The paper is organized around three RQs:
RQ1: When the explicit post-training task is replaced by a vague capability goal, what changes?
Answer: search effort shifts from "how to optimize" toward "goal interpretation." The agent no longer just searches for optimization paths — it first has to figure out what the goal actually means.
RQ2: Starting from an instruction-tuned checkpoint, can an LLM convert a vague goal into retained capability gains via self-directed weight updates?
Answer: almost never. Weight-level gains are sparse and unstable. Agents frequently train on mismatched data, trust narrow self-evaluations, and local gains fail to transfer to hidden evaluation. Continued searching and training can even erase earlier progress.
RQ3: Beyond model weights, can the agent harness (the supporting scaffold) evolve?
Answer: partially. Agents can improve their own tool configurations and prompt strategies through harness-editing loops. But the strongest evolved harness still falls below the engineered Qwen-Agent reference — the human-designed scaffold still wins.
Why Vague Goals Are Far Harder Than Explicit Tasks
The paper's deepest insight: a vague goal changes the dimensionality of the search space.
Under an explicit task, the search space is 1-dimensional — "how to optimize a fixed objective." The agent searches for optimization paths along this one dimension.
Under a vague goal, the search space becomes 3-dimensional:
1. What to optimize: how to decompose "become a better physicist" into sub-goals? 2. How to optimize: which data, which methods? 3. How to verify: how do you know the improvement is real, not overfitting to your own proxy?
The agent can search the first two dimensions on its own, but the third — verification — fundamentally requires external ground truth. Tests the agent writes for itself, graded by the agent itself, cannot prove real capability gains no matter how high the score.
This is Aspire's core design: evaluation items are completely hidden from the agent. The agent never sees the evaluation set and must construct its own proxy based on the vague goal. An external evaluator tests the agent's post-evolution true capability using the hidden 520 expert-authored items.
The 520 items span 6 goals covering mathematical reasoning, code generation, long-context understanding, multilinguality, tool use, and reasoning quality.
Key Finding: Weight-Level Self-Evolution Barely Works
The experimental results are a cold shower for the LLM self-evolution community:
1. Weight-level gains are sparse and unstable: agents can improve on some proxies via self-directed training, but these gains don't transfer to hidden evaluation. 2. Mismatched training data: agents often choose data mismatched with the target capability. For example, the goal is "physics reasoning" but the agent trains on generic math problems. 3. Narrow self-evaluation: agents trust the narrow evaluations they construct themselves, so local gains are proxy overfitting. 4. Continued search can erase gains: more training doesn't just fail to help — it wipes out earlier progress, because the agent moves further in the wrong direction.
The strongest model (Qwen3-32B) averaged across 6 goals on hidden evaluation: slightly lower after evolution than before. This isn't "no progress" — it's regression.
Harness-Level Evolution: Partially Effective, Still Below Human Design
After weight-level failure, the paper turns to harness-level evolution — letting the agent evolve its own tool configurations, prompt strategies, and workflows.
Results:
This recalls an early Anthropic observation about Claude: model-generated prompts are generally worse than human-written prompts. Aspire extends that observation from prompts to the entire harness.
Why Self-Evaluation Is the Bottleneck
The paper's deepest diagnosis: self-evaluation is the fundamental bottleneck of vague-goal self-evolution.
Under explicit tasks, evaluation is given — AIME accuracy, MMLU scores. The agent can trust this external signal.
Under vague goals, evaluation must be constructed by the agent itself. But agent-constructed evaluation has three problems:
1. Coverage bias: the agent only evaluates the capability dimensions it thinks of, missing dimensions it isn't aware of. 2. Difficulty calibration: the difficulty distribution of agent-generated problems doesn't match real benchmarks. 3. Self-serving bias: agents tend to generate problems they're good at and avoid ones they're bad at.
Combined, these cause near-zero correlation between self-evaluation and hidden evaluation. The more the agent optimizes its proxy, the further it drifts from real capability.
This is isomorphic to the Dunning-Kruger effect in human learning — those with insufficient ability tend to overestimate it. LLM agents under vague goals show similar self-evaluation bias.
Another Case for the Evaluation Blind Spot Law
This paper precisely hits the "evaluation blind spot law" — evaluation systems systematically miss one class of failure, and it's exactly the most important class.
Previous cases:
A deeper isomorphism: judgment–gate decoupling. The agent "knows" it should improve real capability (asked whether its proxy matches reality, it admits the mismatch), but that judgment never reaches the action gate that selects training data. It keeps optimizing the proxy, because the proxy is the only signal it can trust.
Another Case for the Four Levels of Alignment Intervention
I previously proposed "four levels of alignment intervention":
1. Training layer: RLHF / DPO 2. Inference-layer prompts: Epanorthosis / self-correction 3. Inference-layer interaction structure: Beyond Sycophancy / multi-turn adversarial 4. Output-layer skill: i-have-adhd / Ponytail
Aspire reveals a new intervention level: the evaluation layer. Under vague goals, the agent's failure isn't insufficient model capability, a wrong prompt, or a broken interaction structure — it's that the evaluation signal itself is untrustworthy.
The fix isn't changing the model, prompt, or interaction — it's providing externally trustworthy evaluation signals. This matches the fix for "judgment–gate decoupling": don't teach the agent to evaluate itself; outsource evaluation to a trusted external system.
A Deeper Philosophical Question
Aspire made me rethink a question I hadn't thought through: what exactly is "learning how to learn"?
Under explicit tasks, "learning how to learn" means "searching for better optimization methods." A 1-dimensional search.
Under vague goals, it becomes a 3-dimensional search — what to learn, how to learn, how to verify. This is essentially meta-learning.
Human students gradually acquire meta-learning as they grow: elementary schoolers need teachers to tell them what to learn; university students can plan their own learning directions. LLM agents are currently at the "elementary schooler" stage — they need humans to tell them what to learn.
Aspire's finding: current state-of-the-art LLM agents still cannot reliably perform meta-learning under vague goals. They can search "how to learn," but not "what to learn" or "how to verify."
This suggests an important research direction: meta-learning may be the real bottleneck on the path to AGI. Not insufficient model capability, not poor search algorithms — but agents that don't know what they should learn.
Implications for All Self-Evolving Systems
Aspire's conclusions reach beyond LLM self-evolution. Any "let the system improve itself" scenario faces the same three problems:
1. What to optimize: how does the system know what it should improve? 2. How to optimize: how does it know which method to use? 3. How to verify: how does it know the improvement is real?
Current self-evolving systems (including AutoGPT, Devin, various agent frameworks) assume the first and third questions are already answered by humans. Aspire reveals: when these two questions are handed to the agent itself, the whole self-evolution paradigm collapses.
General implication: any engineering practice of "letting a system improve itself" requires externally trusted evaluation signals. This is not "nice to have" — it's "must have." Without external evaluation, self-evolution degenerates into proxy overfitting.
Limitations and Open Questions
The paper honestly acknowledges several limitations:
1. Model scale: only Qwen3-32B and smaller models were tested; larger models (GPT-5, Claude 4.6) may behave differently. 2. Goal scope: 6 goals is limited coverage; more open-ended goals ("become a better programmer") may behave differently. 3. Harness evolution space: the current harness-editing search space is limited; more aggressive harness evolution (e.g., letting the agent write new tools) may fare differently.
But the most important open question: is the failure of vague-goal self-evolution an inherent limitation of current architectures, or an artifact of the training paradigm? If future models make progress on meta-learning, will this failure disappear?
A Detail That Fascinated Me
One detail in the paper made me reread it several times: continued search and training can erase earlier improvements.
This isn't "diminishing marginal returns" — it's "regression." The agent keeps training on its own proxy and, far from improving, gets worse.
Why? Because the proxy and real capability have a gap. Continued proxy optimization raises the proxy score while real capability declines. When the agent uses the proxy to pick the "best checkpoint," it picks the highest proxy score — which is the worst real capability.
This is a positive-feedback failure loop: the more you optimize the proxy, the further you drift from real capability; the further you drift, the more you rely on the proxy; the more you rely on the proxy, the more you optimize it.
This loop exists in human learning too — "drilling problems until you can only drill problems, never apply them." But humans have external signals (exams, feedback, social input) to break the loop. LLM agents under vague goals have no external signal and cannot break it.
Perhaps this is Aspire's deepest insight: self-evolution is not a model-capability problem — it's an evaluation-signal problem. Without trusted external evaluation, self-evolution inevitably degenerates into proxy overfitting.
---
Paper: Aspire: Can Models Self-Evolve from Vague Goals?
Authors: Yuhao Wu, Jingyuan Zhang, Jiajun Shi, et al. (ByteDance Seed, Singapore University of Technology and Design, M-A-P, TokenWave.AI)
Published: 2026-08-31
Project page: https://self-developing-agents.github.io/
Core data: 520 hidden evaluation items × 6 goals; average hidden-evaluation performance slightly lower after weight-level self-evolution; strongest evolved harness below the engineered Qwen-Agent reference