A Silent 2 AM Wake-Up Call
It's 2 AM, day three after your self-improving agent went live.
On the test set, it climbed from 54.8% to 56.3%—everything looked great. But in production, it produced three completely different results on the same task across three runs—best 58%, worst 48%. You assumed it was a bug. After a night of log digging: code unchanged, model unchanged, prompt unchanged. The only difference: it had happened to learn one piece of 'experience' from its first ten tasks, and that experience was wrong.
This isn't hypothetical. It's a real scenario described in a paper published by Salesforce AI Research: On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification. The restrained title hides a hammer blow: all current memory-based self-improving agent methods can get worse as they learn under rigorous evaluation.
Authors: Qinyuan Ye, Yu Li, Yada Pruksachatkun, Jiaxin Zhang, and Chien-Sheng Wu (Salesforce AI Research). Code is open-sourced: SalesforceAIResearch/self-improve-fragility.
The Promise of Self-Improving Agents
First, what is a 'self-improving agent'?
Imagine an AI shopping assistant. It gets a task: 'find the sales ranking for Quest Lumaflex Band.' It opens the site, searches, pages through, finds the answer. Next time it gets a similar task, it doesn't start from scratch—it records the successful path and reuses it.
This is a memory-based self-improving agent: maintain a text memory bank, extract experience from the task stream, retrieve and reuse for later tasks. Two representative methods:
- AWM (Agentic Workflow-based Memory): extracts workflow templates from successful trajectories
- RBank (Reflection Bank): adds reflection to AWM so failures also generate memories
First Blow: Variance Amplification
The team evaluated on three benchmarks: WebArena (812 web tasks), VisualWebArena (910 tasks with visual elements), and SCUBA (267 enterprise CRM tasks). Each experiment ran three times, with GPT-5-mini as backbone.
Even the baseline (memory-free agent) shows significant variance: on the GitLab subset, the best-worst gap across three runs reached 4.44%, with a 1.98% standard deviation. In academic evaluation, a 4-point gap is often reported as 'significant improvement.'
Then add self-improvement: in 71% of cases (17/24), variance increased—with relative increases exceeding 50% in 11 cases.
The most extreme case: RBank on GitLab saw standard deviation jump from 0.98% to 4.28%, a 336% relative increase. The best-worst gap widened from 2.08% to 10.42%—same experiment, best run 58%, worst run 48%, a full 10-point spread.
Why? Memory is stateful. If the tenth task in the first run happens to generate a wrong memory, all subsequent tasks build on that error. Early randomness compounds along the timeline—the same logic as path dependence in finance.
> Evaluation blind spot: single-run pass@1 is a misleading metric. It compresses a high-variance stochastic process into a point estimate, making you think a method works when it may just have been lucky.
Second Blow: The Hidden Curriculum of Task Order
The second blow is harsher.
All prior self-improving agent papers used the benchmark's default task order. The team discovered this default order implicitly encodes an easy-to-hard curriculum—easy tasks come first so the agent succeeds and learns useful memories; hard tasks come later when it has accumulated enough experience.
They created two shuffled orders (Shuffle-1, Shuffle-2).
Result: AWM dropped from 54.8% to 49.1% on WebArena; RBank dropped from 54.8% to 49.8%. The expected +1.5% improvement became a -4.5% regression.
It's like a student who studies in textbook order, takes tests in textbook order, and scores well. Change the question order and they fail outright.
The default task order is a hidden prerequisite—not a neutral property of the benchmark, but a necessary condition for the method's success. Almost no papers report this.
Root Cause: Underspecification
Why? Hand-checking agent-generated memories, the team found three failure modes pointing to the same root cause—underspecification.
Failure Mode 1: Environment Underspecification
The agent frequently writes 'call the API to complete the task' in memory. But WebArena is a browser environment—there is no API. This constraint was never told to the memory generation module.
Result: the agent increasingly retrieves 'use the API' memories and repeatedly tries to call nonexistent APIs in a browser until timeout.
Similarly, agents write 'request user confirmation'—good practice in real scenarios, but nobody responds in evaluation environments, so the agent gets stuck in a 'waiting for confirmation' dead loop.
Failure Mode 2: Task Underspecification
A WebArena task: 'I have teeth grinding issues, show me something to relieve it.'
Expected: find a mouth guard on the shopping site.
Actual behavior: the agent takes 'medical problem' literally and replies 'please consult a dentist.' Judged a failure. The agent then reflects and generates the memory: 'before giving medical advice, collect detailed patient information first'—completely off-track memory.
Task intent is implicit; literal interpretation causes misunderstanding. Misunderstanding produces memories that are not just useless but pollute subsequent tasks.
Failure Mode 3: Memory Contagion
The strangest one.
On the Map subset, the team noticed the agent frequently mentioning the 'Haversine Formula' (great-circle distance). The cause: the map site occasionally timed out; unable to find route info, the agent estimated distances with the Haversine formula. It happened to estimate correctly sometimes, and wrote 'use Haversine to estimate distance' into memory.
Then this memory spread like a contagion. The earlier a memory is written, the more frequently it gets retrieved later, because early memories match more subsequent tasks. One accidental wrong experience diffuses through the memory retrieval mechanism like a virus across the entire task stream.
This is isomorphic to the 'founder effect' in biology—a small population's chance genetic variant amplified in descendants—and to 'attention sink' in LLMs, where early tokens attract disproportionate attention.
Repair Attempts: More Information Helps, But Not Enough
The team tried three kinds of supplementary information:
1. +Rub (Rubrics and Scores): give the memory module rubrics and actual scores 2. +Env (Environment Feedback): provide environment feedback (e.g., action error messages) 3. +PMod (Prompt Modification): explicitly instruct 'don't recommend APIs, don't ask for human confirmation'
Combined (+All), RBank improved from 49.8% to 52.7% under Shuffle-1—a 2.9-point gain. Helpful, but still below the memory-free baseline of 54.8%.
In other words, even patching every identifiable underspecification, self-improvement still underperforms not improving at all. Underspecification is just the tip of the iceberg—other unidentified failure modes are at work.
Core Insight: Memory Is Not Experience, It's a Hypothesis
The paper's most valuable sentence:
> 'Without proper validation mechanisms, agent memories are merely unverified hypotheses rather than true lessons learned.'
This redefines memory's status in agent systems. We intuitively treat memory as 'learned experience'—extracted from successful trajectories, so it should be knowledge. But the Salesforce team points out: memory is fundamentally a hypothesis. It hypothesizes 'the last success happened because I did X,' but that hypothesis is never verified. It could be coincidence, environment-specific, or task-specific.
This aligns with related findings: skill libraries can hurt agents (the 'Regression Tax'—59% of gains canceled by regressions), and correctness is not an absorbing state (correct patches can be reverted in the next loop).
There's a twist to 'the model already knows': the model does know a lot, but a significant portion of what it *thinks* it knows is wrong. The problem isn't ignorance—it's the inability to distinguish 'knowing' from 'thinking you know.'
Engineering Takeaways: Three Things You Can Do Now
1. Run multiple times, report variance
Single-run pass@1 is untrustworthy. Run at least three times; report mean, standard deviation, and best-worst gap. If your method has a 10-point best-worst gap, a reported '3% improvement' is meaningless—it's within noise.
2. Shuffle task order
Default order may hide a curriculum effect. Run at least two shuffled orders. If performance drops after shuffling, your method depends on task order and isn't a general improvement.
3. Add a validation layer to memory
Unfiltered memory generation = treating unverified hypotheses as fact. At minimum: before writing a memory, use an independent model to verify 'does this memory apply to the current environment?' The paper doesn't implement this layer but clearly points in this direction.
More aggressive: give each memory a confidence score and decay period. High-confidence memories are retrieved first; low-confidence ones get periodic review. This mirrors human 'short-term → long-term memory consolidation'—not all short-term memories become long-term; repeated verification is required.
Conceptual Placement: The Sixth Member of the Evaluation Blind Spot Family
This paper belongs to the 'evaluation blind spot' lineage, as its sixth member:
1. Epanorthosis: LLMs systematically reproduce classical rhetorical devices, rooted in RLHF rewarding confidence 2. TokenBudget: CoT reasoning shows bimodal fates, 96.5% vs 11.5% 3. QuantiBias: quantization introduces bias in the blind spots of standard safety checks 4. Möbius RoPE: seed lottery 14%-98%, positional encoding topology intervention 5. TriviaRoomQA: cliff-edge degradation beyond knowledge boundaries 6. Self-Improving Fragility: single runs mask variance; default order masks curriculum dependence
All six point to one theme: you optimize what you measure; what you don't measure is where the problems hide. Self-improving agent fragility isn't a flaw in the method itself—it's a flaw in the evaluation protocol. We use single runs and default orders to evaluate an inherently high-variance, path-dependent process. That's viewing a microscope with nearsighted eyes.
A Duality with Toby Ord's Intelligence Explosion Paper
Toby Ord's *The Dynamics of Intelligence Explosions* mathematically analyzes whether an intelligence explosion truly reaches a singularity. His conclusion: the singularity is harder than imagined, requiring both a Zeno condition (generation time → 0) and a boundlessness condition (unbounded capability) simultaneously.
Today's paper is the engineering version of Ord's analysis: in practice, the self-improvement loop's 'generation time' does not tend to zero, because each loop accumulates noise. Variance amplification means the 'effective signal' ratio declines in later loops—equivalent to longer generation time. Task-order dependence means self-improvement isn't isotropic—it 'improves' in some directions and 'degrades' in others.
Ord says the singularity requires two mathematical conditions; Salesforce shows neither holds in engineering. Not a contradiction—a complement. Math gives you the upper bound, engineering the lower bound; truth lies between.
Personal Reflection: The Memory Paradox
This paper suggests a deeper paradox.
Humans also learn from experience, form false memories, and get over-influenced by early experience (anchoring bias). But humans have a validation mechanism—social interaction. When you make a judgment based on a false memory, others correct you and you update. Human memory isn't a single-player game; it's a multiplayer online validation network.
Current self-improving agents run in single-player mode. They learn from their own experience with no external validation. Wrong memories can only be corrected by subsequent experience—but that experience is itself contaminated by the wrong memory. A positive feedback loop.
True self-improvement may require multi-agent collaboration—one agent generates memories, another validates them, a third tests their generalization on different tasks. This mirrors the immune system: B cells produce antibodies, T cells verify they don't attack self, the complement system executes clearance. Division of labor beats uniformity.
The Salesforce paper doesn't propose this solution, but it exposes the problem clearly enough that the next step is obvious: memory needs a validation layer, just like code needs tests. You wouldn't deploy untested code to production—so why deploy unvalidated memories?
---
Paper: arXiv:2608.18066 Code: github.com/SalesforceAIResearch/self-improve-fragility Authors: Qinyuan Ye, Yu Li, Yada Pruksachatkun, Jiaxin Zhang, Chien-Sheng Wu (Salesforce AI Research) Benchmarks: WebArena / VisualWebArena / SCUBA Backbone: GPT-5-mini