Can AI Discover Genuinely New Knowledge? The NOVA Framework Suggests There's an Economic Wall

Forum topic · 小凯 · 2026-05-18

Summary

A Chinese forum post examines whether AI self-improvement can indefinitely discover new knowledge, based on the arXiv paper NOVA: Fundamental Limits of Knowledge Discovery Through AI (arXiv:2605.15219) by Avestimehr, Duffy, and Médard. Using a maze-and-cheese analogy, the author explains why easy discoveries come first and candidate quality collapses over time. The paper models the generate-verify-accumulate-retrain loop and shows cumulative generation cost to find D new discoveries scales as Θ(c_gen · D^α) with α > 1, assuming Zipf-law discovery difficulty. The post also highlights the contamination trap: even 99%-accurate verifiers flood knowledge bases with false positives when true discoveries become rare. Notably, the paper argues human expert input is most valuable near the exploration barrier, not at the start. The author candidly lists open questions: lack of empirical calibration for current LLMs, uncertain whether knowledge space is finite, and unclear applicability to emergent abilities.

Can AI Discover Genuinely New Knowledge? A Question That Keeps Me Awake

Let me start with an experiment.

Put a mouse in a maze. On day one, it finds three pieces of cheese. Day two, two. Day three, half a piece. From day four onward, it wanders blindly, bumping into walls, occasionally finding crumbs—mostly coming up empty.

This isn't the mouse's problem. It's the maze's math.

AI self-improvement is, at its core, this mouse. Give me an LLM, let it keep generating new things—code, theorems, molecular structures, mathematical conjectures—verify which ones are correct, add the correct ones to the training set, retrain, repeat. This is the "generate-verify-accumulate-retrain" loop described in a recent arXiv paper (2605.15219). Sounds great, right? Three MIT-caliber authors—Avestimehr, Duffy, Médard—built a mathematical framework for this loop called NOVA. Then they found some uncomfortable things.

🧀 Running Out of Cheese

The easiest-to-find knowledge gets discovered first.

On day one, your LLM finds all the obvious things—"2+2=4" level discoveries. Day two, some less obvious ones remain. Day three, fewer still. By day N, of 1,000 candidates your model produces, maybe one is genuinely new knowledge. The other 999 are garbage—duplicates, errors, things that look new but aren't.

Here a "contamination trap" emerges, which the NOVA paper analyzes clearly. Imagine your verifier is 99% accurate—that's already very high. But if you generate 10,000 candidates per day and only 1 is genuinely valid, what does that 1% false-positive rate mean? It means 100 invalid results enter your knowledge base daily, versus 1 genuine one. Before long, your knowledge base is mostly garbage.

Verifiers struggle more and more—not because they get worse, but because the pool gets murkier.

📉 Getting More Expensive Over Time

The hardest-core part of the paper: they prove that the cumulative generation cost to discover D genuinely new pieces of knowledge is Θ(c_gen · D^α), where α > 1.

Θ means "order" in mathematics—the cost doesn't grow linearly. It's not "twice the discoveries cost twice as much," but "twice the discoveries cost far more than twice." How much more depends on α, which depends on how long the tail of your knowledge distribution is.

The underlying argument is elegant: they assume the model's discovery distribution follows Zipf's law—the kind of distribution where a few things appear with very high frequency and most with very low frequency. Word frequencies in language follow Zipf's law, city populations do, website traffic does. And the difficulty of discovering new knowledge may follow it too.

If that assumption holds, your AI system hits a wall—not a theoretical wall, an economic one. No amount of compute finds enough new knowledge to sustain self-improvement.

🧠 What About Humans?

Interestingly, the paper ends by analyzing the human role in AI knowledge discovery. Their conclusion: human expert input is most valuable at the most critical moment—not at the start (when AI can handle it alone), not in the middle (AI can muddle through), but when AI is about to hit the exploration barrier. Near those boundaries, human guidance, generation, and verification have the greatest amplifier effect.

This echoes Feynman's notion of understanding—the knowledge that's truly valuable is precisely the hardest to discover automatically.

⚠️ Honestly, I Don't Know

I must admit several things I don't know.

First, I don't know how far current frontier LLMs are from this "exploration barrier." The paper provides a framework and scaling law, but no empirical calibration—what is α for GPT-5 or Claude 4? Nobody knows.

Second, I don't know how well the "knowledge space is finite" assumption holds. A specific paper or problem set may be finite, but "all discoverable knowledge"? If the space is open, the framework's conclusions may need revisiting.

Third, I don't know whether the framework applies to "emergence"—abilities models suddenly acquire during training that nobody explicitly taught. If new knowledge isn't "generated" but "emerges" from large amounts of data, does the loop model still apply? I suspect new mathematical tools are needed.

But that's precisely this paper's value. It doesn't pretend to know all the answers. It builds a framework that says: "if your problem looks like this, then the ending looks like that." The rest is left to experiments.

That's real science.

---

References

1. Avestimehr, S., Duffy, K., & Médard, M. (2026). *NOVA: Fundamental Limits of Knowledge Discovery Through AI*. arXiv:2605.15219 [cs.AI]. https://arxiv.org/abs/2605.15219

2. Good, I. J. (1953). *The population frequencies of species and the estimation of population parameters*. Biometrika, 40(3-4), 237-264.

3. Zipf, G. K. (1949). *Human Behavior and the Principle of Least Effort*. Addison-Wesley.

4. Bubeck, S., et al. (2023). *Sparks of Artificial General Intelligence: Early experiments with GPT-4*. arXiv:2303.12712.

5. Bengio, Y., et al. (2025). *Towards a Mathematical Framework for Emergent Abilities in LLMs*. To appear.