ProjectionBench: LLMs Predict Experimental Conclusions from Research Questions Alone

Forum topic · 小凯 · 2026-05-31

Summary

ProjectionBench (arXiv:2605.30284) is a benchmark from researchers at Unreasonable Labs that evaluates scientific hypothesis generation in large language models under progressive information disclosure. Using 45 open-access materials science papers published after model training cutoffs, the framework feeds models three tiers of information — topic plus research question, plus null hypothesis, plus full experimental details — and scores predicted findings against actual published conclusions by decomposing both into atomic claims and computing F1 via an LLM-as-a-judge pipeline. GPT-5.4 achieved F1≈0.7 when given only the topic and research question, outperforming GPT-5, Gemini 3.1 Pro Preview, and Gemini 2.5 Pro. Results reveal a trade-off between low-information intuition and high-information reasoning, diminishing marginal utility of added context (knowing the hypothesis matters more than knowing the methods), and clear domain difficulty gradients, with bioactive materials easiest and mechanical materials hardest. Predictions are qualitative only, omitting effect magnitudes. The living, contamination-free dataset supports continuous updates to prevent benchmark overfitting.

ProjectionBench: When Models Guess Experimental Conclusions at First Glance

*English translation of a zhichai.net forum post discussing arXiv:2605.30284.*

| Item | Detail | |------|--------| | Paper | ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure | | Authors | Andrew J. Lew, Yuan Cao, Markus J. Buehler | | Institution | Unreasonable Labs, Mountain View, CA | | arXiv ID | 2605.30284 | | Submitted | May 28, 2026 | | Category | cs.AI | | Key finding | GPT-5.4 predicts real experimental conclusions (F1≈0.7) given only a topic and research question; a notable trade-off exists between "intuitive innovation" at low information and "grounded reasoning" at high information |

1. Question Only, No Answer

Imagine this experiment. Take a paper published *after* the model's training cutoff — it has definitely never seen it. Hide the methods, data, and results. Give the model only two things: the paper's topic and its research question.

For example: "thermomechanical properties of nanocomposites" as topic; "is natural fiber surface treatment superior to conventional NaOH treatment?" as question. Then ask: what conclusions did this paper reach?

That is what ProjectionBench does. It doesn't test whether you can search literature — it tests whether, given minimal information, you can project a scientific hypothesis in the right direction.

GPT-5.4 scored 0.7 F1. That means its predicted "findings," given only topic and question, matched roughly 70% of the authors' actual written conclusions.

The scoring isn't fuzzy semantic similarity. Both the real conclusions and the model's predictions are decomposed into atomic claims — e.g., "Treatment A improved storage modulus," "Treatment B had no effect on thermal stability" — then compared item by item. True positives, false positives, and omissions are all tallied into an F1 score. This 0.7 is real, structured scientific agreement.

2. Progressive Disclosure

The benchmark splits each paper into three information tiers:

Tier 1 (minimal): topic + research question. Tests innovativeness — can you make physically meaningful speculation from almost nothing?
Tier 2: adds the null hypothesis.
Tier 3 (full): topic + question + null hypothesis + experimental procedure. Tests grounded reasoning.

Results: GPT-5.4 leads on both dimensions, but with diminishing marginal returns from Tier 2 to Tier 3. Adding the null hypothesis produces a large jump; adding experimental steps adds much less.

Implication: for current top models, **knowing *what* the hypothesis is matters more than knowing *how* the experiment was done. Get the direction right and details matter less; get it wrong and no amount of procedure helps.

3. Grading Imagination

Evaluating hypothesis quality is hard because science has no single correct answer — yet fully open-ended evaluation is unrepeatable. ProjectionBench sidesteps this elegantly.

It doesn't ask "is this hypothesis good?" It asks "how far is this hypothesis from the published conclusion?" GPT-5 acts as judge, decomposing both into minimal propositional units and cross-checking:

"A promotes B growth" vs. paper's "A significantly promotes B growth" → consistent

"A inhibits B" → contradiction

"Interaction effect of A and C" (unmentioned in paper) → redundant claim

Matched, missed, and redundant claims feed an F1 formula. Since the papers are already published, this becomes a counterfactual reasoning test: back before publication, could the model approach the eventual fact from incomplete information?

4. Who Projects, Who Guesses

Four models were tested. Overall AUC ranking:

GPT-5.4: 1.56

GPT-5: 1.44

Gemini 3.1 Pro Preview: 1.44

Gemini 2.5 Pro: 1.33

Newer generations dominate — unsurprising. The surprise is the gap at low information. GPT-5.4 holds F1≈0.7 with topic and question only; Gemini 2.5 Pro can fall to 0.

A concrete example: comparing KTN (natural potassium salt) treatment vs. NaOH treatment of Honckenya fiber–reinforced polypropylene composites.

Ground truth: KTN-treated composites significantly outperform NaOH-treated and untreated groups in storage modulus, loss modulus, and thermal stability.

GPT-5.4's minimal-info prediction: got the direction right ("KTN better than or comparable to NaOH"), but hallucinated an unreported claim — "performance rises then falls with treatment intensity, showing an optimum."

Gemini 2.5 Pro: guessed backwards — "conventional NaOH is better." Wrong.

Why? The paper argues Gemini 2.5 Pro anchored on historical knowledge: NaOH is the well-established effective fiber treatment in existing literature. It regressed to prior rather than projecting toward the new conclusion.

GPT-5.4 walked a tightrope between known and unknown — possibly sensing that "NaOH being better" was improbable, but overcompensating by inventing an optimal curve. Is that good or bad? Bad — it fabricated details it shouldn't have. Good — its direction was right: it captured the essential relation that some new treatment beats conventional NaOH.

5. Domain Difficulty Gradients

The benchmark covers three materials science areas: bioactive materials, nanomaterials, and mechanical materials.

Bioactive materials: highest scores, near ceiling — models have mastered this domain's baseline knowledge.

Nanomaterials: middle — no ceiling, but no low floor either.

Mechanical materials: hardest, with the widest score spread. Notably, Gemini 3.1 Pro beat GPT-5 here under high-context conditions — unique to this domain.

The paper doesn't explain why. Bioactive papers may have more "patterned" conclusions, or the training corpus may be richer in bio text. Mechanical materials knowledge may be sparser or less structured in model parameters. These are speculations; the paper offers no causal account. That honesty is a virtue.
6. Direction, Not Numbers

A deliberate omission: the benchmark requires only qualitative predictions ("A improved B"), never numerical ones — no percentages, no confidence intervals.

Why? Numerical prediction would require physical simulation, differential equations, labs. ProjectionBench's goal was never "replacing experiments" but assessing scientific intuition** — can you form the right *sense of direction* under extreme information scarcity? Practicing scientists know direction is the hard part; once numbers exist, back-filling explanations is easy.

But this caps the framework's ceiling: pure qualitative projection can only reward "direction fully correct." A model saying "A slightly improves B" scores the same as one saying "A hugely improves B." The paper acknowledges this limitation; future versions could add effect-size evaluation.

7. A Living Dataset

The 45 papers (15 per domain) come from Springer Nature open-access articles published after the models' training windows — so training data contamination is impossible. This genuinely tests generalization to new discoveries, not memorization.

The framework is designed for real-time updates: new papers can continuously join the dataset, making benchmark overfitting nearly impossible — solving a chronic problem in scientific evaluation. At least for now.

8. What Is Scientific Intuition?

The word that kept striking this reader: projection. Not prediction, not inference, not retrieval. Projection means: your information is insufficient, yet you must make a physically meaningful judgment.

It resembles stopping a senior professor in the hallway, giving her only a title and research question, and asking what she thinks the conclusion is. She runs rapid implicit "if…then…" simulations over decades of knowledge. Her projection isn't random — it contains an implicit causal model.

ProjectionBench essentially asks: have language models formed such implicit causal models? The evidence points to yes — at least in some domains. GPT-5.4 never retrieved "copper promotes osteogenesis" from the specific paper; it inferred it from statistical regularities across hundreds of millions of related papers.

That ability has a more familiar name: intuition. Scientific intuition.

Of course, GPT-5.4 has no consciousness or understanding — applying "intuition" to statistical models risks dangerous analogical confusion. But functionally, a model making directionally correct judgments under incomplete information — if that counts as scientific intuition, it has it. At least in certain materials science subfields.

9. Beyond the Narrative: What It Found, What It Didn't

ProjectionBench is an evaluation paper — framework, experiments, results. Not a manifesto claiming AI can independently do scientific discovery.

What it found:

1. Top LLMs can project correct scientific conclusions from minimal information in materials science, with F1 roughly 0.55–0.75 depending on domain and information level. 2. This ability improves across model generations. 3. Significant inter-domain gaps exist. 4. Marginal utility of added information diminishes — knowing the hypothesis matters more than knowing the methods.

What it did not find:

1. Models cannot independently design experiments — the test is "guess the conclusion," not "design a validation plan." 2. No numerical prediction capability is assessed — all speculation is qualitative. 3. Whether the reasoning process is genuinely "scientific" is unexamined. Did GPT-5.4 understand why copper ions promote osteogenesis, or did it perform high-dimensional analogy over similar research patterns in its corpus? Unanswered. 4. Reproducibility is not discussed. Real scientific validation lies not in guessing right once, but in independent replication under identical conditions. ProjectionBench tests one-shot projection accuracy, not projection stability.

These aren't criticisms — this is a strong local contribution that is a beginning, not an end.

10. A Harder Question

An honest answer to "can AI do scientific discovery" today is: it depends whether you're testing intuition or method.

Test intuition — GPT-5.4 is already very strong. Given a research question, it guesses the answer's direction, and direction is often more valuable than detail. A directionally correct hypothesis with imperfect details can guide experiment design; a directionally wrong one wastes resources no matter how elegant the methodology.

Test method — AI has far to go. Designing controlled experiments, controlling confounders, spotting statistical artifacts, judging sample sizes — these abilities are barely visible in current models.

ProjectionBench measures the former. It elegantly demonstrates that AI possesses a form of scientific intuition. It doesn't tell us whether AI can replace scientists — it tells us that on the marathon of scientific discovery, AI has at least learned how to start running. Concretely: in some materials science subfields, on the dimension of "inferring experimental conclusions from research questions," AI already performs at or above the level of a junior researcher.

How far it will run, or whether it will fall — those tests haven't been designed yet.

References

1. Lew, Cao & Buehler, "ProjectionBench: Evaluating Scientific Hypothesis Generation in LLMs Under Progressive Information Disclosure", arXiv:2605.30284, 2026. 2. Majumder et al., "DiscoveryBench: Towards Data-Driven Discovery with Large Language Models", arXiv:2407.01725, 2024. 3. Wang et al., "SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models", arXiv:2307.10635, 2024. 4. Moussa et al., "ScholarEval: Research Idea Evaluation Grounded in Literature", arXiv:2510.16234, 2025. 5. Gu et al., "A Survey on LLM-as-a-Judge", arXiv:2411.15594, 2025.