Can AI Reason Like a Scientist? Insights from the SIRBench-V1 Benchmark

Forum topic · ✨步子哥 · 2025-10-18

Summary

A new benchmark called SIRBench-V1 tests whether large language models (LLMs) can perform scientific inductive reasoning in biology and chemistry. The benchmark contains 710 task instances across seven tasks, using synthetic and counterfactual scenarios (such as randomized DNA codon tables) to prevent models from relying on memorized knowledge. Results show a large gap between real and synthetic tasks: Gemini-2.5-Flash scored 87.09% on real DNA table inference but only 32.06% on synthetic versions, a drop of over 55 points, and similar patterns appeared for Claude-3.5-Haiku and GPT-4.1. The study also compared four reasoning strategies (implicit, explicit, self-consistency, and hypothesis refinement), finding that implicit inductive reasoning (~43.81%) often outperformed more elaborate strategies. Performance degraded on long-context tasks, with accuracy falling from 60% to 20% as DNA sequence length increased from 16 to 512. The conclusion: LLMs excel as knowledge repositories but struggle with genuine rule discovery in novel, non-mathematical scientific tasks.

Can large language models (LLMs) truly discover scientific truths the way scientists do? A recent arXiv paper (https://arxiv.org/pdf/2509.16226) tackles this question with SIRBench-V1, a new benchmark evaluating scientific inductive reasoning in biology and chemistry, focused on non-mathematical rule-based tasks.

What is SIRBench-V1?

SIRBench-V1 is a benchmark for testing LLM inductive reasoning: deriving general rules from specific observations, a core skill of scientific discovery. It includes:

710 task instances across seven tasks (three biology, four chemistry)
Synthetic and counterfactual tasks (e.g., randomized DNA codon tables where ATG maps to K instead of the standard M) to ensure models cannot rely on memorized knowledge and must learn rules in-context
Tasks such as DNA table inference, molecular design (generating SMILES from text prompts), molecular description, reaction prediction, and IUPAC name conversion

Key findings

Memory vs. reasoning: a large gap

On real tasks based on standard scientific rules, models perform impressively: Gemini-2.5-Flash achieved 87.09% on real DNA table inference.
On synthetic tasks, performance collapses: the same model dropped to 32.06% (a 55.03-point decline), showing heavy reliance on memorized patterns.
Claude-3.5-Haiku reached 43.81% on synthetic DNA translation vs. Gemini's 11.72%; the pattern holds across models including GPT-4.1.

Chemistry task results (Gemini-2.5-Flash, molecular design)

Exact match: 0.33
BLEU: 0.73
Validity: 0.80
Morgan similarity: 0.85
FCD: 8.19 (lower is better)

Models handle structured generation better than open-ended description: molecular description achieved only BLEU-2 of 0.49 and 63.30% LLM-as-judge accuracy. Reaction prediction reached 54.17% accuracy, while name prediction (iupac2smiles) was only 20.00%.

Four reasoning strategies compared

1. Implicit inductive reasoning — predict directly from in-context examples (~43.81%) 2. Explicit inductive reasoning — formulate and apply hypotheses (~38.66%) 3. Self-consistency — sample multiple hypotheses, majority vote (~41.97%) 4. Hypothesis refinement — iteratively refine hypotheses with feedback (~39.06%)

Surprisingly, implicit reasoning often beat explicit strategies, suggesting models struggle to consistently articulate and apply rules. Hypothesis refinement offered limited gains, often producing fragmented outputs misaligned with task requirements.

Long-context challenges

On DNA transformation tasks, accuracy fell from 60% to 20% as sequence length increased from 16 to 512.
Models prefer many short examples (64 samples × length 100: 35.14% accuracy) over few long ones (4 samples × length 1600: 24.47%).

Conclusion and future directions

SIRBench-V1 shows LLMs excel at recalling scientific facts but struggle to discover new rules in synthetic, non-equation, or long-context scenarios. Bridging this gap may require:

Improved in-context learning to extract patterns from few examples
Better long-context handling via stronger attention mechanisms
Task-aligned reasoning strategies for open-ended generation tasks

References

1. SIRBench-V1 paper (2025): https://arxiv.org/pdf/2509.16226 2. Molecular design metrics — SIRBench-V1, Section 4.2 3. DNA translation analysis — SIRBench-V1, Table 7 4. Reasoning strategies — SIRBench-V1, Section 3.3 5. Long-context challenges — SIRBench-V1, Section 4.1