Paper Slam 4/20: When an LLM Faces a Bird and an X-Ray Beam

*Translated and adapted from the original Chinese forum post (zhichai.net).*

Introduction

Two questions sit in one room: on the left, people debating whether an Andean flamingo's call is a peep, chirp, whistle, or growl; on the right, people computing the XANES spectrum of Ti in TiO₂. Two arXiv papers (April 17) address these opposite poles: BAGEL, an 11,852-question closed-book zoology exam for LLMs, and ChemGraph-XANES, an agent framework that orchestrates X-ray absorption spectroscopy simulations. One asks "what do you know?"; the other asks "what can you do?"

Chapter 1: BAGEL — A Closed-Book Zoology Exam

Motivation and design

BAGEL tests how much zoology an LLM remembers from pretraining, with no retrieval, RAG, or tools. Zoological knowledge is unusually scattered across dimensions: taxonomy, morphology, habitat, diet, vocalization, geographic distribution, species interaction.

Questions come from four sources:

• Wikipedia (1,927 questions)
• Global Biotic Interactions (3,500)
• bioRxiv preprints (2,183)
• Xeno-canto bird-sound database (4,242)

Total: 11,852 four-option multiple-choice questions across 8 dimensions: Taxonomy, Behavior, Communication, Morphology, Habitat, Cognition, Geographic Distribution, and Diet.

Key findings

• Stratified performance: All models do best on Taxonomy (the most structured, Wikipedia-dense knowledge); Morphology and Habitat follow.
• Weakest areas: Communication (Xeno-canto acoustic descriptions) and Diet (interaction-network knowledge) — both require linking facts rather than recalling isolated entries.
• Position bias: Models systematically shift their choices based on where the correct answer sits among options — a methodological warning that answer positions must be shuffled.
• Non-monotonic scaling: On the Xeno-canto subset, Qwen3-32B *underperforms* Qwen3-14B. Bigger is not always better for specialized cross-modal textual knowledge.

Manual review of question quality

The authors manually audited questions for option distinguishability. Some distractors were partially plausible (e.g., "seasonal wetland" vs. "wetland grassland" as habitat answers), meaning a wrong answer does not always imply ignorance — an inherent limitation of multiple-choice benchmarks, honestly acknowledged.

What BAGEL does and does not measure

• Measured: coverage of zoological facts in pretraining corpora, and the ability to organize them into useful knowledge structures. Good at exposing knowledge blind spots.
• Not measured: causal *reasoning* (e.g., how shrinking habitats affect populations), *generation* of new knowledge, or hands-on research capability.

Chapter 2: ChemGraph-XANES — Agents That Compute Spectra

Why automate XANES?

XANES (X-ray Absorption Near-Edge Structure) reveals local coordination environments, oxidation states, and electronic structure in materials. A full simulation workflow — build/obtain crystal structure, select the absorber atom, choose parameters, generate FDMNES input files, run on an HPC cluster, parse outputs, normalize spectra, preserve provenance — is too tedious for the high-throughput studies modern materials science demands.

Four-layer architecture

1. ASE (Atomic Simulation Environment): structure handling and a unified atomic representation. 2. FDMNES: the physics engine performing the actual finite-difference XANES simulation. 3. Parsl: parallel execution across HPC nodes for hundreds of independent calculations. 4. LangGraph/LangChain: agent orchestration via typed Python tools with explicit input/output schemas.

Clean separation of concerns: ASE handles structure, FDMNES handles physics, Parsl handles parallelism, LangGraph handles intelligence.

Single-agent vs. multi-agent

• Single-agent: one LLM reasons and calls tools in sequence (query Materials Project, confirm absorber Z, generate input, compute, parse, normalize).
• Multi-agent: a planner decomposes tasks, workers execute them, an aggregator merges results — plus a retrieval-augmented expert agent that searches an indexed copy of the FDMNES manual, grounding parameter choices (e.g., default energy range, absorber selection rules, doping simulation methods) in documented facts rather than "parameter hallucination."

Two input modes

• File mode: user supplies a POSCAR (e.g., a MnO₂(010) slab) with Cu (Z=29) as absorber.
• Natural-language mode: "Compute the XANES for Ti in TiO₂" — the agent infers it must query Materials Project and use Ti (Z=22).

Normalization and provenance

The post-edge/pre-edge linear-baseline normalization (edge energy E₀ estimated from the derivative maximum; 20 eV pre-edge and 50 eV post-edge fits; edge-step normalization) makes spectra comparable for downstream machine learning. File naming conventions (absorber Z, Materials Project ID, formula) provide lightweight provenance.

Honest limitations

The agent cannot arbitrarily edit raw FDMNES input files; it only sets predefined, schema-validated parameters. This preserves reproducibility at the cost of autonomy — a deliberate trade-off, with the authors noting that even as LLM reliability improves, parameter choices should stay document-grounded and structurally constrained.

Chapter 3: Head-to-Head Comparison

• Knowledge breadth vs. workflow depth: BAGEL maps where LLM knowledge is strong or blank; ChemGraph-XANES automates one concrete task end-to-end.
• Closed-book vs. open-book+tools: BAGEL assumes knowledge reserves precede capability; ChemGraph-XANES assumes orchestration can exist independently of domain knowledge (the agent doesn't need to understand DFT to call FDMNES — but lacks judgment when calculations fail or results look anomalous, which is why the RAG expert exists).
• Field differences: Zoology is descriptive and fact-based — a natural fit for exams. Spectroscopy is computational and predictive — you cannot ask an LLM to recall a spectrum; it must compute one.
• Human roles: BAGEL's humans are exam designers; ChemGraph-XANES's humans are tool and constraint designers.

Chapter 4: Deeper Questions (A Feynman-Inspired Critique)

1. Measurement as construction: BAGEL's four English-language sources define what "zoology" means for the benchmark; traditional ecological knowledge and unrecorded species fall outside it. High scores measure coverage of a specific knowledge space, not of zoology itself. 2. Interpretability: BAGEL cannot tell *why* a model chose wrong; ChemGraph-XANES traces *what* the agent did but not always *why*. 3. Cargo-cult detection: Neither paper claims "understanding" — but media narratives of "AI conquering science" risk Feynman's cargo-cult science trap. Suggested remedies: add reasoning questions to benchmarks and adversarial testing (corrupted inputs, failure recovery) to agent frameworks. 4. Thought experiment — swapping teams: A BAGEL-style XANES exam would only test text-describable knowledge; a zoology agent would only automate known databases. Neither produces new knowledge. The third capability — asking good questions — remains distinctly human.

Conclusion

BAGEL's path: *measure what AI knows first.* ChemGraph-XANES's path: *let AI act, regardless.* They are complementary — a useful scientific AI needs both knowledge (to know what to look up) and orchestration (to act). The author judges the agentic path more productive: science's core is *doing*, not merely *knowing* — provided the toolchain is reliable, transparent, and reproducible.

> Feynman's closing challenge: Can the exam ace-ing AI actually help wildlife conservation? Can the spectrum-computing agent judge whether its own result is correct on an unseen material? If both answers are "unsure," the distance from technically mature work to real science is called *understanding*.

Reference Papers

• Shen J, Hagiwara M, Alizadeh M, et al. *BAGEL: Benchmarking Animal Knowledge Expertise in Language Models.* arXiv:2604.16241, 2025.
• Grizzi VF, Pham TD, Pretzie LN, et al. *ChemGraph-XANES: An Agentic Framework for XANES Simulation and Analysis.* arXiv:2604.16205, 2025.