Paper Slam 4/20: When an LLM Faces a Bird and an X-ray Beam

Introduction

Imagine a room. On the left, people are debating what type of call an Andean flamingo makes — peep, chirp, whistle, or growl. On the right, others say: "Compute the XANES spectrum for Ti in TiO₂." These two tasks — one zoology trivia, one materials simulation — point at the same fundamental question: should LLMs in science be encyclopedic students or hands-on assistants? Two arXiv papers (April 17) stand at opposite ends of this question: BAGEL builds an 11,852-question closed-book animal knowledge exam; ChemGraph-XANES builds an agent framework that orchestrates X-ray absorption spectroscopy simulations. One asks "what do you know," the other "what can you do."

Key points

Chapter 1: BAGEL — a closed-book zoology exam for LLMs

• Design: 11,852 four-option multiple-choice questions across 8 knowledge dimensions (Taxonomy, Behavior, Communication, Morphology, Habitat, Cognition, Geographic Distribution, Diet), drawn from four sources:
• Wikipedia: 1,927 questions
• Global Biotic Interactions (GloBI): 3,500
• bioRxiv preprints: 2,183
• Xeno-canto bird-sound database: 4,242
• Closed-book evaluation: no retrieval, no RAG, no tool calls. The rationale: if an LLM cannot recall what it has already seen, claims of "understanding" are moot.
• Source heterogeneity: Wikipedia is structured encyclopedic text; bioRxiv is cutting-edge specialized prose; GloBI tests relational networks (who eats whom); Xeno-canto converts acoustic features into text descriptions that must be linked to species — a cross-modal association task, not audio perception.
• Findings:
• All models perform best on Taxonomy (most structured knowledge); worst on Communication and Diet, which require linking facts rather than recalling isolated entries.
• Position bias: some models systematically favor certain answer positions — a methodological warning for any MCQ benchmark.
• Non-monotonic scaling: on Xeno-canto, Qwen3-32B underperforms Qwen3-14B. Bigger is not always better on specialized knowledge types.
• Authors manually audited questions; some distractors are partially plausible (e.g., "wetland grassland" vs "seasonal wetland"), so a wrong answer does not always mean ignorance. After filtering, overall discriminative quality is adequate.
• What it does not test: reasoning about zoological scenarios, generating new knowledge, or doing research. BAGEL measures the LLM as a *knowledge base*, not as a scientist.

Chapter 2: ChemGraph-XANES — an agent that computes spectra

• Why: XANES (X-ray Absorption Near-Edge Structure) simulations involve a multi-step workflow (obtain crystal structure, choose absorber atom, generate FDMNES inputs, run on HPC, parse and normalize spectra, track provenance). Manual execution is untenable for high-throughput studies of hundreds or thousands of structures.
• Four-layer architecture:
• ASE — structure handling and unified atom representation
• FDMNES — the physics engine (finite-difference XANES simulation)
• Parsl — parallel execution across HPC nodes
• LangGraph/LangChain — typed tool orchestration driven by natural language
• Single vs multi-agent: a single agent loops reasoning and tool calls; a multi-agent mode adds a planner, workers, an aggregator, and notably a retrieval-augmented expert agent that searches a locally indexed FDMNES manual to ground parameter choices (e.g., default absorber selection rules, doping simulation methods, default energy range of −5 to 60 eV in 0.5 eV steps), combating parameter hallucination.
• Two input modes: file mode (e.g., a POSCAR for an MnO₂(010) slab with Cu, Z=29, as absorber) or natural language mode ("Compute the XANES for Ti in TiO₂" — the agent infers the Materials Project lookup and Z=22).
• Normalization pipeline: optional E₀ estimation from the derivative maximum, pre-edge (−20 eV) and post-edge (+50 eV) linear baselines, edge-step normalization — essential for comparability and downstream machine learning. File naming (absorber Z, Materials Project ID, formula) provides lightweight provenance.
• Limits: the agent cannot arbitrarily edit FDMNES input files; it is constrained to schema-validated tools. Autonomy is traded for reproducibility. The authors argue parameter choices should always remain documented and structurally validated even as models improve.

Chapter 3: Head-to-head

• Knowledge breadth vs workflow depth: BAGEL maps where LLM knowledge is strong or blank; ChemGraph-XANES automates one workflow deeply. BAGEL assumes knowledge reserves precede capability; ChemGraph-XANES assumes orchestration can stand apart from domain knowledge. Both assumptions are partially true and partially fragile: an agent that cannot judge whether a spectrum looks reasonable hits a domain-knowledge bottleneck — hence the RAG expert agent.
• Closed-book vs open-book: BAGEL's design embeds the assumption that only pretraining knowledge counts; in real deployments LLMs always have retrieval and tools. Closed-book tests intrinsic quality; open-book agentic tests task achievement — two different definitions of "intelligence."
• Field differences: zoology is descriptive and classificatory — its facts are text-representable and quiz-testable. Spectral computation is predictive — you cannot quiz an LLM into drawing TiO₂'s XANES curve; it must be computed. LLMs fit naturally as knowledge bases in the first kind of science and as orchestrators in the second.
• Human roles: in BAGEL, humans write good questions; in ChemGraph-XANES, humans design tools and constraints — giving AI freedom within boundaries.

Chapter 4: Deeper questions (a Feynman-style critique)

1. Measurement is construction: BAGEL's four sources define its own blind spots — English-text-representable knowledge only; traditional ecological knowledge and field intuition fall outside. A high score covers a *specifically defined* knowledge space, not "zoology" writ large. 2. Explainability: multiple-choice only says right/wrong, not why. Agent traces record *what* was done, not *why* (e.g., why a 6 Å cluster radius). Pattern matching is not decision-making. 3. Cargo cult detection: benchmarks and agent frameworks can create the illusion that "AI is systematically conquering science." Neither paper claims understanding — but media narratives do. Needed: reasoning-augmented knowledge tests and adversarial testing of agents (faulty inputs, anomaly detection, failure recovery). 4. Thought experiment — swap the tasks: a BAGEL-style XANES quiz could not verify whether a spectrum is *correct*; a zoology agent could not *discover* a new species missing from databases. Knowledge tests and workflow automation each have a ceiling; neither produces new knowledge. The scarce resource remains the human ability to ask good questions.

Conclusion: two futures

BAGEL's path: measure what AI knows first, then decide how to use it — toward an all-knowing AI. ChemGraph-XANES's path: let it act regardless — toward an all-doing assistant. They are complementary, but if forced to choose, the author judges the orchestration path more productive: science's core is not knowing but doing — provided the toolchain stays reliable, transparent, and reproducible. A Feynman-style closing challenge: can the exam ace-ing AI actually aid wildlife conservation? Can the spectrum-computing agent judge whether its own result is correct? If both answers are "unsure," the distance from these mature technologies to real science is called *understanding*.

References

• Shen J, Hagiwara M, Alizadeh M, et al. BAGEL: Benchmarking Animal Knowledge Expertise in Language Models. arXiv:2604.16241 [cs.CL, cs.AI], 2025.
• Grizzi VF, Pham TD, Pretzie LN, et al. ChemGraph-XANES: An Agentic Framework for XANES Simulation and Analysis. arXiv:2604.16205 [cond-mat.mtrl-sci, cs.AI, physics.chem-ph], 2025.