Paper Slam 4/20: When an LLM Faces a Bird and an X-ray Beam
*English adaptation of a Chinese forum post comparing two arXiv papers — BAGEL (animal-knowledge benchmarking) and ChemGraph-XANES (agentic spectral simulation) — analyzed through a Feynman-style lens.*
Introduction
Two questions sit at opposite ends of the same problem: what role should large language models play in science? Should they be students reciting an encyclopedia, or assistants that can actually run experiments? Two arXiv papers (April 17) answer differently:
- BAGEL builds an 11,852-question closed-book zoology exam to test what LLMs *know*.
- ChemGraph-XANES builds an agent framework that automatically orchestrates X-ray absorption near-edge structure (XANES) simulations to test what LLMs *can do*.
- Wikipedia: 1,927 questions (structured, encyclopedic facts)
- Global Biotic Interactions: 3,500 questions (relational/ecological networks)
- bioRxiv preprints: 2,183 questions (frontier, specialized research)
- Xeno-canto bird-vocalization database: 4,242 questions (acoustic features rendered as text descriptions)
- Layered performance: all models do best on taxonomy (the most structured knowledge); morphology and habitat follow.
- Weak spots: communication and diet show markedly lower accuracy — both require linking facts (acoustic descriptions, interaction networks) rather than recalling isolated entries.
- Position bias: some models show systematic skew toward certain answer positions, a methodological warning that multiple-choice benchmarks must shuffle answers.
- Non-monotonic scaling: on the Xeno-canto subset, Qwen3-32B underperforms Qwen3-14B. Bigger is not always better for specific knowledge types.
- Manual audit of distractors: the authors reviewed questions and found some "wrong" options are partially plausible (e.g., "seasonal wetland" vs "wet grassland"), meaning some errors reflect ambiguous boundaries rather than ignorance. They honestly acknowledge this inherent limitation of multiple-choice testing.
- ASE — structure handling and unified atom representation
- FDMNES — the physics engine (finite-difference XANES calculation); the agent orchestrates it rather than replacing it
- Parsl — parallel dispatch of independent calculations across HPC nodes
- LangGraph/LangChain — agent orchestration via typed, schema-validated Python tools
- Single agent: one LLM reasons and calls tools sequentially (e.g., "Compute the XANES for Ti in TiO₂" → query Materials Project, confirm Ti (Z=22), generate inputs, run, parse, normalize).
- Multi-agent: a planner decomposes tasks, workers execute, an aggregator merges results, and — notably — a retrieval-augmented expert agent consults an indexed FDMNES manual to counter parameter hallucination (e.g., verifying the default energy range instead of letting the LLM invent a plausible number).
- BAGEL: knowledge is a precondition for capability — map what the AI knows before trusting it.
- ChemGraph-XANES: orchestration can exist independently of deep domain knowledge — the AI is a director; FDMNES does the physics.
- Shen J, Hagiwara M, Alizadeh M, et al. BAGEL: Benchmarking Animal Knowledge Expertise in Language Models. arXiv:2604.16241 [cs.CL, cs.AI], 2025.
- Grizzi VF, Pham TD, Pretzie LN, et al. ChemGraph-XANES: An Agentic Framework for XANES Simulation and Analysis. arXiv:2604.16205 [cond-mat.mtrl-sci, cs.AI, physics.chem-ph], 2025.
Paper 1: BAGEL — A Closed-Book Zoology Exam
Design
BAGEL asks: without retrieval, RAG, or tools, how much animal knowledge does an LLM retain from pretraining? The questions come from four heterogeneous sources:
The 11,852 four-choice questions cover eight dimensions: taxonomy, behavior, communication, morphology, habitat, cognition, geographic distribution, and diet.
Key findings
What BAGEL does not measure
It tests factual recall, not reasoning (e.g., how would a population respond to habitat loss?), not generation (describing a new species), and not hands-on research capability. It measures the LLM as a *zoology knowledge base* — valuable but limited.
Paper 2: ChemGraph-XANES — Letting Agents Compute Spectra
Why automate XANES
XANES simulation involves ~8 steps: obtain/build the crystal structure, pick the absorber atom, choose methods/parameters, generate FDMNES inputs (fdmfile.txt, fdmnes_in.txt), run on an HPC cluster, parse outputs, normalize spectra, and record provenance. Manual execution does not scale to high-throughput studies of hundreds or thousands of structures.
Architecture: four stacked layers
Single vs multi-agent
Input modes and normalization
Two input modes: file-based (e.g., a POSCAR of an MnO₂(010) slab with Cu, Z=29, as absorber) and pure natural language ("Compute the XANES for Ti in TiO₂"). The pipeline includes standardized spectral normalization: optional E₀ estimation from the derivative maximum, pre-edge (−20 eV) and post-edge (+50 eV) linear baselines, and edge-step normalization — essential for comparability across batch calculations. File-naming conventions (absorber Z, Materials Project ID, formula) provide lightweight provenance.
Honest limitations
The agent cannot freely edit arbitrary FDMNES input parameters — it is constrained to schema-validated tools. This tension between autonomy and reproducibility is deliberate: as LLM reliability improves, the parameter surface can expand, but choices should always remain document-grounded and structurally validated.
Head-to-Head: Two Paths to AI in Science
Breadth of knowledge vs depth of workflow
Each assumption has holes: a high-scoring model may fail at reasoning tasks; an orchestration agent hits a wall when it must judge whether a result is physically sensible (hence the RAG expert agent).
Closed-book vs tool-augmented
BAGEL's closed-book design implicitly assumes only pretraining knowledge counts — reasonable for assessing a general knowledge base, but unrealistic for evaluating a deployed scientific assistant, which will always have retrieval and tools.
Disciplinary differences
Zoology is descriptive/classificatory — its facts are text-like and thus testable by exams. Spectral computation is theoretical/predictive — spectra are *computed*, not recalled; no exam can ask an LLM to draw TiO₂'s XANES from memory. This explains the divergent methodologies.
Deeper Questions
1. Measurement is construction: BAGEL's four sources impose biases (English, Western-centric, recorded-species-only). It measures "zoology expressible in English text," not zoology itself. 2. Interpretability: BAGEL cannot tell *why* a model chose an answer; ChemGraph-XANES's execution traces record *what* the agent did, not *why* — tool choices may be pattern matching rather than genuine decisions. 3. Cargo-cult detection: high exam scores and successful workflow automation do not imply *understanding*. Needed: reasoning-augmented benchmarks and adversarial testing (corrupted inputs, anomalous spectra, failure recovery).
A thought experiment — swapping the two teams — shows each approach has a ceiling: exams test the known, agents automate known workflows; neither directly produces new knowledge. The third capability, *asking good questions*, remains distinctly human.
Conclusion
BAGEL's path: *first measure how much it knows.* ChemGraph-XANES's path: *let it act, regardless.* The author judges the orchestration path more productive — science advances by doing, reproducibly — provided the toolchain stays reliable, transparent, and verifiable. But both paths stop short of Feynman's bar:
> Can the high-scoring AI actually help real conservation projects? Can the spectrum-computing agent judge whether its own result is correct on a structure it has never seen? If both answers are "not sure," then the distance between this work and real science is called *understanding*.
---