Paper Slam 4/20: When LLMs Face a Bird and an X-ray Beam

Forum topic · 小凯 · 2026-04-28

Summary

This forum post compares two April 17 arXiv papers representing opposite approaches to AI in science. BAGEL is an 11,852-question closed-book multiple-choice benchmark testing LLM knowledge of zoology across eight dimensions (taxonomy, behavior, communication, morphology, habitat, cognition, geographic distribution, diet) built from Wikipedia, Global Biotic Interactions, bioRxiv, and Xeno-canto. ChemGraph-XANES is an agentic framework that automates X-ray absorption near-edge structure (XANES) simulations by orchestrating ASE, FDMNES, Parsl, and LangGraph/LangChain, including a retrieval-augmented expert agent that consults FDMNES documentation. Key findings: LLMs perform best on structured taxonomy knowledge and worst on communication and diet; a surprising non-monotonic scaling result on Xeno-canto (Qwen3-32B underperforms Qwen3-14B); and the ChemGraph framework deliberately constrains agent freedom via schema-validated tools to preserve reproducibility. The author argues knowledge breadth (closed-book testing) and workflow depth (tool orchestration) are complementary, but tool orchestration is ultimately more productive for science, since AI's real value lies in reliably executing verifiable workflows rather than memorizing facts.

Paper Slam 4/20: When an LLM Faces a Bird and an X-ray Beam

This is a structured summary of a long-form Chinese forum post comparing two arXiv papers released April 17: BAGEL (arXiv:2604.16241), a zoology knowledge benchmark for LLMs, and ChemGraph-XANES (arXiv:2604.16205), an agentic framework for X-ray absorption spectroscopy simulations. One asks "what does the model know?"; the other asks "what can the model do?"

Key points

BAGEL: a closed-book zoology exam for LLMs

11,852 four-choice questions across 8 dimensions: Taxonomy, Behavior, Communication, Morphology, Habitat, Cognition, Geographic Distribution, and Diet.
Sources: Wikipedia (1,927), Global Biotic Interactions (3,500), bioRxiv preprints (2,183), Xeno-canto bird-vocalization database (4,242).
Deliberately closed-book: no retrieval, no RAG, no tool use — testing only what pretraining retained.
Findings:
All models score best on Taxonomy; worst on Communication and Diet, which require linking facts across sources (acoustic descriptions, interaction networks) rather than recalling isolated entries.
Clear position bias in some models — answer order influences choices, a methodological warning for MCQ benchmarks.
Non-monotonic scaling on Xeno-canto: Qwen3-32B underperforms Qwen3-14B, suggesting bigger is not always better for specific knowledge types.
The authors manually audited questions; some distractors are arguably plausible (e.g., "wetland grassland" vs "seasonal wetland"), an inherent limitation of multiple-choice evaluation.
BAGEL measures factual recall, not reasoning, generation, or experimental capability — it tests the LLM as a "zoology knowledge base."

ChemGraph-XANES: an agent that runs spectral simulations

Automates the full XANES workflow: structure retrieval, absorber selection, FDMNES input generation, HPC execution, output parsing, normalization, and provenance.
Four-layer stack: ASE (structure handling), FDMNES (physics engine), Parsl (parallel execution on HPC), LangGraph/LangChain (agent orchestration with typed, schema-validated tools).
Supports single-agent and multi-agent modes (planner / workers / aggregator), plus a retrieval-augmented expert agent that answers parameter questions by consulting the indexed FDMNES manual — mitigating "parameter hallucination."
Two input modes: file-based (e.g., a POSCAR for an MnO₂(010) slab with Cu, Z=29, as absorber) and natural-language ("Compute the XANES for Ti in TiO₂", resolved via Materials Project lookup, Z=22).
Standardized normalization pipeline: optional E₀ estimation, pre-edge (−20 eV) and post-edge (+50 eV) linear baselines, edge-step normalization — critical for comparability and downstream machine learning.
Key limitation: agents cannot freely edit FDMNES input files; only predefined, validated parameters are exposed, preserving reproducibility. The authors argue to expand agent control gradually as LLM reliability improves.

The comparison

Knowledge breadth vs workflow depth: BAGEL maps where LLM zoology knowledge is strong or blank; ChemGraph-XANES automates one concrete computational task end-to-end.
Closed-book vs open-book with tools: BAGEL's philosophy assumes knowledge reserve precedes capability; ChemGraph assumes orchestration can substitute for domain knowledge, delegating physics to FDMNES. The latter matches real scientific practice — no scientist works closed-book.
Field matters: zoology is descriptive and fact-based, well suited to MCQ testing; spectral simulation is predictive and computational — no exam can test it, the model must compute.

Deeper reflections (Feynman-style questions)

1. Measurement is construction: BAGEL's four English-language sources define its own boundary of "zoology" — traditional ecological knowledge and non-textual expertise are silently excluded. 2. Interpretability: BAGEL can't explain why an answer is wrong; LangChain traces show what the agent did, not why. 3. Cargo-cult risk: high exam scores and successful automation don't imply "understanding." Suggested remedies: reasoning-based questions in benchmarks, adversarial testing (faulty inputs, anomaly detection) for agents.

Conclusion

The two paths are complementary — a useful scientific AI needs both knowledge and tool orchestration — but the author judges the ChemGraph direction more productive: science's core is not knowing but doing, provided the toolchain remains reliable, transparent, and reproducible. Neither path directly produces *new* knowledge; asking good questions remains the human researcher's core competitive edge.

References

Shen J, Hagiwara M, Alizadeh M, et al. *BAGEL: Benchmarking Animal Knowledge Expertise in Language Models*. arXiv:2604.16241 [cs.CL, cs.AI], 2025.
Grizzi VF, Pham TD, Pretzie LN, et al. *ChemGraph-XANES: An Agentic Framework for XANES Simulation and Analysis*. arXiv:2604.16205 [cond-mat.mtrl-sci, cs.AI, physics.chem-ph], 2025.