English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

SciResearch-Eval: A Benchmark for Scientific Judgment in Autonomous Experimentation Agents

Forum topic · 小凯 · 2026-11-17

Summary

SciResearch-Eval (arXiv:2611.21541) is an interactive benchmark designed to test the scientific judgment of autonomous AI research agents. Built around a standard laboratory problem—measuring a material's thermal properties by heating it and fitting a heat-conduction simulator—the benchmark requires an agent to design its own experiments, use a supplied simulator that may be under-resolved or physically wrong, and report each property with a 95% confidence interval. According to the authors, it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. Evaluation on a held-out split covering 93 tasks, 9 models, and 2,511 episodes shows that models can handle procedural traps. However, when the simulator is missing key physics, most models report incorrect values while claiming 95% certainty: their reported ranges contain the ground truth in only 20–62% of cases, indicating poor calibration. The forum post also includes a Chinese-language summary of a separate wildfire vulnerability study (Wildfire-SSR), which fuses pre-fire street-level panoramas with satellite imagery using a DINOv3 encoder and cross-attention fusion, achieving a building-localization F1 of 0.6735 on roughly 37,000 structures affected by the 2025 Los Angeles Eaton and Palisades fires, completing evaluation in 37 minutes on a single GPU.

Paper Overview

Research area: ML Authors: Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora Release date: 2026-11-16 arXiv: 2611.21541

Abstract (translated)

We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20–62% of cases.

*Note: the source post's excerpt of the original abstract is truncated at this point.*

Related summary in the post

The post also carries a Chinese-language summary of a geolocation-aware multimodal framework, Wildfire-SSR, for building-level wildfire vulnerability assessment:

  • Combines macro-scale satellite context with micro-scale street-view evidence; a vision-language model extracts wildfire-relevant building and site attributes from pre-fire street-view panoramas as structured semantic text.
  • Features are spatially aligned with satellite imagery via a decoupled cross-attention fusion decoder, with a frozen satellite-pretrained DINOv2/DINOv3-style encoder capturing overhead visual context.
  • Evaluated on ~37,000 affected buildings from the 2025 Los Angeles Eaton and Palisades fires; rotated positional embeddings gave the best spatial grounding (building localization F1 = 0.6735; five-class pixel-level risk harmonic-mean F1 = 0.2476), outperforming unimodal baselines and ungrounded multimodal configurations.
  • The model shows a conservative bias toward higher risk classes, reducing severe risk underestimation, and completes an end-to-end study-area assessment in 37 minutes on a single GPU.
*Automatically collected on 2026-11-18.*

Tags

#machine-learning#benchmark#autonomous-agents#scientific-judgment#calibration#multimodal#wildfire-risk

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637214