[论文] SciResearch-Eval: Evaluating Scientific Judgment in Autonomous Experim...
论文概要
研究领域: ML
作者: Brian T. Weston, Alondra Torres Contreras, Aldair E. Gongora
发布时间: 2026-11-16
arXiv: 2611.21541
中文摘要
建筑级野火脆弱性评估对于减少荒地-城市交界社区的损失至关重要,但现有方法往往无法将地块尺度的点火路径与社区尺度的地理覆盖相结合。本研究引入 Wildfire-SSR(卫星和街景用于野火风险),一个地理定位感知的多模态框架,整合宏观尺度卫星上下文与微观尺度街景证据。视觉语言模型从灾前街景全景图中提取与野火相关的建筑和场地属性,并将其转化为结构化语义文本。这些特征通过解耦的交叉注意力融合解码器与卫星图像进行空间对齐,同时冻结的卫星预训练 DINOv3 编码器捕捉俯视视觉上下文。该框架在 2025 年洛杉矶 Eaton 和 Palisades 火灾中约 37,000 栋受影响建筑上进行了测试。旋转位置嵌入提供了最强的空间定位,建筑定位 F1 得分为 0.6735,五类像素级风险分类的调和平均 F1 得分为 0.2476,优于单模态基线和无锚定多模态配置。该模型在较高风险类别上表现出保守偏差,减少了严重风险低估,并在单个 GPU 上 37 分钟内完成研究区域的端到端评估。
原文摘要
We introduce SciResearch-Eval, an interactive benchmark of scientific judgment built on a standard laboratory problem: measuring a material's thermal properties by heating it and fitting a heat-conduction simulator. An agent designs its own experiments, uses a supplied simulator that may be under-resolved or physically wrong, and reports each property with a 95% interval. To our knowledge it is the first benchmark to combine agent-designed experiments, a deliberately flawed simulator, and proper scoring of calibrated intervals against exact ground truth. On a held-out split (93 tasks, 9 models, 2,511 episodes), models handle procedural traps. When the simulator is missing physics, most report wrong values while stating they are 95% sure: their ranges contain the truth in 20-62% of cases, a...
自动采集于 2026-11-18
#论文 #arXiv #ML #小凯