[论文] From Prose to Properties: Testing Research Claims with ARATest for Age...

论文概要

研究领域: ML 作者: Yifeng He, Jiachen Liu, Hao Chen 发布时间: 2026-10-24 arXiv: 2610.30495

中文摘要

AI 智能体现在自己跑实验、报结论，研究产出主张的速度快过研究者核验的速度。每个实证主张都蕴含一个必须在其案例上成立的性质，但它只以散文陈述这一性质——没有显式性质，就没有检验能判定哪些证据会证伪主张，验证便追不上智能体生产的主张。ARATest 通过抽象「性质骨架」——跨主张复现的参数化关系——逼近主张语义，并把骨架作为可执行测试检查。类型化声明把主张绑定到骨架，固定其比较量、案例、范围与数值策略；共享案例类型让每个生成器按主张的统计单元供给案例；三值语义给每个性质判定「成立/违反/不确定」之一，每个违反携带可复现的 witness；默认生成器读取 agent-native 研究工件（ARA）链接到每条主张的证据，auto-research 生成器派智能体运行开源代码获取新案例；DissClaimer 插件供应该智能体并把这些生成器作为持续测试运行。33 个 ARA 工件的 243 条主张中，54 条获得忠实声明并绑定证据：32 条成立、8 条违反、14 条不确定。两处违反暴露了已发表论文中经作者确认的错误。ARATest 检出全部 553 个种子故障、对 19 个有效控制零误报，优于两个 LLM 裁判。新鲜运行揭示了报告证据无法呈现的行为。这些结果表明：一个固定的测试，可以同时对报告证据与新鲜证据检查显式的主张性质。

原文摘要

AI agents now run experiments and report findings, so research produces claims faster than researchers can check them. Each empirical claim implies a property that must hold over its cases, but it states that property only in prose. Without an explicit property, no test can decide which evidence would falsify a claim, so validation cannot keep pace with the claims agents produce. To make claims testable, we approximate a claim's semantics by abstracting its property skeleton, a parameterized relation that recurs across claims. We present ARATest, a property-based testing library that checks these skeletons as executable tests. A typed declaration binds a claim to a skeleton and fixes the claim's compared quantities, cases, scope, and numerical policy. A shared case type makes every generator supply cases in the claim's statistical unit. A three-valued semantics assigns each property one of three verdicts: holds, violated, or inconclusive. Every violation carries a replayable witness. A default generator reads the evidence that agent-native research artifacts (ARA) link to each claim, and auto-research generators dispatch an agent to run released code for fresh cases. Our DissClaimer plugin supplies that agent and runs these generators as continuous testing. Of 243 claims in 33 ARA artifacts, 54 received faithful declarations bound to evidence: 32 held, eight were violated, and 14 remained inconclusive. Two violations exposed errors in published papers that their authors confirmed. ARATest detected all 553 seeded faults without a false alarm on 19 valid controls, outperforming two LLM judges. Fresh runs exposed behaviors that reported evidence could not show. These results suggest that one fixed test can check explicit claim properties against both reported and fresh evidence.

---

*自动采集于 2026-10-24*

#论文 #arXiv #ML #小凯