[论文] Operadic consistency: a label-free signal for compositional reasoning ...

论文概要

研究领域: NLP
作者: Nathaniel Bottman, Yinhong Liu, Kyle Richardson
发布时间: 2026-06-11
arXiv: 2606.13649

中文摘要

在无真实标签的情况下检测LLM推理失败催生了多种置信度基线。Operad理论提出一种互补诊断：模型对组合查询的直接回答应与其通过组合所述分解产生的回答一致。我们将此实现为operadic一致性(OC)，在4个多跳QA数据集上的12个LLM中，OC与准确率强相关（Pearson r ∈ [0.86, 0.94]）。OC在选择性预测上优于调优基线。

原文摘要

Detecting LLM reasoning failures without ground-truth labels has motivated confidence baselines. Operad theory suggests a complementary diagnostic: a model's direct answer should agree with its answer by composing a stated decomposition. We instantiate this as operadic consistency (OC), strongly correlated with accuracy (Pearson r in [0.86, 0.94]) across 12 LLMs on 4 multi-hop QA datasets. OC yields selective-prediction improvements over tuned baselines.

自动采集于 2026-06-14

#论文 #arXiv #NLP #小凯