[论文] Stochastic Estimation of Transduced Language Models

论文概要

研究领域: NLP 作者: Vésteinn Snæbjarnarson, Samuel Kiegeland, Manuel de Prada Corral, Ryan Cotterell, Tim Vieira 发布时间: 2026-08-27 arXiv: 2608.27428

中文摘要

转导语言模型（TLMs）将预训练的源语言模型与功能有限状态转导器组合，诱导出目标字符串上的语言模型。计算TLM下目标前缀的概率等于对所有转导器映射到以该前缀开头的目标字符串的源字符串的源模型概率求和。这个集合可能是指数级大或无限的。先前工作使用基于源前缀概率的计算捷径，然后用阈值剪枝束求和近似结果，产生有未知误差的下界。相反，我们无放回地重采样源前缀，并按包含概率的倒数重新加权每个选择的前缀。我们证明递归应用这种校正可以得到目标前缀概率的无偏估计器，并让我们估计阈值剪枝损失的质量。我们的束求和算法扩展保留的源前缀并采样保留哪些前缀，随着更多概率质量被添加到运行估计中而减少其数量。这在文本百科和DNA上实现了更好的计算-方差权衡。

原文摘要

Transduced language models (TLMs) compose a pretrained \emph{source} language model with a functional finite-state transducer to induce a language model over \emph{target} strings. Computing the probability of a target prefix under a TLM amounts to summing the source-model probabilities of all source strings that the transducer maps to target strings beginning with that prefix. This set can be exponentially large or infinite. Prior work uses a computational shortcut based on source prefix probabilities, then approximates the resulting sum with threshold-pruned beam summing. This produces a lower bound with unknown error. Instead, we resample source prefixes without replacement and reweight each selected prefix by the inverse of its inclusion probability. We show that applying this correcti...

#论文 #arXiv #NLP #小凯