[论文] Breakdown of Local Denoising as Semantic Speciation

论文概要

研究领域: ML
作者: Guangkuo Liu, Mert Okyay, Yifan F. Zhang, Fangjun Hu, Rahul Nandkishore, Xun Gao
发布时间: 2026-09-29
arXiv: 2609.38176

中文摘要

生成模型的动力学表现出两个看似不同的时间窗口：一个是物种形成窗口（speciation window），样本在此阶段确定语义类别；另一个是非局部性窗口（nonlocality window），局部上下文窗口在此阶段变得不足以支持生成。受多种前沿模型中两者近乎同时出现的证据启发，我们通过语义信息的空间分布来研究它们之间的关系。在'共同原因'假设下，我们证明非局部性窗口必须位于物种形成窗口之内。该假设假设语义标签解释了远距离token之间相关性的一个比例，这一条件对许多真实数据集而言是自然的。我们进一步给出了两个窗口收缩为系统规模增长时的单一极限时间的条件，定义了一个'相变'，并在高斯混合模型中解析验证了这一行为。这些结果共同确定了语义信息解释物种形成与非局部性同时出现的条件，连接了关于生成模型中语义结构涌现的两种互补视角。

原文摘要

The dynamics of generative models exhibit two apparently distinct temporal windows: a speciation window, in which a sample commits to a semantic class, and a nonlocality window, in which local context windows become insufficient for generation. Motivated by evidence of their near-concurrence in a variety of frontier models, we investigate their relationship through the spatial distribution of semantic information. Under a "common cause" hypothesis, we prove that the nonlocality window must lie in the speciation window. This hypothesis postulates that semantic labels explain a fraction of the correlations between distant tokens, a condition that is natural for many real datasets. We further give conditions under which both windows shrink to a single limiting time as system size grows, defin...

自动采集于 2026-10-01

#论文 #arXiv #ML #小凯