[论文] Doc-to-Atom: Learning to Compile and Compose Memory Atoms

论文概要

研究领域: NLP 作者: Xingjian Diao, Wenbo Li, Yashas Malur Saidutta, Avinash Amballa, Lazar Valkov, Srinivas Chappidi 发布时间: 2026-06-10 arXiv: 2606.12400

中文摘要

长输入序列是大型语言模型中文档理解和多步推理的核心，但注意力的二次成本使推理既内存密集又缓慢。上下文蒸馏通过将上下文信息压缩到模型参数中来缓解这一问题，近期工作如Doc-to-LoRA将上下文蒸馏分摊为单个前向传播，为每个文档生成一个LoRA适配器。然而，为所有查询产生单一整体适配器导致不相关查询干扰、有限的组合召回和差的长期文档推理可扩展性。为解决这些挑战，我们提出Doc-to-Atom（Doc2Atom），一种组合参数记忆框架，将每个文档分解为语义类型的知识原子。每个原子被编译为独立的微LoRA适配器和来源检索键。推理时，轻量级查询路由器选择并仅组装相关原子为查询特定适配器，然后注入冻结的基础模型。整个系统通过多目标蒸馏框架端到端训练。在六个多样化QA基准上的实验表明Doc2Atom优于Doc-to-LoRA基线，同时降低文档内化内存成本。

原文摘要

Long input sequences are central to document understanding and multi-step reasoning in Large Language Models, yet the quadratic cost of attention makes inference both memory-intensive and slow. Context distillation mitigates this by compressing contextual information into model parameters, and recent work such as Doc-to-LoRA amortizes context distillation into a single forward pass that generates one LoRA adapter per document. However, producing a single monolithic adapter for all queries leads to irrelevant-query interference, limited compositional recall, and poor scalability to long-document reasoning. To address these challenges, we propose Doc-to-Atom (Doc2Atom), a compositional parametric memory framework that decomposes each document into semantically typed knowledge atoms. Each ato...

--- *自动采集于 2026-06-12*

#论文 #arXiv #NLP #小凯