[论文] ArchSpace: Recording, Reviewing, and Referencing LLM Architecture Rese...

论文概要

研究领域: NLP
作者: Jie Wang, Tong Zhu, JiaHao Kuang, Zhe Xu, Yunkai Chai, Dehong Zheng, Yu Cheng, Tao Ji, ZhouHan Lin, Qi Zhang, Bowen Zhou
发布时间: 2026-11-12
arXiv: 2611.17332

中文摘要

模型架构决定 LLM 的扩展行为、训练效率和推理成本，是工业界与学术界共同关注的核心。工业界如 DeepSeek、Qwen、Kimi 等团队持续推出架构创新，但公开报告鲜少记录哪些想法成功了、哪些被放弃了、以及为什么。学术界则受限于算力和不一致的基线，评估往往停留在小规模单点对比，难以验证改进能否扩展到更大规模或产生更广泛影响。ArchSpace 作为记录、评审与引用架构研究的公共平台，同时弥合这两个缺口：每个提案连同实验与推理过程一并记录，由社区评审，并被后续工作引用。为在昂贵的大规模训练前评估想法，ArchSpace 利用缩放定律从小规模实验预测更大规模的模型损失——这是工业界广泛采用的做法。我们提供初始参考使平台可直接使用：Qwen3.5 与 OLMo3 基线、拟合好的缩放定律、以及可复用的训练配方。通过评审的想法可成为新的基线，ArchSpace 由此旨在拓宽塑造下一代模型的想法来源。

原文摘要

Model architecture shapes the scaling behavior, training efficiency, and inference cost of LLMs, making it a central focus of both industry and academia. In industry, teams such as those behind DeepSeek, Qwen, and Kimi routinely introduce architectural innovations, yet public reports rarely document which ideas succeeded, which were abandoned, and why. In academia, limited compute and inconsistent baselines often restrict evaluations to small-scale, single-point comparisons, making it difficult to establish whether improvements hold at larger scales or translate into broader impact. ArchSpace addresses both gaps as a public platform for Recording, Reviewing, and Referencing Architecture Researc}: each proposal is recorded with its experiments and reasoning, reviewed by the community, and r...

自动采集于 2026-11-14

#论文 #arXiv #NLP #小凯