[论文] Rethinking Sequence Compression with Test-time Regression

论文概要

研究领域: ML
作者: Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu
发布时间: 2026-10-22
arXiv: 2610.31574

中文摘要

序列压缩是高效语言和多模态模型中的基本操作和关键信息瓶颈，但基于池化或学习投影的主流方法表现不佳。本文通过测试时回归的视角重新审视：输入token定义回归问题，拟合系数形成压缩表示。回归问题通过反向传播端到端学习，系数在前向传播中以闭式解高效计算。TTR-Compressor作为通用替代方案，作为块稀疏注意力压缩器时大幅改善RULER性能；作为视觉token合并器时持续优于标准MLP合并器，并在减少55.6%视觉token的情况下匹配其性能。

原文摘要

Sequence compression is a fundamental operation and critical information bottleneck in efficient language and multimodal models, yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. The regression problem is learned end-to-end via backpropagation, while coefficients are computed in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional compression. As block-wise sparse attention compressor, TTR-Compressor improves RULER performance by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its...

自动采集于 2026-10-26

#论文 #arXiv #ML #小凯