Summary
A recent arXiv paper (2610.31574) proposes TTR-Compressor, a new approach to sequence compression in efficient language and multimodal models. The authors, including Sihan Yang, Jie Huang, and Weiyang Liu, reframe sequence compression as a test-time regression problem: input tokens define a regression task whose fitted coefficients form the compressed representation. The regression problem is learned end-to-end via backpropagation, while the coefficients themselves are computed efficiently in closed form during the forward pass. Unlike prevailing methods based on pooling or learned projections, which often underperform, TTR-Compressor serves as a general-purpose alternative. As a block-wise sparse attention compressor, it substantially improves performance on the RULER benchmark. For visual token merging in multimodal models, it consistently outperforms the standard MLP merger and matches its performance while reducing visual tokens by 55.6%. Published October 2026 on arXiv, this work offers a unified, principled framework for both text and vision token compression.
Paper Overview
Field: Machine Learning
Authors: Sihan Yang, Jie Huang, Hanxuan Li, Yifan Guo, Weigao Sun, Yiran Zhong, Weiyang Liu
Published: 2026-10-22
arXiv: 2610.31574
Abstract
Sequence compression is a fundamental operation and critical information bottleneck in efficient language and multimodal models, yet prevailing approaches based on pooling or learned projections remain underperforming.
The authors revisit sequence compression through the lens of test-time regression:
- Input tokens define a regression problem whose fitted coefficients form the compressed representation.
- The regression problem is learned end-to-end via backpropagation.
- Coefficients are computed in closed form during the forward pass, making the method efficient.
This leads to TTR-Compressor, a general alternative to conventional sequence compression techniques. Key results:
- Block-wise sparse attention compression: TTR-Compressor improves RULER performance by a large margin.
- Visual token merging: It consistently outperforms the standard MLP merger and matches its performance while reducing visual tokens by 55.6%.
Links
- Paper: <https://arxiv.org/abs/2610.31574>
---
*Auto-collected on 2026-10-26*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178636476