Summary
Weight Pair Encoding (WeightPE) is a new method that explicitly fine-tunes neural network weights so they admit a smaller grammar. It places a lossy Re-Pair compressor inside a straight-through estimator: the network's int8 weights are flattened into a single string, near-matching Re-Pair patterns are made exactly equal within a global L2 budget, and the network computes with the rewritten weights while training passes through them. Unlike flat codebooks of fixed-size entries, grammars offer variable-length patterns that are reused hierarchically inside larger ones. On the MLP weights of ViT-B/16 and ViT-L/16 fine-tuned on CIFAR-10, WeightPE produces Re-Pair grammars 0.43x and 0.38x the size of those from equivalent int8 QAT runs, at a cost of 1.9 and 1.1 accuracy points respectively. The trend also extends to other grammar compressors (LZ78, SEQUITUR) that the network was not fine-tuned for. The authors claim this is the first work using grammar size as an explicit training objective for network weights.
Paper Overview
Field: Machine Learning
Authors: Irene Tallini, Daniele Solombrino, Alberto Cazzaniga, Emanuele Rodolà
Published: 2026-09-25
arXiv: 2609.31564
Abstract
We show that neural network weights can be explicitly fine-tuned to admit a smaller grammar. Weight Pair Encoding (WeightPE) does so by placing a lossy Re-Pair compressor inside a straight-through estimator. The int8 weights of the network are flattened into one string, and near-matching Re-Pair patterns are made exactly equal within a global L2 budget. The network computes with the rewritten weights and trains through them with a straight-through estimator.
Unlike a flat codebook of fixed-size entries, a grammar offers variable-length patterns and reuses them hierarchically inside larger ones.
On the MLP weights of ViT-B/16 and ViT-L/16 fine-tuned on CIFAR-10, WeightPE produces a Re-Pair grammar 0.43x and 0.38x the size of the one produced by an equivalent int8 QAT run, at a cost of 1.9 and 1.1 accuracy points respectively. This trend extends to different grammar compressors (LZ78, SEQUITUR) that the network was not fine-tuned for. To the authors' knowledge, this is the first time grammar size has been used as an explicit training objective for network weights.
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178635341