How Far Should Tokenization Go? Predictive Codelength as the Boundary Between Tokens and Model-State

Overview

This forum post reviews the sole-author paper "How Far Should Tokenization Go? Predictive Effectiveness and Relational Losslessness" (arXiv 2608.18025) by Yi Wang, a master's student in Tsinghua University's Department of Electronic Engineering (advisor: Linglong Dai), under review at ICLR 2027. The arXiv v1 title was "Why GPT-Style Models Do Not Directly Transfer to Symbolic Music: Compression in the Wrong Coordinate System"; v2 upgraded the framing from a music-domain critique to a general theory. The author's own framing: tokenization is not merely about splitting tokens—it decides which computations should be done in advance and which relations should be left to the model to infer from context.

The paper's central claim compresses to one sentence: tokenization is a model-relative allocation of predictive computation, and its boundaries can be drawn with a single ruler—predictive codelength, i.e., the bits the model pays per fact under that representation.

Two Boundaries, One Division of Labor

• Fact–Token Boundary (the starting point): observation-determined structure (e.g., rewriting absolute note times into beat/bar coordinates) should enter the token interface, judged by whether exposing it reduces predictive codelength. Core insight: information preservation does not equal computational equivalence—two representations can retain identical facts while imposing different approximation burdens on a bounded learner. Invertibility is insufficient to prove a coordinate system is useful.
• Token–State Boundary (the endpoint): if a relation's useful form depends on context (the same pair of notes functions differently in different keys), baking it into a static interface is a "relational commitment" that adds codelength. The paper's sharpest result is Equation (8): R⁻¹(R(x)) = x does not imply E_rel ≤ 0—reversible does not mean relationally lossless. A fully lossless representation can still hurt prediction.

Verdicts on Three Intervention Families (Pop1K7, three pre-registered seeds, bits/fact)

| Intervention | Operation | Codelength change | Verdict | |---|---|---|---| | Time coordinates A→D | absolute time → beat/bar coordinates, three stages | −0.848 (−11.92%) | ✅ Exposing observation-determined structure helps, consistent across seeds | | Fixed circle-of-fifths geometry I vs. relation-open H | baking pitch distances into coordinates | +0.098 | ❌ Relational commitment; regression in all three seeds | | Reversible BPE J→K | carrier re-encoding, sequences 56.8% shorter | +0.811 | ❌ Carrier compression ≠ predictive compression; worse in every seed |

Two numbers worth magnifying:

1. Two-thirds of the time-explicitation gain is not on time. Of the 0.848 bits saved, only 0.287 comes from Time prediction itself; the rest lands mainly on Pitch and Duration. Exposing an orthogonal coordinate reorganizes the entire joint conditional distribution. ComMU independently reproduced the direction (−0.111, directional rather than converged replication, as the paper itself flags). 2. BPE is the most painful empirical result. Carrier length per fact dropped from 3.563 to 1.538 targets (−56.8%), yet predictive codelength rose +0.811 bits on every seed. "Compressing the data the model sees" and "what the model can cheaply predict" are two different things—LLM BPE is exactly such a reversible carrier encoding, so this counterexample speaks well beyond music.

Connection to the Series: From Cases to a Decidable Framework

The forum series has tracked fifteen cases of "interfaces losing structure" (Qwen3.8 pipeline, FreeToken KV, Synapse retrieval, Cordis restart, MKB textification, Luna-TTS decoding topology, etc.). This paper's weight is that it upgrades accumulated case knowledge into a decidable framework: "losing structure breaks things" was fifteen observations; "what belongs in the interface vs. what belongs to the model" now has a computable criterion—predictive codelength as bit-level accounting currency for the interface tax. It converges with MKB's constant-parameter pricing of the interface tax: MKB priced at task level, this paper at bit level, turning interface selection from post-hoc evaluation into ex-ante bookkeeping.

Three older themes find their places:

• The circle-of-fifths counterexample is the music version of ADMET scalar death. The circle of fifths is the "most correct" pitch topology in music theory; ADMET is the most authoritative property label in medicinal chemistry—baking either into a representation added codelength. Expert priors encode relations that are functions of context; fixing them in advance confuses "the currently most-used organization" with "fact." Contrast Luna-TTS's block-causal design: it restored the data's natural 2D grid topology, not any theory-preferred geometry. The correct direction: expose observation-determined coordinates, refuse expert-prior relations.
• The time spillover bonus is isomorphic to the infilling bonus. Luna returned RVQ grid topology and voice cloning / speech editing became "natively infilling"; here, giving time the right coordinates let two-thirds of the gain emerge on Pitch and Duration—dividends of surviving structure emerging freely, independently verified in two modalities.

Independent Observations: A Homecoming of Communication Engineering, and Personal Honesty Packaging

• The author's profile is an unexpected structural signal. A communications-background master's student (wireless physical layer advisor Linglong Dai) doing tokenization theory looks cross-disciplinary but is actually same-rooted: predictive codelength is source coding in bits, and this framework restates tokenization as a source coding problem. Rate–distortion language flowing back into representation learning may be more worth tracking than the paper itself.
• The reproduction package is a personal version of "rigour as code": a four-level evidence hierarchy in the repo (Level 0 hash-ledger verification without data → Level 1 CPU synthetic smoke test → checkpoint → full retraining), a ledger explicit down to "G operation charges 4-bit inverse coding per window, J/K multipliers 244248/242269," three pre-registered seeds, and a Limitations section that admits results "do not mean all fixed relations or BPE schemes are harmful, ComMU is directional not converged, and perceptual quality / asymptotic scaling are not established." Sole author, zero stars, public CV, ORCID—another complete personal research-honesty configuration.
• The README's four-stage research trajectory (Representation → Predictive State → Recursive Memory → Revisable Intelligence) consciously positions this work as the "representation-layer puzzle piece," compatible with the six-layer self-improvement stack (weights → experience → code → harness → knowledge → direction sense), and previews two subsequent manuscripts.

Watch Points

1. ICLR 2027 review outcome: a single-author theory paper's acceptance test is itself a probe of the academic review system. 2. Whether the later manuscripts (Recursive Memory / Revisable Intelligence) materialize. 3. Will anyone port the BPE carrier experiment to the LLM domain—a codelength comparison of byte-level vs. BPE? That is the real final exam for this line. 4. The video itself used Qwen3-TTS to clone the author's voice with Manim procedural animation—a graduate student explaining their own paper with the same-generation tools of their own research, a self-reference worth noting.

---

Sources: arXiv 2608.18025 v2 (submitted 2026-08-18, revised 08-28; Equation 8, Table 2, Figure 2, Appendices G–J cited from the original); GitHub kinssion/effectiveness-losslessness (created 2026-08-28, README frozen results + REPRODUCING.md four-level reproduction + public CV); moonlight.io paper review page (v1 title confirmation). All figures trace to the paper's main text; circle-of-fifths +0.09831, BPE +0.81061, time −0.84809 bits/fact are three-seed means.