An Overlooked Inequality
Process 1,000 characters of Chinese with GPT-4, versus 1,000 characters of English, and the cost can differ by up to 5x.
Not because Chinese is billed at a higher rate — it's because the tokenizer splits Chinese into more tokens.
Specifically, BBPE (Byte-Pair Encoding on UTF-8 bytes) is the mainstream tokenization scheme today. Under UTF-8 encoding:
- English letters (ASCII): 1 byte → 1 base symbol
- Cyrillic, Arabic letters: 2 bytes → 2 base symbols
- Chinese, Japanese, Korean (CJK): 3 bytes → 3 base symbols
- Emoji (non-BMP): 4 bytes → 4 base symbols
- Characters with ≤ 2 UTF-8 bytes (ASCII, Latin, some Cyrillic/Arabic) → UTF-8 path, 1-2 base symbols
- Characters with 3 UTF-8 bytes (CJK, most non-BMP characters) → UTF-16 path, 2 base symbols
- Characters with 4 UTF-8 bytes (emoji and other characters outside BMP) → UTF-16 path, 4 base symbols (same as BBPE)
- UTF-8 bytes use GPT-2's byte alphabet (0x00-0xFF)
- UTF-16 bytes use a Private Use Area (U+E100-E1FF)
- MYTE: 1,138,306 / 1,155,724 (missing 17,418)
- SCRIPT-BPE: applies NFC normalization and filters Private-Use and unassigned scalars
- UBE: 1,155,724 / 1,155,724 (100%)
This means the same content written in Chinese consumes 3-5x more tokens than in English. On the FLORES-200 benchmark, BBPE's average token count is 200.188, versus only 34.863 for English — a 5.7x gap. Some languages (e.g., Shan) reach 15x.
This gap means: 1. Higher API bills for Chinese users 2. Chinese text takes up more room in the context window 3. Chinese models have an "effective context" 3-5x shorter than English 4. Chinese data gets "diluted" in multilingual training
Universal Byte-Level Encoding (UBE) proposes an extremely simple fix.
UBE's Core Idea: One Routing Rule
UBE's idea is simple enough to seem suspicious: route each character to UTF-8 or UTF-16 based on its UTF-8 byte count.
The specific rules:
The effect: the encoding floor for CJK drops from 3 to 2, while ASCII stays at 1.
It's like a bilingual postal system: local mail goes by land (cheap), distant mail goes by air (fast but expensive). UBE picks the "cheapest" encoding path per character.
Why Not Just Use UTF-16 Everywhere?
Since UTF-16 reduces CJK to 2 bytes, why not use it universally?
Because UTF-16 turns ASCII into 2 bytes too. English goes from 1 to 2 — English users pay double. That's robbing Peter to pay Paul.
BBPE16 (Kim et al., 2026) does exactly this — a global switch to UTF-16. CJK gets cheaper, but ASCII gets more expensive. Fine for CJK-heavy scenarios (e.g., Chinese ASR), but a net loss in mixed-script settings (Chinese-English webpages, code plus comments).
UBE's cleverness: not picking one encoding globally, but choosing the optimal encoding per character. ASCII takes the UTF-8 path and stays at 1 byte; CJK takes the UTF-16 path and drops to 2. Neither side loses.
Marker-Free Switching
The routing rule sounds simple, but there's a technical catch: how does the decoder know whether a base symbol came from UTF-8 or UTF-16?
The traditional fix is marker tokens — prefix each span with "this is UTF-8" or "this is UTF-16." But markers get absorbed by BPE merges, breaking reversibility.
UBE's solution is elegant: use two disjoint 256-symbol alphabets.
The alphabets are disjoint, so the decoder just checks which alphabet a base symbol belongs to and knows its path. No marker tokens, no restrictions on BPE merges, 100% reversibility guaranteed.
It's like two people speaking different dialects — you can tell where someone is from by their accent, no introduction needed.
100% Reversibility: Zero Failures Across 204 Languages
Reversibility is a tokenizer's baseline requirement: decoding after encoding must recover the original text. BBPE occasionally fails in practice (e.g., Unicode normalization changing characters).
UBE achieves a 100% pass rate across FLORES-200's 204 languages — zero decode failures.
More strictly, in the complete Unicode 17 audit (all Unicode scalar values plus official normalization, grapheme-break, and emoji test suites), UBE is the only scheme that recovers every input:
No Architecture Changes
Another key property: UBE changes neither the Transformer architecture nor the BPE training pipeline.
1. BPE's pair-counting and merge-selection logic are completely unchanged 2. Only the base alphabet grows from 256 to 512 (the union of two disjoint 256-symbol sets) 3. BPE learns the vocabulary from the 512-symbol routed stream 4. Transformer architecture, training, and inference are all unchanged
So UBE can be a drop-in tokenizer replacement — no model redesign needed. Tokenizer swaps are usually invasive; UBE confines the change to the tokenizer layer.
The Encoding Floor Concept
UBE introduces a useful concept: encoding floor — the minimum number of base symbols a character costs under a given scheme.
| Character class | BBPE | BBPE16 | UBE | |---------|------|--------|-----| | ASCII (a-z, 0-9) | 1 | 2 | 1 | | Latin extended (é, ñ) | 2 | 2 | 2 | | CJK | 3 | 2 | 2 | | Emoji (😀) | 4 | 4 | 4 |
UBE is never worse than BBPE or BBPE16 in any class, and clearly better for ASCII and CJK. It's not optimal on one encoding — it's no worse than optimal on all of them.
Why This Matters
Token fairness isn't a mere technical detail:
1. Economic fairness: Chinese users pay more for the same service. In an 8K context window, English fits ~8,000 words; Chinese might fit only 2,000-3,000. 2. Performance fairness: Shorter "effective context" for Chinese means worse long-document comprehension. 3. Training fairness: Chinese data gets diluted — the same token budget trains on less Chinese text. 4. Evaluation fairness: Token-level metrics (perplexity per token) aren't comparable between Chinese and English models.
UBE isn't "making Chinese cheaper" — it's "making the encoding scheme stop systematically penalizing certain scripts." This is a fairness problem at the infrastructure level.
A Deeper Insight: Encoding Choice Is a Bias
UBE reveals something: encoding schemes are not neutral.
UTF-8 was optimized for ASCII — ASCII at 1 byte, everything else multi-byte. Reasonable in the 1980s (computing was mostly English-speaking), but in the global AI era of 2026, it's a systematic bias.
BBPE inherits that bias — it runs BPE directly on UTF-8 bytes, so ASCII wins by default. BBPE16 tries to correct it but overcorrects.
UBE's approach: acknowledge that encoding choice is a bias, then design a routing mechanism that minimizes it. Not "bias-free" — emoji still costs 4 bytes — but choosing the "maximally fair" option among all feasible ones.
Encoding schemes aren't technical details; they're power allocation. Whose script is cheaper has their content processed, stored, and retrieved more cheaply. UBE turns this implicit power allocation into explicit engineering design.
---
Paper: Universal Byte-Level Encoding: UTF-8/UTF-16 Routing for Tokenizer Equity Across Scripts