The Same Sentence, 9x the Price for African Users
Write an email with GPT. In English, it costs about 100 tokens. In Chinese, roughly 150. In Amharic (Ethiopia's official language), 700. In N'Ko (a West African script), 892.
Commercial LLMs bill by tokens, measure latency by tokens, and allocate context windows by tokens. So the same meaning expressed in N'Ko costs 8.92x more money, takes 8.92x longer, and consumes 8.92x more context space than in English.
This isn't model bias—it's tokenizer bias. And it happens before the model is even invoked: the unfairness is baked in before inference begins.
This is The African Language Tax—the first study to systematically quantify the tokenization penalty for African languages.
Tokenizers: The Overlooked Entry Point of Bias
The LLM pipeline has three stages: tokenization → embedding → inference. Most bias research focuses on inference (discriminatory outputs); some on embeddings (biased word vectors). Almost no one has systematically studied the tokenization stage.
But the tokenizer is the first gate. It splits text into subword tokens, each mapped to an ID. Commercial models bill by token count, so "how many tokens a sentence splits into" directly determines cost.
The problem: mainstream tokenizers (GPT's tiktoken, Llama's SentencePiece, Claude's tokenizer, etc.) are trained mostly on English, and their splitting efficiency varies enormously across languages.
English "hello world" might be 2 tokens. Chinese "你好世界" might be 4–6. A single Amharic word might be 7–9 tokens—because the tokenizer has never seen the script and falls back to character-level splitting.
This gap is called token fertility: how many tokens a word splits into in different languages. Higher fertility, higher cost.
20 Languages, 11 Tokenizers, One Brutal Ranking
The paper tested 20 African languages spanning five language families and three writing systems (Latin, Ge'ez/Ethiopic, N'Ko), using parallel corpora (FLORES-200+) so that text content is identical across languages—any difference measured is purely tokenizer-driven, not content-driven.
11 frontier tokenizers were tested: GPT-5/o200k_base, GPT-4/cl100k_base, Claude, Llama 3, Gemma 4, Qwen 3, and more.
Key numbers:
| Metric | Value | |--------|-------| | Median token premium (GPT-5, all African languages) | 1.88x | | Highest token premium (N'Ko, GPT-5) | 8.92x | | Amharic premium (GPT-5) | 7.4x | | Best tokenizer (Gemma 4) average premium | 2.38x | | Worst tokenizer (cl100k_base) average premium | 3.31x | | Lowest effective context window share (N'Ko vs English) | 11% |
Every African language is more expensive than English on every tokenizer. No exceptions. The best tokenizer (Gemma 4) only lowers the average premium from 3.31x to 2.38x—it doesn't eliminate it.
The Script Effect: More Important Than the Language Itself
The most interesting finding: the main driver of token premium is not the language but the writing system.
- African languages in Latin script (Hausa in Latin, Swahili, Yoruba, etc.): moderate premiums of 1.5x–3x. These share the English alphabet, so tokenizers can reuse English subwords.
- Ge'ez-script languages (Amharic, Tigrinya): high premiums, 5x–7x. Ge'ez script is rare in training corpora, so tokenizers split almost character by character.
- N'Ko: highest premiums, 7x–9x. N'Ko is a West African script invented in 1949 and nearly absent from mainstream tokenizer training data.
Real Money: Deployment Cost Calculations
The paper translates token premiums into deployment economics across three scenarios:
Scenario A: High-concurrency chat. A chat API serving 1 million requests/day at 500 tokens per English request costs X. In Amharic, each request becomes 3,700 tokens: total cost 7.4X. At GPT-5 pricing ($5/M input tokens), English users cost $2,500/day; Amharic users $18,500/day. That's a $5.8M/year gap.
Scenario B: Generation-heavy workloads. For long-form generation (reports, summaries), the output-side premium applies too. A 1,000-token English report requires 8,920 tokens in N'Ko—8.92x more expensive and 8.92x slower (since autoregressive generation is token-by-token).
Scenario C: Context-limited settings. On a 4K-token lightweight model, an English user gets 4,000 tokens of context; an N'Ko user effectively gets 4,000/8.92 ≈ 448 tokens—just 11% of English. Many long-context tasks (RAG, long-document summarization) become infeasible in N'Ko.
The Qwen 3 Exception: An Interesting Finding
Among all tokenizers, Qwen 3's handling of N'Ko is an outlier. Where others show 8–9x premiums, Qwen 3 brings it down substantially. The reason is simple: Qwen 3's training data includes more African and Asian non-mainstream scripts.
This shows tokenizer bias is fixable—with enough non-mainstream script data, premiums can drop sharply. The barrier isn't technical; it's that mainstream tokenizer vendors lack incentive, because the commercial market for African languages is too small.
Honest Caveats
1. Token fertility isn't the only cost. The paper acknowledges model quality gaps (models are also worse in African languages), network latency, and other factors. Real-world inequality is larger than what was measured. 2. Parallel corpus translation quality could affect measurements. If FLORES-200+ Amharic translations are unusually long, token counts would be artificially inflated. The paper used SIB-200 and MAFAND-MT for robustness checks (Pearson r=0.9998), indicating stable results. 3. 20 languages don't cover all of Africa. Africa has 2,000+ languages; the 20 tested are major ones, but long-tail languages may fare worse. 4. H4 (correlation between premium and model accuracy) uses third-party data, so causality isn't fully confirmed. Languages with high token premiums see worse model performance, but that may stem from scarce training data rather than tokenization itself.
Implications for the Industry
The paper's biggest contribution isn't the numbers—it's putting a neglected issue on the table: tokenizers are the first gate of AI fairness, and that gate filters people before the model is ever invoked.
Mainstream AI fairness research aims to make model outputs non-discriminatory, but this paper shows: even with a perfectly unbiased model, unequal token allocation creates structural discrimination. An African user and an English user on the same model pay different prices, wait different times, and get different context lengths—all before inference begins.
The fix is clear: expand tokenizers' script coverage. Gemma 4 proves the path works, cutting the average premium from 3.31x to 2.38x. Qwen 3's N'Ko performance shows that with enough training data, a 9x premium can be substantially reduced.
The issue is incentive. Mainstream tokenizers are maintained by commercial companies, and the African-language market is too small to motivate investment. The author (Olaoye Anthony Somide, of DataLens Africa Research and CipherSense AI Technologies) released an open-source measurement tool, afri-fertility, hoping to drive community attention.
When we discuss AI democratization, we can't only discuss open model weights and cheap APIs. If the tokenizer itself is collecting an "African tax," then open and cheap only apply to the English-speaking world.
---
Paper: The African Language Tax: Quantifying the Cost, Latency, and Context Penalty of Tokenizing African languages in Frontier LLMs arXiv: https://arxiv.org/abs/2606.24460 HTML: https://arxiv.org/html/2606.24460v1 Tool: https://github.com/CipherSenseAI/afri-fertility Author: Olaoye Anthony Somide (DataLens Africa Research / CipherSense AI Technologies Ltd)