Building an AI-Specific Compressed Language Inspired by DeepSeek-OCR's Visual Compression

Forum topic · ✨步子哥 · 2025-10-23

Summary

This forum post proposes an AI-specific compressed language that transfers DeepSeek-OCR's visual context compression idea into pure text. Instead of mapping text to vision tokens, the author maps sentence embeddings to sequences of rare Chinese characters that are not human-readable but could densely activate an LLM's internal representations. The approach is grounded in compressive sensing: a semantic vector (e.g., d=768) is projected through a random or learned matrix to a much shorter vector (m=16~32), whose values are quantized against a codebook of rare characters (K=128~512). The post presents four progressive stages: (1) a static prototype in Java demonstrating embedding-to-Hanzi encoding with cosine similarity around 0.86-0.89; (2) a learnable encoder-codebook-decoder in PyTorch trained with cosine reconstruction loss; (3) an entropy-regularized, noise-augmented version plus a verification plan comparing hidden-state activations between original text and compressed sequences in Qwen2.5; and (4) an inter-agent communication protocol where two LLMs exchange compressed Hanzi sequences as a token-efficient channel. Claimed benefits include 10x-20x context compression, text-level analogues of DeepSeek-OCR's context optical compression, and potential memory-decay modeling via progressive sequence blurring. Full Java and Python code listings accompany each stage.

This post (translated from zhichai.net) explores constructing an AI-specific compressed language by transferring the "visual compression" idea of DeepSeek-OCR into the linguistic domain itself — replacing vision tokens with sequences of rare Chinese characters that are dense, machine-oriented semantic carriers rather than human-readable text.

Key idea

DeepSeek-OCR maps text to a small set of vision tokens via a DeepEncoder, then decodes them back with a MoE-LLM:

f_{enc}: X_{text} \rightarrow Z_{vision}, \quad f_{dec}: Z_{vision} \rightarrow \hat{X}_{text}

The author abstracts this into a purely symbolic scheme: project a semantic embedding $v \in \mathbb{R}^d$ through a matrix $W_{proj}$ into a low-dimensional space, then quantize each dimension against a codebook of rare characters (an "AI Hanzi alphabet"). The resulting short character sequence is unreadable to humans but is hypothesized to activate similar internal representations in an LLM.

The theoretical grounding is compressive sensing: if the semantic signal is sparse in some basis, it can be recovered from few measurements. The author notes conditions such as the RIP (Restricted Isometry Property) of the projection matrix.

Four progressive stages

1. Static prototype (Java): A random Gaussian matrix compresses a d-dimensional embedding to m=16 values, each mapped to a rare character via amplitude bucketing. Demo output: a 16-character sequence with reconstruction cosine similarity ≈ 0.89. A second demo calls a Flask/sentence-transformers backend (paraphrase-multilingual-MiniLM-L12-v2) and reports similarity ≈ 0.86.

2. Learnable codebook (PyTorch): An encoder → codebook (K=128 learnable vectors) → decoder architecture trained with loss $\mathcal{L} = 1 - \cos(f_\theta^{-1}(f_\theta(v)), v)$ , using nearest-neighbor/soft quantization to select characters.

3. Entropy-constrained training + LLM verification: Adds entropy regularization and noise injection for sparse, robust assignments; then proposes verifying "activation equivalence" by feeding both the original sentence and its compressed Hanzi sequence into Qwen2.5-7B-Instruct and comparing last-layer hidden states (cosine similarity / KL divergence). A similarity of ~0.8–0.9 would suggest the model treats the sequence as semantically equivalent.

4. Multi-agent communication protocol: Two LLM agents share an encoder/decoder and codebook, exchanging short Hanzi strings (e.g., ~16 characters per message) as a token-efficient semantic channel — a text-only version of "Context Optical Compression."

Proposed research directions

Alignment of the compressed language with LLM embedding layers via contrastive/fine-tuning
Adaptive, multi-resolution encoding (variable-length sequences by semantic complexity)
Compressed storage of long conversation history in memory buffers
A "self-forgetting" mechanism mimicking DeepSeek-OCR's visual forgetting via progressive blurring/shortening
Self-organizing codebook evolution in multi-agent systems

Conclusion

The author argues that an "AI-specific Chinese" is essentially a sparse symbolic basis: its goal is not human readability but efficient model activation. DeepSeek-OCR's optical compression is reinterpreted as symbolic compressive sensing — moving from a 2D visual subspace to a 1D symbolic one — with claimed potential compression ratios of 10×–20× for LLM context. The post includes complete runnable Java and Python code for each stage (projection, Hanzi encoding, codebook training, and LLM hidden-state verification).