Building an AI-Specific Language from DeepSeek-OCR's Visual Compression Idea

Forum topic · ✨步子哥 · 2025-10-23

Summary

This forum post proposes an AI-specific compressed language inspired by DeepSeek-OCR's visual context compression. The author suggests replacing image tokens with sequences of rare Chinese characters: semantic embeddings (e.g., 768-d vectors from a multilingual SentenceTransformer) are projected to a low-dimensional space via a linear matrix grounded in compressive sensing theory, quantized through a learnable codebook of rare characters, and decoded back to approximate the original embedding. The post progresses through four stages: (1) a static random-projection prototype with a Java demo showing ~0.89 cosine similarity; (2) a trainable encoder-codebook-decoder pipeline in PyTorch with cosine reconstruction loss; (3) entropy regularization and noise augmentation for sparse, robust codebooks, plus a proposed verification method comparing hidden-state activations of original text vs. compressed character sequences in Qwen2.5-7B; and (4) an inter-agent communication protocol where LLMs exchange compressed character strings as a low-token-cost semantic channel. Claimed compression ratios are 10x-20x. The post includes runnable Java and Python code (Flask embedding server, codebook training, verification scripts) and discusses future directions such as multi-resolution encoding, memory decay via 'visual forgetting', and self-organizing codebooks in multi-agent systems. It is a speculative conceptual proposal, not a peer-reviewed result.

Building an AI-Specific Language from DeepSeek-OCR's Visual Compression Idea

This post (translated/summarized from Chinese) explores constructing an AI-specific compressed language: transferring DeepSeek-OCR's "visual compression representation" idea into a purely textual compression space, using rare Chinese character sequences as dense intermediate representations of LLM activation features — readable by machines, not humans.

Core Idea

DeepSeek-OCR maps text to visual tokens via DeepEncoder, then decodes back to text with an MoE-LLM:

f_{enc} : X_{text} \rightarrow Z_{vision}, \quad f_{dec} : Z_{vision} \rightarrow \hat{X}_{text}

The author abstracts this into a symbolic analog: project a semantic vector into a "Hanzi embedding space":

z = W_{proj} v + b

with a dedicated character codebook acting as a sparse basis, motivated by compressive sensing: if the semantic signal is sparse in some basis, few measurements suffice to recover it. Conditions cited: the projection matrix should satisfy the RIP property, and the LLM must be strong enough to invert the representation.

Key points

Stage 1 — Static prototype: A Gaussian random projection matrix maps 128-d (or 768-d) semantic vectors to 16 compressed dimensions, each bucketed to a rare character. A Java demo (pure simulation) reports cosine similarity ≈ 0.89 between original and reconstructed vectors.
Stage 2 — Learnable codebook: A PyTorch encoder → codebook (K=128 characters) → decoder model is trained with loss 1 − cos(decode(encode(v)), v) using paraphrase-multilingual-MiniLM-L12-v2 embeddings. Nearest-neighbor (argmin over cdist) quantization selects characters.
Stage 3 — Entropy-constrained training + verification: Adds softmax soft-quantization, an entropy penalty (encouraging sparse, confident codebook assignments), and Gaussian noise for robustness. Verification proposes feeding both original text and compressed character strings into Qwen2.5-7B-Instruct and comparing last-layer hidden states via cosine similarity and KL divergence of activation spectra; similarity of 0.8–0.9 would suggest activation equivalence.
Stage 4 — Inter-agent protocol: Agents share an encoder/codebook/decoder; one LLM compresses its output into a short character string (e.g., 16–32 characters), transmits it, and the receiving LLM reconstructs approximate semantics — a text-only version of "context optical compression" with claimed 10x–20x compression.

Mapping to DeepSeek-OCR

| OCR module | Proposed analog | |---|---| | DeepEncoder (visual compression) | Hanzi compression encoder (symbolic) | | MoE decoder | LLM backbone | | Vision tokens | AI-internal character tokens |

Code artifacts included

ChineseCompressionDemo.java — random projection + character encoding simulation
AICHanziCompressor.java + embed_server.py (Flask + SentenceTransformer) — end-to-end compress/encode/reconstruct
train_codebook.py — learnable codebook training (PyTorch)
train_entropy_codebook.py — entropy + noise regularized training
verify_llm_equivalence.py — hidden-state comparison in Qwen2.5-7B-Instruct

Proposed future directions

Semantic-aware co-training so LLMs natively understand compressed characters
Multi-resolution / adaptive-length encoding based on semantic complexity
Injecting compressed sequences at the embedding layer and probing attention patterns (KL divergence)
Memory decay via progressive "blurring" of character sequences, analogous to DeepSeek-OCR's visual forgetting
Multi-agent systems where AI-human-unreadable character language evolves self-organized codebooks

Caveats

This is a speculative, conceptual proposal from a forum author. The reported similarity numbers come from toy simulations and a simplistic pseudo-inverse decoder; the claimed LLM activation equivalence is a proposed experiment, not a demonstrated result. Whether real LLMs assign meaningful semantics to such synthetic character sequences remains unverified.

Conclusion

The author's thesis: "AI-specific Chinese" is essentially a symbolic sparse basis — its goal is not human readability but efficient model activation. DeepSeek-OCR introduced "visual algebra" to text; this proposal builds a "symbolic algebra compression" mechanism inside language itself, from 2D optical subspace to a 1D symbolic space.

Building an AI-Specific Language from DeepSeek-OCR's Visual Compression Idea

Summary

Building an AI-Specific Language from DeepSeek-OCR's Visual Compression Idea

Core Idea

Key points

Mapping to DeepSeek-OCR

Code artifacts included

Proposed future directions

Caveats

Conclusion

Tags