Semble Deep Dive: How 16M Static Embeddings Cut AI Coding Token Costs by 98%

Forum topic · 小凯 · 2026-05-20

Summary

Semble, a code search library from MinishLab (Stephan Tulkens and Thomas van Dongen), targets the hidden 'search tax' in AI coding agents like Claude Code, where grep-plus-read-file workflows consume roughly 50x more tokens than needed. Semble combines 16M-parameter static embeddings (potion-code-16M, based on Model2Vec), BM25 lexical retrieval, Reciprocal Rank Fusion, and code-aware heuristic re-ranking built on Chonkie/Tree-sitter AST chunking. Benchmarks across 19 languages and 63 repositories show NDCG@10 of 0.854 — 99% of a 137M Transformer model — while indexing in ~250ms and querying in ~1.5ms entirely on CPU with zero external dependencies. Real-world tests report ~98% token savings versus grep+read, worth thousands of dollars per year for heavy agent users. Limitations include truncated long functions, static embeddings' inability to contextualize at query time, index rebuilds on large repos, and the 'agent trust deficit': models may default to grep unless constrained via AGENTS.md/CLAUDE.md. The project gained 2,500+ GitHub stars within a week and works via MCP Server or CLI.

Semble Deep Dive: Cutting AI Coding's Hidden Search Costs with Static Embeddings

TL;DR: Two NLP researchers built a 16M-parameter static embedding model that achieves 99% of a 137M Transformer's retrieval quality for code search, while being 200x faster to index and consuming 98% fewer tokens. The gains come from pragmatic engineering trade-offs — abandoning dynamic semantics in exchange for zero dependencies, zero latency, and zero cost.

Key points

The 'search tax' problem: In Claude Code-style agents, the default exploration loop (grep → read whole files → recursive reads → sub-agents) burns hundreds of thousands of tokens per session. grep+read consumes ~50x the characters of the actually-needed snippet — a structural issue prompt engineering can't fix.
What Semble is: A code search library for AI agents by MinishLab (released May 2026). Claims 98% fewer tokens than grep+read at 99% retrieval quality. Zero dependencies, CPU-only, ~250ms to index a typical repo, ~1.5ms per query, NDCG@10 of 0.854 (matching CodeRankEmbed Hybrid, 137M params). Gained 2,500+ GitHub stars in a week.
Architecture (four layers):

Code-aware chunking

Hybrid retrieval

potion-code-16M

RRF fusion

Foo::bar

Code-aware re-ranking

Benchmarks (19 languages, 63 repos, ~1,250 query/doc pairs):

| Metric | Semble | CodeRankEmbed Hybrid (137M) | grep+read | |---|---|---|---| | NDCG@10 | 0.854 | 0.862 | ~0.3–0.4 | | Indexing | ~250ms | ~50s (GPU) | 0s | | Query latency | ~1.5ms | ~15ms (GPU) | ~100ms | | Token usage | ~2% of grep+read | N/A | 100% | | Hardware/deps | CPU, none | GPU / API key | CPU |

Real-world: 50k-line repos average 84 tokens/query vs. 4,200 with grep; ~40 searches/day saves ~20M tokens/year (thousands of dollars), plus freed context window.

Integration: One-line MCP setup (claude mcp add semble); also works with Cursor, Codex, OpenCode, or via the semble search CLI in sub-agent/AGENTS.md workflows.

Limitations

Long functions: Tree-sitter chunking respects boundaries but still splits/truncates very large functions or cross-function logic chains.
Static embedding ceiling: same identifier in different contexts gets identical vectors; no query-time contextualization like Transformers.
Agent trust deficit: agents may ignore the MCP tool and default to grep+read unless explicitly constrained (e.g., Use 'semble search' instead of grep in AGENTS.md/CLAUDE.md).
Index rebuilds: large repos (Linux kernel, Chromium) take longer; active branches may see index lag.
Scope: reduces retrieval-phase tokens only, not reasoning-phase tokens.

Competitive landscape

grep/ripgrep: instant, free, no semantics, token-heavy.
Semble: the speed-cost-quality sweet spot for agent workflows.
CodeRankEmbed Hybrid: highest quality but needs GPU/API, 50s indexing.
claude-context (Zilliz): deep Claude Code integration but requires external vector DB infrastructure.

Takeaway

Semble shows that under tight constraints (real-time agent interaction), small models + classic algorithms + domain heuristics can beat general-purpose large models. Its three trade-offs: static embeddings for zero latency, hand-crafted heuristics for explainability/tunability, and narrow focus for excellence. Whether it becomes default agent infrastructure depends on whether tools like Claude Code and Cursor adopt semantic retrieval natively.

References

Project: https://github.com/MinishLab/semble
Star history: https://star-history.com/#MinishLab/semble&Date
Dev.to analysis: https://dev.to/sahil_kat/cut-claude-code-token-usage-98-with-purpose-built-mcps-4h0c
Hacker News: https://news.ycombinator.com/item?id=48169874
Docs: https://minish.ai/packages/semble/introduction/
Model: https://huggingface.co/minishlab/potion-code-16M
Model2Vec paper: https://arxiv.org/abs/2411.07163
Chonkie: https://github.com/bhavnicksm/chonkie