How Zilliz Built a Semantic Highlight Model to Cut Token Costs in RAG
Source: How We Built a Semantic Highlight Model To Save Token Cost for RAG — Hugging Face Blog, published by Zilliz, January 2026.
> Note: This article is an editorial digest of an industrial engineering blog. Details and quantitative results should be verified against the original post.
Background and Motivation
In retrieval-augmented generation (RAG) pipelines, documents retrieved from a vector database are commonly passed in their entirety to the large language model (LLM). This design is simple but wasteful: retrieved chunks often contain substantial content that is irrelevant to the user's query, inflating prompt token counts, increasing inference cost and latency, and sometimes diluting answer quality with noise.
Zilliz's engineering team addressed this by adding a semantic highlighting stage between retrieval and generation. The goal is to let a small, purpose-built model identify which sentences or spans inside each retrieved chunk actually matter for answering the query, and pass only those highlights to the generator.
The Approach
At a high level, the pipeline works as follows:
1. Retrieve — as usual, relevant document chunks are fetched from the vector store (e.g., Milvus) via dense or hybrid search. 2. Highlight — instead of forwarding the raw chunks, a lightweight semantic highlight model scores or selects the query-relevant sentences/spans within each chunk. 3. Generate — only the highlighted, condensed context is inserted into the prompt, reducing token usage while preserving the information needed for a correct answer.
Why It Matters
- Cost: fewer prompt tokens directly translates to lower API or GPU inference costs, which matters at production scale.
- Latency: shorter prompts mean faster prefill and time-to-first-token.
- Quality: removing irrelevant context can reduce distraction and hallucination risk, though aggressive highlighting risks dropping necessary evidence.
- Recall of relevant evidence vs. compression ratio — highlighting too aggressively can omit key facts.
- Extra stage latency vs. saved generation cost — the highlight model must be much smaller and faster than the generator to pay off.
- Evaluation — token savings are easy to measure, but downstream answer quality requires careful end-to-end evaluation.
- A Survey of Graph Retrieval-Augmented Generation for Customized LLMs (arXiv:2501.13958)
- A Survey on Retrieval-Augmented Text Generation for LLMs (arXiv:2404.10981)
- Agentic RAG: A Survey (arXiv:2501.09136)
- RAFT: Adapting Language Model to Domain Specific RAG
Trade-offs and Open Questions
As with any context-compression technique, the design involves balancing: