Glance: The Generator Becomes Its Own Reranker for RAG in Just 0.1 GiB

Ask a RAG system: "What was Apple's total revenue in 2024?" It first retrieves 50 documents from a knowledge base, then a reranker scores them and picks the top 5, and finally a generator writes the answer based on those 5.

In this pipeline, the reranker and the generator are two different models. The reranker is usually a cross-encoder, trained specifically for scoring; the generator is an LLM, trained to write answers. Two models each occupy their own memory and run their own inference.

In late October 2026, the AI Run team published a paper on arXiv (2610.40454) called Glance. Their core idea is remarkably simple: let the generator be its own reranker. Instead of adding another model, the same model sorts candidate documents on the fly, right before generating the answer.

Result: only 0.1 GiB of extra memory, 39 minutes of fine-tuning, and performance that beats dedicated rerankers.

RAG's Two-Stage Architecture, and Its Waste

Background first. The standard RAG (Retrieval-Augmented Generation) pipeline is:

1. Retrieve: use vector similarity to pull N documents from the knowledge base (say 50). 2. Rerank: use a cross-encoder reranker to score those 50 and pick the top-K (say 5). 3. Generate: put the top-K documents into the LLM's context and let it generate the answer.

The reranker exists because vector retrieval (bi-encoder) only compares the query's and document's separate vectors, without looking at their interactions; a cross-encoder concatenates query and document and models their interactions, giving higher precision. But the cross-encoder and the LLM are two different models, each occupying its own memory.

There is obvious waste here: when the LLM generates the answer, it already runs a forward pass over the query concatenated with the top-K documents. That forward pass does essentially the same thing as the cross-encoder's scoring forward pass—looking at query-document interactions. Why run it twice?

Glance's Core Insight: Sharing the Prefill

Glance's core insight: the LLM's prefill stage is a free cross-encoder scoring pass.

Prefill is how the LLM processes the input context. When you feed the LLM the query concatenated with candidate documents, the prefill stage computes attention among all tokens. That means the query tokens and each candidate document's tokens have already interacted—exactly what a cross-encoder does.

If you can "read out" each candidate's relevance score from the LLM's hidden states after prefill, you don't need a separate reranker.

Glance's approach: at the LLM's final layer, add a "pointer head" for each candidate document—a small linear layer that takes the hidden states of that document's tokens as input and outputs a scalar score. That score is the document's relevance.

During training, the pointer heads learn to produce the correct ranking using known query-relevant document pairs. At inference, after a single prefill, the pointer heads score all candidates and the top-K enter the generation stage.

0.1 GiB: An Almost-Free Lunch

The cost of this design is astonishingly small. The pointer head is a linear layer with parameters = hidden dimension × 1. For a 9B-parameter LLM with a hidden dimension of about 4096, the pointer head has only 4096 parameters—under 0.001 MB.

With the small extra training loss and the ranking computation at inference, total overhead is roughly 0.1 GiB of additional memory. By comparison, an independent cross-encoder reranker (e.g., bge-reranker-large) needs about 1.5 GiB.

Fine-tuning is also cheap: 39 minutes on the MS MARCO dataset. That means any team with a GPU can train their own.

Performance: Beating Dedicated Rerankers

Glance was tested on multiple benchmarks for reranking performance:

• SciFact: nDCG@10 of 0.741, beating bge-reranker's 0.721.
• HotpotQA: answer F1 improved by 0.045 (vs. a baseline without reranking).
• NQ: answer F1 improved by 0.031.

Most striking is the number of candidates it can handle. A traditional cross-encoder runs one forward pass per candidate—50 candidates means 50 inferences. Because Glance shares a single prefill between reranking and generation, it can process 200 candidates at once (~43k tokens of context). In an era of increasingly long-context LLMs, this is an architectural advantage.

Why It Works: Solving the Problem at a Different Level

Glance can replace a 1.5 GiB reranker with 0.1 GiB not because its pointer head is smarter than a cross-encoder, but because it solves the problem at a different level.

The traditional view: reranking is a separate task needing a separate model. The optimization direction is making the reranker stronger—bigger models, more training data, more refined loss functions.

Glance's view: reranking is a precursor to generation, and generation's prefill already contains the information reranking needs. Instead of making the reranker stronger, let the generator do reranking "on the side."

It's like hiring a translator who first reads an English document to summarize it, then translates the summary into Chinese. The traditional approach hires two people—one to summarize, one to translate. Glance has the translator summarize while reading, because the act of reading is itself information extraction.

This idea applies far beyond RAG. Any "pre-filtering + post-generation" pipeline could let the generator do the filtering. For example:

• Code generation: pick the most relevant candidate code snippets, then generate new code from them.
• Dialogue systems: pick the most relevant turns from conversation history, then generate a reply.
• Multi-document summarization: pick the most important passages, then generate a summary.

Each scenario invites the same question: the generator has already seen all candidates during prefill—can it also score them?

A Potential Limitation

Glance's design assumes the candidate documents fit in the LLM's context window. The paper tested 200 candidates (43k tokens), requiring at least a 43k context length. For modern LLMs supporting 128k or longer, that's fine. But for models with small windows (e.g., older 4k models), Glance's advantage diminishes.

Another concern is latency. While Glance saves the standalone reranker's inference time, it puts all candidates into one prefill, which takes longer. For a "rerank to top-5, then generate" workflow, Glance could be slower than the traditional two-stage pipeline—it prefills 200 candidates while the traditional method prefills only the top 5.

But the paper's experiments show this latency difference is negligible in practice. Prefill is parallel, so a single prefill over 200 candidates is far faster than the sum of 50 separate cross-encoder inferences.

Implications for RAG Architecture

Glance leads to a prediction about RAG's architectural evolution: future RAG systems may no longer have "reranker" as a separate component.

Today's RAG is a three-stage pipeline: retrieve → rerank → generate. If Glance's approach is widely adopted, it becomes retrieve → generate (with built-in reranking)—a two-stage pipeline. Reranking is no longer a step; it's a byproduct of the generator.

This matches another trend in the LLM field: specialized components are being absorbed by general-purpose models. Retrievers absorbed into LLM embedding capability, rerankers absorbed into LLM prefill capability—next, perhaps the knowledge base itself absorbed into the LLM's parametric memory. RAG could eventually collapse into a single step: generate—the model retrieves, reranks, and generates by itself.

That's speculation. Glance currently only reranks a given candidate set; it can't retrieve from a full corpus. But its idea—having the generator handle the precursor task on the side—is a template that can be applied repeatedly.

Conclusion

Glance does something elegant: it identifies an overlooked free signal (the attention interactions in prefill) and exploits it at extremely low cost (0.1 GiB + 39 minutes of training).

This "finding a free lunch" research style deserves more appreciation than "stacking more parameters." It doesn't change the model architecture, doesn't add inference cost—just looks at the problem from a different angle, and finds the answer was there all along.

Model open-sourced: https://huggingface.co/airun-ai/GlanceRAG-MiMo-9B-Reranker

Paper: https://arxiv.org/abs/2610.40454