SIRA Explained: How Meta Combines a Frozen LLM with BM25 for Single-Shot, Training-Free Retrieval

Forum topic · 小凯 · 2026-05-28

Summary

SIRA (Super-Intelligent Retrieval Agent), a retrieval framework from Meta Superintelligence Labs and Rice University, replaces multi-turn agentic search with a single, corpus-discriminative BM25 query compiled by a frozen LLM. The method performs bidirectional vocabulary enrichment: offline, an LLM proposes query terms for each document (filtered by document-frequency statistics) to enrich the inverted index; online, the LLM generates a predicted 'answer sketch' of discriminative terms (must exist in the enriched index, DF > 0), which are combined with the original query in one weighted BM25 call. Without dense embeddings, neural rankers, RL training, or answer generation, SIRA achieves an average Recall@10 of 0.691 across ten BEIR benchmarks, beating supervised dense retrievers like E5 (0.648), SPLADE (0.625), and the RL-trained multi-turn system Search-R1 (0.616), with up to +36.4% on SciDocs. In QA tasks, its top-10 answer coverage of 84.7% on NQ and 77.6% on HotpotQA exceeds six RL-trained end-to-end systems. Limitations include per-query LLM cost, an unspecified DF threshold, dependence on the LLM's parametric knowledge, text-only scope, and static-corpus assumptions.

SIRA: Meta's Single-Shot LLM + BM25 Retrieval Agent

Key points

Problem: Dense retrieval suffers from an information bottleneck — fixed-dimensional embeddings lose discriminative details, and semantically related documents become indistinguishable. Multi-turn agentic search (ReAct, IRCoT, Search-R1) compensates but explodes latency, inflates context, and suffers from "lost in the middle."
Core idea: SIRA redefines "superintelligence" in retrieval as compressing multi-turn exploratory search into a single *corpus-discriminative* retrieval action. The LLM never reads returned documents; it consults index statistics instead.
Architecture (bidirectional vocabulary enrichment):
*Corpus side (offline)*: An LLM reads each document and proposes terms a searcher might use that don't appear in the text (synonyms, abbreviations, jargon). A document-frequency (DF) filter prunes terms too common in the corpus (threshold τ·|C|); survivors are injected into the BM25 inverted index as n-grams.
*Query side (online)*: The LLM generates an "expected answer sketch" — discriminative context terms the relevant document likely contains, explicitly forbidden from guessing the answer itself. Each term must have DF > 0 in the enriched index or it is discarded as hallucination.
*Final retrieval*: one weighted BM25 call: score(d) = BM25(q_orig, d) + w · BM25(q_exp, d).
Why BM25 wins: IDF naturally rewards rare, domain-specific terms that dense embeddings dilute; and BM25 is transparent and controllable — the LLM chooses the knobs, corpus statistics validate them.

Benchmark results

Pure retrieval on ten BEIR benchmarks, average Recall@10:

| Method | Recall@10 | |:--|--:| | SIRA | 0.691 | | E5 (supervised dense) | 0.648 | | SPLADE (learned sparse) | 0.625 | | Search-R1 (RL multi-turn + E5) | 0.616 | | BM25 baseline | 0.530 | | GrepRAG (LLM tools) | 0.280 | | ShellAgent (LLM multi-turn) | 0.253 |

SIRA ranked first on 8 of 10 datasets. Largest gains vs E5: SciDocs +36.4% (0.2676 vs 0.1962), CQADupStack +22.6%, ArguAna +14.2%. Mean NDCG@10 of 0.572 also led all baselines.

QA coverage (top-10 contains correct answer): NQ — SIRA 84.7% vs HiPRAG 71.2%, Search-R1 48.0%; HotpotQA — SIRA 77.6% vs A²Search 69.0%, Search-R1 49.5%. Notably, the baselines are end-to-end RL-trained QA systems with readers, while SIRA is a training-free, general-purpose retriever.

Same-LLM comparison: GrepRAG and ShellAgent use the identical backbone (Qwen3.6-35B-A3B-FP8, frozen, 3B active) yet trail SIRA by 41.0 and 43.8 absolute points — evidence the gap lies in retrieval interface design, not LLM capability.

Limitations

Per-query LLM cost: still cheaper than multi-turn agents, but can't run fully offline and has an inference latency floor.
Underspecified hyperparameter: the DF threshold τ is described only qualitatively; tuning is corpus-dependent.
Garbage in, garbage out: performance depends on the LLM's parametric knowledge; in very obscure domains proposals may be poor, though the DF filter acts as a safety net.
Lexical-only: no multimodal retrieval; limited semantic structure for code search.
Static-corpus assumption: heavily updated corpora require re-running the offline enrichment, weakening the amortization claim.

Why it matters

SIRA reframes the LLM's role: not the retrieval system's *user* (issuing and iterating on queries), but its *controller* — predicting the evidence profile, validating discriminative power against corpus statistics, and compiling a single weighted query. Retrieval shifts from a trial-and-error process to a decision process.

Paper information