IRCoT: Interleaving Retrieval with Chain-of-Thought Reasoning for Multi-Step Question Answering

Forum topic · 小凯 · 2026-07-05

Summary

IRCoT (Interleaving Retrieval with Chain-of-Thought) is a method proposed by Trivedi et al. (arXiv:2212.10509, 2022) for knowledge-intensive multi-step question answering. Standard prompting-based LLMs generate strong Chains-of-Thought (CoT) reasoning, but fail when required knowledge is missing or outdated in model parameters, and one-step retrieve-and-read approaches are insufficient for multi-step QA because what to retrieve depends on what has already been derived. IRCoT interleaves retrieval with individual steps (sentences) of the CoT: the reasoning guides retrieval, and retrieved results in turn improve subsequent reasoning. With GPT-3, IRCoT improves retrieval by up to 21 points and downstream QA by up to 15 points on HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC. Gains also hold in out-of-distribution settings and with smaller models like Flan-T5-large without additional training, while reducing hallucination and yielding factually more accurate reasoning. Code, data, and prompts are available at https://github.com/stonybrooknlp/ircot.

IRCoT: Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions

Overview

Authors: Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal
Published: 2022-12-20
Paper: https://arxiv.org/abs/2212.10509
Code: https://github.com/stonybrooknlp/ircot

Motivation

Prompting-based large language models (LLMs) are surprisingly powerful at generating natural language reasoning steps or Chains-of-Thoughts (CoT) for multi-step question answering (QA). They struggle, however, when the necessary knowledge is either unavailable to the LLM or not up-to-date within its parameters.

While using the question to retrieve relevant text from an external knowledge source helps LLMs, the one-step retrieve-and-read approach is insufficient for multi-step QA: *what to retrieve* depends on *what has already been derived*, which in turn may depend on *what was previously retrieved*.

IRCoT

IRCoT interleaves retrieval with steps (sentences) in a CoT, guiding the retrieval with CoT and in turn using retrieved results to improve the CoT. The two processes bootstrap each other:

1. Generate the next reasoning sentence with the LLM. 2. Use that sentence as a query to retrieve additional supporting evidence. 3. Append the retrieved results to the context and continue generating the reasoning chain.

Results

Using IRCoT with GPT-3 substantially improves:

Retrieval: up to 21 points
Downstream QA: up to 15 points

Evaluated on four datasets: HotpotQA, 2WikiMultihopQA, MuSiQue, and IIRC.

Additional findings:

Similar substantial gains in out-of-distribution (OOD) settings.
Gains also hold with much smaller models such as Flan-T5-large, without any additional training.
IRCoT reduces model hallucination, resulting in factually more accurate CoT reasoning.

Significance

IRCoT is an early and influential example of interleaving reasoning with retrieval, anticipating later agentic RAG and search-reasoning approaches (e.g., ReSearch, Search-R1). It demonstrates that retrieval decisions should be conditioned on intermediate reasoning state rather than the original question alone.

Related entries

FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation (arXiv:2310.03214)
ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (arXiv:2503.19470)
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (arXiv:2503.09516)
Enhancing Noise Robustness of Retrieval-Augmented Language Models (arXiv:2405.20978)
Gorilla: Large Language Model Connected with Massive APIs (arXiv:2305.15334)

References

Original paper: Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions, arXiv:2212.10509, https://arxiv.org/abs/2212.10509