Beyond Relevant Documents: A Knowledge-Intensive Approach for Query-Focused Summarization Using Large Language Models

Forum topic · 小凯 · 2026-07-05

Summary

This August 2024 arXiv paper (2408.10357) by Zhang, Huang, Vakulenko, Xu, Rajapakse, and Kanoulas addresses query-focused summarization, where the goal is to produce a concise answer to a natural-language query rather than merely return a list of relevant documents. The authors argue that classical retrieve-then-summarize pipelines underuse the parametric knowledge of large language models and treat LLMs as passive rewriters. They propose a knowledge-intensive framework that couples retrieval with LLM reasoning to integrate external evidence with model-internal knowledge, enabling faithful, query-aligned summaries. Reported contributions include a unified task formulation, modular architecture designs, and experiments demonstrating improvements over retrieval-only and vanilla RAG baselines on standard benchmarks. The paper is discussed within the Response Generation chapter of a community literature list and is recommended for readers building retrieval-augmented generation, agentic search, and knowledge-grounded conversational systems.

Overview

Title: Beyond Relevant Documents: A Knowledge-Intensive Approach for Query-Focused Summarization using Large Language Models Authors: Weijia Zhang, Jia-Hong Huang, Svitlana Vakulenko, Yumo Xu, Thilina Rajapakse, Evangelos Kanoulas Source: https://arxiv.org/abs/2408.10357 (arXiv, August 2024) Category: Response Generation (literature list entry)

---

Motivation

Classical query-focused summarization (QFS) systems operate in a two-stage retrieve-then-summarize fashion: a retriever returns relevant passages, and a generator rewrites them into an answer. The authors observe that such pipelines underuse the parametric knowledge stored inside large language models (LLMs) and treat LLMs as passive rewriters rather than reasoning agents. They argue that producing a high-quality answer requires integrating external retrieved evidence with the LLM's internal knowledge, especially when documents are incomplete, redundant, or only partially relevant.

Core Idea

The paper reframes QFS as a knowledge-intensive task: rather than competing with retrieval, the LLM acts as a reasoning module that decides how to combine retrieved passages with its own world knowledge to produce a query-aligned, faithful summary.

Main Contributions

A unified formulation of knowledge-intensive QFS that goes beyond surface-level relevant-document selection.
A modular framework for combining retrievers with LLM reasoning, including prompt construction, evidence filtering, and answer synthesis.
Empirical comparison against retrieval-only and vanilla retrieval-augmented generation (RAG) baselines.
Discussion of failure cases such as hallucination, citation drift, and partial coverage.
Identification of open problems: evaluation beyond ROUGE, attribution accuracy, latency vs. quality trade-offs, and multilingual QFS.

Method (High-Level)

1. Query/document encoding into dense or hybrid sparse-dense representations. 2. Evidence retrieval producing a candidate set of passages. 3. Evidence filtering / re-ranking using either a cross-encoder or the LLM itself. 4. Knowledge integration inside the LLM via structured prompts that explicitly request integration of external and parametric knowledge. 5. Answer generation with optional inline citations and refusal behavior when evidence is insufficient.

Evaluation

Standard QFS benchmarks are used (e.g., answer-focused summarization datasets). Metrics typically include:

ROUGE / BERTScore for surface and semantic fidelity.
Citation precision and recall for attribution quality.
Human evaluation for faithfulness, relevance, and comprehensiveness.

Quantitative numbers should be verified against the PDF, but the narrative emphasizes gains over document-centric baselines, particularly when retrieved evidence is noisy.

Key Takeaways

Mere relevance is not sufficient for query-focused summarization; the LLM must reason over evidence.
Treating LLMs as active reasoners rather than passive rewriters improves answer quality.
The framework is modular: existing retrievers, re-rankers, and generators can be plugged in.
Attribution and refusal are first-class concerns, not afterthoughts.

Limitations and Future Work

Cost of LLM-based re-ranking and filtering can be high at scale.
Evaluation still leans on string-overlap metrics.
Cross-lingual and multi-hop QFS scenarios are underexplored.
Integration with agentic tool use and structured knowledge bases is left to future work.

Practical Engineering Checklist

| Concern | Question | Recommendation | |--------|----------|----------------| | Latency | p99 budget? | Cache top queries, early-stop re-ranking, async evidence filtering | | Attribution | Can outputs cite sources? | Enforce inline citation spans; post-hoc verifier | | Faithfulness | Hallucination risk? | Refuse when evidence is weak; constrain to retrieved facts | | Cost | Tokens per query? | Use small models for filtering, large only for synthesis | | Index freshness | Embedding version drift? | Version embeddings, support rollback |

Related Entries in the List

Neural Headline Generation: A Comprehensive Survey (Mar 2025, Neurocomputing)
Cite Before You Speak: Enhancing Context-Response Grounding in E-commerce (2503.04830)
Improving Generative Ad Text on Facebook using Reinforcement Learning (2507.21983)

Glossary

| Term | Meaning | |------|---------| | QFS | Query-Focused Summarization | | RAG | Retrieval-Augmented Generation | | LLM | Large Language Model | | Attribution | Linking generated claims to source evidence | | Refusal | LLM declining to answer when evidence is insufficient |

Actionable Recommendations

Researchers: Reproduce against QFS benchmarks; report statistical significance and inference cost.
Engineers: Extract the retrieval, filtering, and integration modules as independent services; measure end-to-end latency.
Product managers: Track user-perceived faithfulness (citations, refusals) alongside offline ROUGE.

Reference

arXiv:2408.10357 — Zhang et al., *Beyond Relevant Documents: A Knowledge-Intensive Approach for Query-Focused Summarization using Large Language Models*, August 2024. URL: https://arxiv.org/abs/2408.10357