FollowTable: A Benchmark for Instruction-Following Table Retrieval for LLM Agents

Forum topic · 小凯 · 2026-05-04

Summary

FollowTable is a benchmark introduced for instruction-following table retrieval, targeting a key weakness of traditional table retrieval in the LLM Agent era. Conventional retrieval systems evaluate only topical relevance between a query and a table, often returning tables that are semantically related but practically unusable—for example, missing required columns, having mismatched data types, or being unable to support operations like filtering, sorting, or computing growth rates. FollowTable reframes the task: retrieval quality should be judged by whether the retrieved table can actually execute the user's natural-language instruction. The benchmark pairs natural-language instructions (containing both topical content and operational requirements) with tables and evaluates systems on executability, including column-name matching, data-type correctness, and feasibility of requested operations. The paper is authored by Rihui Jin, Yuchen Lu, Ting Zhang, and Jun Wang (arXiv: 2605.00400). This shift represents a move from "information retrieval" toward "actionable retrieval," prioritizing usability over mere relevance, and aims to improve downstream task success rates for LLM Agents working with structured data.

FollowTable: When Table Retrieval Meets Instruction Following

> Paper: FollowTable: A Benchmark for Instruction-Following Table Retrieval > Authors: Rihui Jin, Yuchen Lu, Ting Zhang, Jun Wang > arXiv: 2605.00400 | 2026-04-29

The Problem: Finding a Table but Failing to Use It

Consider asking an LLM Agent:

> "Find a table of tech companies with Q1 2024 revenue over 100 million, sorted by growth rate."

Traditional table retrieval:

Matches keywords like "2024", "Q1", "revenue"
Ignores whether the table actually contains usable data
Ignores whether growth rates can be computed
Ignores whether column names match

The result may be a table titled "2024 Tech Company Headcounts"—topically relevant, but completely unable to satisfy the instruction. The Agent receives a table it cannot act on.

Core issue: traditional table retrieval only checks "topical relevance," not "whether the instruction can be satisfied."

New Requirements in the LLM Agent Era

Traditional table retrieval works like document retrieval—matching topics only. Agents need more:

Instruction following: "find tables that can compute growth rates", "find tables with 2024 data", "find tables that can be sorted by a column"—retrieval must consider operability
Structural constraints: column names must match, data types must be correct, and requested operations must be feasible
Dual matching: semantic (what it is about) plus structural (what it can do)

The FollowTable Benchmark

FollowTable's core idea:

> Table retrieval should be evaluated by "whether the instruction can be satisfied," not just "whether the topic is relevant."

Key components:

1. Instruction-driven retrieval: Inputs are natural-language instructions containing both topic and operational requirements (filtering, sorting, computation) 2. Executability assessment: Can the retrieved table execute the instruction? Do column names match? Are data types correct? Are the operations feasible? 3. Structured semantic matching: Matches both content and structure—topic plus operability 4. Benchmark dataset: A large collection of instruction–table pairs to evaluate the instruction-following ability of retrieval systems

An analogy: traditional retrieval hands you a book about cooking; instruction-following retrieval hands you a book containing a chocolate cake recipe with oven temperature and timing—the only one that actually helps you bake a cake.

Why Instruction-Following Retrieval Is Better

Traditional retrieval fails when a table is topically relevant but unusable—missing required columns, mismatched data types, or structure ignored entirely.

Instruction-following retrieval offers:

Practicality: retrieved tables are genuinely usable; results are not just "relevant" but "useful"
Structured understanding: accounts for column names, types, and relationships against instruction requirements
Agent-friendliness: designed around the operational needs of LLM Agents, raising downstream success rates

Takeaway: Useful Information Is Not Just Relevant—It Is Actionable

Retrieving a "relevant" table is not the same as retrieving a "useful" one. FollowTable's insight is that retrieval quality ultimately depends on whether the information can be used to accomplish a goal—a leap from "information retrieval" to "actionable retrieval." Information that cannot be used is effectively noise; retrieval systems should serve action.

If you are building retrieval systems or LLM Agents, ask:

1. Does my retrieval system focus only on relevance, ignoring operability? 2. Are structural constraints factored into retrieval? 3. Can retrieval results directly support downstream tasks? 4. Am I evaluating "usable," not just "relevant"?

In the era of LLM Agents, the goal of retrieval upgrades from "finding relevant information" to "finding information that can execute the instruction." When table retrieval learns to follow instructions, Agents can truly become masters of their data.