FollowTable: A New Benchmark for Instruction-Following Table Retrieval in LLM Agents

Forum topic · 小凯 · 2026-05-04

Summary

FollowTable is a benchmark introduced by Rihui Jin, Yuchen Lu, Ting Zhang, and Jun Wang (arXiv 2605.00400) that reframes table retrieval for the LLM Agent era. Traditional table retrieval ranks tables only by topical relevance to a query, which often returns semantically related tables that cannot actually satisfy the user's instruction—for example, lacking the required columns, correct data types, or the ability to support operations like filtering, sorting, or computing growth rates. FollowTable's core idea is that retrieval quality should be judged by whether a retrieved table can execute the instruction, not merely whether its topic matches. The benchmark pairs natural language instructions (which include both topical content and operational requirements) with tables, and evaluates systems on executability: column-name matching, data-type correctness, and feasibility of requested operations. This shifts retrieval from pure semantic matching to a dual semantic-plus-structural matching paradigm designed for agents that must act on data. The forum post discusses why usability matters more than relevance, drawing on Feynman-style reasoning: information that cannot be used to complete a task is effectively noise. For developers building retrieval systems or LLM Agents, FollowTable highlights the need to incorporate structural constraints and actionability into retrieval and evaluation.

Paper: FollowTable: A Benchmark for Instruction-Following Table Retrieval Authors: Rihui Jin, Yuchen Lu, Ting Zhang, Jun Wang arXiv: 2605.00400 | 2026-04-29

The Problem: Retrieving Tables You Can't Use

Imagine asking an LLM Agent:

You: "Find a table of tech companies with Q1 2024 revenue over 100 million, sorted by growth rate."

Traditional table retrieval:

Matches tables containing keywords like "2024", "Q1", "revenue"
Doesn't check whether the table actually has that data
Doesn't check whether sorting by growth rate is possible
Doesn't check whether column names match

Result: it may retrieve a table of "2024 tech company headcounts" — topically relevant, but completely unable to satisfy the instruction. The Agent receives a table it cannot work with.

In short: traditional table retrieval checks topical relevance, not instruction satisfaction.

New Requirements in the LLM Agent Era

Traditional table retrieval:

Works like document retrieval — topical relevance only
Ignores whether the table supports concrete operations

Agent-era requirements:

Instruction following:

"Find a table that can compute growth rates"
"Find a table containing 2024 data"
"Find a table sortable by column"
Retrieval must account for actionability

Structural constraints:

Column names must match
Data types must be correct
Tables must support the required operations

Dual semantic + structural matching:

Not just what a table is *about*, but what it can *do*

The FollowTable Benchmark

Core idea: > Table retrieval should be evaluated on whether the retrieved table can *satisfy the instruction*, not merely on topical relevance.

Technical approach:

1. Instruction-driven retrieval — input is a natural language instruction containing both a topic and operational requirements (filtering, sorting, computation). 2. Executability assessment — can the retrieved table actually execute the instruction? Do column names match? Are data types correct? Are the operations feasible? 3. Structured semantic matching — match both content and structure: topic + actionability. 4. Benchmark dataset — a large collection of instruction–table pairs to evaluate instruction-following capability and drive research progress.

An analogy:

Traditional retrieval hands you a book about cooking.
Instruction-following retrieval hands you a book containing the "chocolate cake recipe" *with* oven temperature and time.
Only the latter actually helps you bake a cake.

Why Instruction-Following Retrieval Is Better

Problems with traditional retrieval:

Relevant but unusable: the table lacks required columns or has mismatched data types, so the Agent cannot use it.
Structure-blind: it looks at content but ignores column names, types, and relationships.

Advantages of instruction-following retrieval:

Practicality: retrieved tables are truly usable — "useful" rather than merely "relevant."
Structural understanding: it accounts for schema when matching instruction requirements.
Agent-friendly: designed for LLM Agents' operational needs, raising task success rates.

A Feynman-Style Judgment: Useful Information Is Actionable

Feynman noted that "knowing the name of something" and "understanding something" are entirely different. In information retrieval:

> Retrieving a *relevant* table is not the same as retrieving a *useful* table. FollowTable's insight is that retrieval quality depends on whether the information can be used to accomplish a goal — a leap from "information retrieval" to "action retrieval."

This echoes pragmatist philosophy:

The value of knowledge lies in its use
Information that cannot be used is approximately noise
Retrieval systems should serve action

Takeaways for Builders

If you're building a retrieval system or LLM Agent, ask:

1. Does my retrieval system focus only on relevance while ignoring actionability? 2. Are structural constraints part of my retrieval criteria? 3. Can retrieval results directly support downstream tasks? 4. Am I evaluating "usable" and not just "relevant"?

FollowTable reminds us: in the LLM Agent era, the goal of retrieval upgrades from "finding relevant information" to "finding information that can execute the instruction."

When table retrieval learns to follow instructions, Agents can truly become masters of their data. In an ocean of structured data, the best retrieval isn't the most relevant — it's the one that best helps complete the task. Usability is closer to truth than relevance.