FollowTable: When Table Retrieval Meets Instruction Following — A New Data Challenge for LLM Agents

Forum topic · 小凯 · 2026-05-04

Summary

FollowTable is a benchmark for instruction-following table retrieval, proposed by Rihui Jin, Yuchen Lu, Ting Zhang, and Jun Wang (arXiv 2605.00400, 2026-04-29). The paper argues that traditional table retrieval only measures topical relevance between a query and a table, ignoring whether the table can actually satisfy the operational requirements of an instruction—such as filtering, sorting, or computing growth rates. As a result, LLM Agents often retrieve tables that are semantically related but unusable: missing required columns, wrong data types, or structures that cannot support the requested operations. FollowTable reframes retrieval quality as a dual semantic-plus-structural matching problem: tables must match not only what a query is about, but what it needs to do. The benchmark provides a large set of instruction-table pairs with executability assessment—checking column-name alignment, data-type correctness, and operation feasibility—to evaluate how well retrieval systems support downstream agent tasks. The core insight: in the LLM Agent era, retrieval should return information that is actionable, not merely relevant, shifting from 'information retrieval' toward 'retrieval for action.'

> Paper: FollowTable: A Benchmark for Instruction-Following Table Retrieval > Authors: Rihui Jin, Yuchen Lu, Ting Zhang, Jun Wang > arXiv: 2605.00400 | 2026-04-29

1. The Embarrassment of "Finding a Table but Not Being Able to Use It"

Imagine asking an LLM Agent:

You: "Find a table of tech companies with Q1 2024 revenue over 100 million, sorted by growth rate"

Traditional table retrieval:

Finds tables containing "2024", "Q1", "revenue"
Doesn't check whether the table actually contains such data
Doesn't check whether it can be sorted by growth rate
Doesn't check whether column names match

Result:

Retrieves a table of "2024 tech company headcounts"
Semantically relevant, but completely fails the requirement
The Agent gets a table it cannot use to complete the task

The problem: traditional table retrieval only looks at topical relevance, not whether the table can satisfy the instruction.

2. New Requirements for Table Retrieval in the LLM Agent Era

Traditional table retrieval:

Similar to document retrieval
Checks topical relevance between query and table
Ignores whether the table can support specific operations

New needs in the LLM Agent era:

Instruction following:

"Find a table that supports computing growth rates"
"Find a table containing 2024 data"
"Find a table that can be sorted by column"
Retrieval must account for operability

Structural constraints:

Column names must match
Data types must be correct
The table must support the required operations

Dual semantic + structural matching:

Not just "what it is about"
But also "what it can do"

3. The FollowTable Benchmark

The paper introduces FollowTable with a core innovation:

Core idea: > Table retrieval should be evaluated on whether it can satisfy instructions, not merely on topical relevance.

Technical approach:

1. Instruction-driven retrieval

Input: natural language instructions
Containing not only topics
But also operational requirements
E.g., filtering, sorting, computation

2. Executability assessment

Can the retrieved table execute the instruction?
Do column names match?
Are data types correct?
Is the operation feasible?

3. Structured semantic matching

Matching not only content
But also structure
Topic + operability

4. Benchmark dataset

A large collection of instruction–table pairs
Evaluating retrieval systems' instruction-following ability
Driving research progress

An analogy:

Traditional retrieval: hands you a book about cooking
Instruction-following retrieval: hands you a book containing a "chocolate cake recipe" with "oven temperature and time"
Only the latter actually helps you bake the cake

4. Why Instruction-Following Retrieval Is Better

Problems with traditional retrieval:

Topically relevant but unusable:

Retrieves a "relevant" table
But it lacks required columns
Or data types don't match
The Agent cannot use it

Ignores structure:

Only looks at table content
Not table structure
Column names, types, relationships

Advantages of instruction-following retrieval:

Practicality:

Retrieved tables are genuinely usable
The Agent can complete its task
"Useful" rather than just "relevant"

Structured understanding:

Understands table structure
Column names, types, relationships
Matches instruction requirements

Agent-friendly:

Designed for LLM Agents
Accounts for Agents' operational needs
Improves Agent success rates

5. A Feynman-Style Judgment: Useful Information Is Not Only Relevant, but Operable

Feynman said:

> "Knowing the name of something and truly understanding something are completely different."

In information retrieval:

> "Retrieving a 'relevant' table does not equal retrieving a 'useful' table. FollowTable's insight: true retrieval quality depends on whether the information can be used to accomplish a goal. This is a leap from 'information retrieval' to 'retrieval for action.'"

This reflects pragmatist philosophy:

Knowledge's value lies in its use
Unusable information ≈ noise
Retrieval systems should serve action

6. Takeaways

If you're building a retrieval system or an LLM Agent, ask yourself:

1. "Does my retrieval system focus only on relevance while ignoring operability?" 2. "Are structural constraints incorporated into retrieval?" 3. "Can retrieval results directly support downstream tasks?" 4. "Am I evaluating 'usability' and not just 'relevance'?"

FollowTable reminds us: in the era of LLM Agents, the goal of retrieval upgrades from "finding relevant information" to "finding information that can execute the instruction."

When table retrieval learns to "follow instructions," Agents can truly become masters of their data. In the ocean of structured data, the best retrieval is not the most relevant one, but the one that best helps complete the task.

In the world of information, usability is closer to truth than relevance.