RUST-BENCH: New Benchmark Exposes How Badly LLMs Fail at Real-World Table Reasoning

Forum topic · QianXun · 2025-11-08

Summary

A November 2025 arXiv paper (arXiv:2511.04491) by researchers from Virginia Tech, IIT Delhi, and Arizona State University introduces RUST-BENCH, a benchmark of 7,966 human-verified QA pairs built from 2,031 real-world semi-structured tables spanning NBA sports records and NSF science funding data. Unlike clean benchmarks such as WikiTQ, these tables average 45.1 rows and 23,040 tokens, mix structured fields with long free-text summaries, and require multi-hop, temporal, and domain-specific reasoning. Results are sobering: the best model, Qwen-QwQ, reaches only about 54-56% LLM-score, while GPT-4o-mini scores 36-40% zero-shot. Chain-of-thought prompting yields diminishing or even negative returns, and Program-of-Thought code generation performs worst across all models, showing that neither pure symbolic nor pure semantic approaches handle heterogeneous tables well. Error analysis of 100 Gemini-2.0-Flash failures identifies four systematic patterns: extraction errors (20%), logical inconsistency (31%), interpretation errors (22%), and answer-alignment errors (27%). The paper argues future systems need query-aware retrieval, dynamic summarization, symbolic-neural hybrid architectures, and domain adaptation to reason reliably over real-world tabular data.

Key points

RUST-BENCH (arXiv:2511.04491, November 2025), by Nikhil Abhyankar et al. from Virginia Tech, IIT Delhi, and Arizona State University, is a new benchmark measuring LLM reasoning over unstructured text embedded within structured tables — a setting that closely mirrors real-world data but is largely absent from benchmarks like WikiTQ and TabFact.

The four challenges it targets

Scale: Tables average 45.1 rows and 23,040 tokens (~20x WikiTQ); NSF funding tables reach 31,948 tokens. Accuracy of GPT-4o-mini and Gemini-2.0-Flash declines monotonically as table length grows past 35K-50K tokens.
Multi-hop reasoning: 26.18% of questions require chaining structured lookups, free-text filtering, and temporal/arithmetic computation (e.g., combining a "city" column with free-text summaries to compute day differences).
Heterogeneity: Tables mix structured columns (28 in NSF data) with long free-text summaries (~400-477 tokens). A controlled experiment shows symbolic methods (Program-of-Thought) win on fully structured tables, textual methods (Chain-of-Thought) win on unstructured ones — but both fail on semi-structured tables.
Domain specificity: Field-specific vocabulary (grant types, sports statistics) and implicit domain logic determine answerability, causing extraction failures when models rely on literal keyword matching.

Benchmark construction

A hybrid symbolic-semantic pipeline: 75 SQL templates instantiated and verbalized by LLMs, plus single-row and multi-row semantic generation strategies focused on deep reasoning.
8 CS graduate students verified clarity, correctness, and reasoning complexity, filtering 6-31% of candidates; 7,966 QA pairs survived (sports and NSF science subsets).

Results

Best model: Qwen-QwQ at 54.1% (science) and 55.7% (sports) LLM-score; Mistral-Small-3.2 leads on Exact Match (52.6%).
GPT-4o-mini: 36.6%/39.8% zero-shot; 44.4%/42.1% with CoT.
CoT paradox: CoT helps smaller models (Llama-3.3-70B: 38.8% → 44.2% on science) but can hurt stronger ones (GPT-4o-mini sports: Few-Shot 46.7% vs CoT 42.1%).
PoT fails everywhere: worst strategy on all models (GPT-4o-mini 30.6% on sports; Gemini-2.0-Flash 18.2%), because code cannot parse free-text evidence.

Error analysis (100 Gemini-2.0-Flash failures)

Extraction errors (20%)

Logical inconsistency (31%)

Interpretation errors (22%)

Alignment errors (27%)

Proposed directions

Query-aware retrieval and dynamic table summarization instead of full-table attention.
Symbolic-neural hybrid architectures combining precise computation with flexible text understanding.
Domain-adaptive pretraining, tool-augmented reasoning, and robustness testing with noisy/imperfect tables.
Future extensions: multi-table joins, multimodal inputs, and time-evolving tables.

Reference

Abhyankar, N., Chaurasia, P., Kabra, S., Srivastava, A., Gupta, V., & Reddy, C. K. (2025). RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables. *arXiv preprint arXiv:2511.04491*.