RUST-BENCH: New Benchmark Exposes How Badly LLMs Struggle with Real-World Table Reasoning

Forum topic · QianXun · 2025-11-08

Summary

A team from Virginia Tech, IIT Delhi, and Arizona State University introduced RUST-BENCH (arXiv:2511.04491), a benchmark of 7,966 human-verified QA pairs over 2,031 real-world tables from NBA sports records and NSF science funding data. Unlike clean benchmarks such as WikiTQ, these tables average 45.1 rows and 23,040 tokens, mixing structured columns with long free-text fields. Results show top models collapsing: GPT-4o-mini achieves only 36.6-39.8% zero-shot accuracy, while the best model, Qwen-QwQ, reaches just 54-56%. Notably, Program-of-Thought prompting fails across all models, and Chain-of-Thought shows diminishing or negative returns for larger models. Error analysis of 100 cases reveals four failure patterns: extraction errors (20%), logical inconsistency (31%), interpretation errors (22%), and alignment errors (27%). The paper argues real table reasoning requires fusing symbolic precision with semantic flexibility, and outlines future directions including hybrid symbolic-neural architectures, domain adaptation, robustness testing, and multimodal extension.

Key points

RUST-BENCH (arXiv:2511.04491), by Nikhil Abhyankar et al. from Virginia Tech, IIT Delhi, and Arizona State University, is a benchmark for LLM reasoning over unstructured text embedded within structured tables — a gap that existing benchmarks like WikiTQ and TabFact (short, clean tables) fail to capture.

Why real-world tables are hard

Analysis of 2,031 real tables identified four challenge dimensions:

Scale: Tables average 45.1 rows / 23,040 tokens (~20x WikiTQ); NSF funding tables reach 31,948 tokens. Accuracy of GPT-4o-mini and Gemini-2.0-Flash declines monotonically as table length grows from 10K to 85K tokens, with a sharp drop past 35K–50K tokens.
Multi-hop reasoning: 26.18% of questions require chaining structured lookups, free-text filtering, and temporal/arithmetic computation.
Heterogeneity: Tables mix structured columns with long free-text fields (e.g., 477-token NSF abstracts, ~400-token game summaries). An experiment converting the same table to purely structured, semi-structured, and purely unstructured forms showed PoT excels on structured tables, CoT on unstructured ones — but both struggle on the semi-structured original.
Domain specificity: Fields like "Standard Grant" vs "Continuing Grant" and dates like "Award Effective Date" require domain knowledge beyond surface keyword matching.

How the benchmark was built

Symbolic path: 75 SQL templates instantiated by LLMs and converted to natural-language questions → 2,510 QA pairs.
Semantic path: single-row (deep reasoning on one record; 3,517 pairs) and multi-row strategies (3–5 related records; 27% discarded in validation).
Human verification: 8 CS graduate students vetted clarity, answer correctness, and reasoning complexity; 6–31% of data was filtered, leaving 7,966 QA pairs.

Main results

Best model: Qwen-QwQ at 54.1% (science) / 55.7% (sports) LLM-score; Mistral-Small-3.2 led on Exact Match (52.6%).
GPT-4o-mini: 36.6% (science) / 39.8% (sports) zero-shot; rising to 44.4% / 42.1% with Chain-of-Thought.
CoT paradox: CoT helps small models (e.g., Llama-3.3-70B: 38.8% → 44.2%) but yields diminishing or negative returns for stronger ones (GPT-4o-mini sports: CoT 42.1% vs Few-Shot 46.7%).
PoT collapse: Program-of-Thought was the worst strategy on every model (e.g., Gemini-2.0-Flash: 18.2% on sports), because code cannot parse free-text fields.

Error taxonomy (100 Gemini-2.0-Flash failures)

1. Extraction errors (20%): missing evidence scattered across rows during cross-row scanning. 2. Logical inconsistency (31%): reasoning steps list multiple correct items but the final answer omits some. 3. Interpretation errors (22%): misreading temporal phrases or domain-specific fields. 4. Alignment errors (27%): correct reasoning but wrong output format (e.g., summing values when the question asked to list them).

Future directions proposed

Hybrid symbolic-neural architectures with dynamic summarization and query-aware retrieval.
Domain-adaptive pretraining and tool-augmented reasoning.
Robustness testing with noisy tables (missing cells, typos, schema drift) and better calibration.
Multimodal and multi-table extensions.

References

1. Abhyankar, N., Chaurasia, P., Kabra, S., Srivastava, A., Gupta, V., & Reddy, C. K. (2025). RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables. *arXiv preprint arXiv:2511.04491*. 2. Pasupat, P., & Liang, P. (2015). Compositional semantic parsing on semi-structured tables. *ACL 2015*. 3. Chen, W., et al. (2020). HybridQA: A dataset of multi-hop question answering over tabular and textual data. *Findings of EMNLP*. 4. Liu, N. F., et al. (2023). Lost in the middle: How language models use long contexts. *TACL*. 5. Wei, J., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. *NeurIPS*.