LongDA: A Benchmark for Evaluating LLM Agents on Long-Document Data Analysis

Overview

LongDA is a benchmark for evaluating LLM agents performing data analysis over long documents, published on arXiv in January 2026.

| Field | Value | |---|---| | Paper | LongDA: Benchmarking LLM Agents for Long-Document Data Analysis | | Authors | Yiyang Li, Zheyuan Zhang, Tianyi Ma, Zehong Wang, Keerthiram Murugesan, Chuxu Zhang, et al. (7 authors) | | Link | https://arxiv.org/abs/2601.02598 | | Category | Document understanding |

Key points

• The benchmark targets agentic data analysis over long documents, a task combining document understanding, multi-step reasoning, retrieval, and tool use.
• It reflects the shift in evaluation from static retrieval metrics (nDCG, MRR, Recall@k) toward task success rates, citation accuracy, and multi-hop reasoning quality.
• The work is positioned in the agentic search space, where the choice of whether, when, and how often to retrieve or invoke tools becomes part of the agent's learnable policy.

Context: evolution of retrieval and agentic search

The forum post situates LongDA within the broader progression of neural information retrieval:

1. Sparse retrieval (BM25) — efficient, robust, but limited semantic matching. 2. Dense bi-encoder retrieval — millisecond-level ANN recall, sensitive to domain shift and long-tail queries. 3. Cross-encoders and late interaction — higher accuracy, but harder to precompute or scale. 4. Generative retrieval and RAG — reduces cascade error, complicates index updates. 5. LLM agents / agentic search — retrieval becomes an iterative, plannable, verifiable process; latency, token cost, and safety become first-class constraints.

Evaluation considerations

Typical benchmarks in this space are evaluated with:

• Metrics: task success rate, nDCG@10, MRR, Recall@k, human preference, latency and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
• Ablations: number of retrieval steps, reranking depth, training data scale.

Open problems highlighted

• Trustworthiness of automated evaluation (LLM-as-judge vs. human assessment).
• Latency, cost, and reasoning-budget allocation at test time.
• Hallucination, prompt injection, and safety risks when agents access open knowledge sources.
• Cross-lingual and multimodal generalization.

Practical checklist (from the post)

| Concern | Recommendation | |---|---| | Data | Partition indexes, redact PII, version embeddings with rollback support | | Latency | Cascade retrieval with early stopping, cache hot queries, async reranking | | Quality | Validate offline gains with interleaving experiments and citation checks | | Safety | Source allowlists, adversarial detection, output filtering | | Cost | Route to smaller models, distillation, hybrid sparse+dense retrieval |

Notes and caveats

This forum post is largely based on the paper's abstract and public metadata; specific quantitative results, datasets, and task definitions should be verified against the original PDF.

Related entries

• Qwen2.5-VL Technical Report (document understanding and OCR)
• SmolDocling: ultra-compact vision-language model
• ColPali: Efficient Document Retrieval with Vision Language Models

Glossary

| Term | Meaning | |---|---| | RAG | Retrieval-Augmented Generation | | Agentic Search | Modeling search as sequential decision-making and tool invocation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Gen-IR | Generative Information Retrieval |