HaystackCraft: A Benchmark for Realistic Long-Context and Agentic Evaluation

Forum topic · ✨步子哥 · 2025-12-11

Summary

HaystackCraft is a new benchmark introduced in arXiv:2510.07414 (October 2025) by researchers from Georgia Tech, Meta AI, UIUC, and NUS that goes beyond synthetic needle-in-a-haystack (NIAH) tests for long-context LLMs. Built on the full English Wikipedia hyperlink network with multi-hop questions, it introduces "haystack engineering"—constructing realistic noisy contexts that reflect biased heterogeneous retrieval and agentic workflow errors. The benchmark compares sparse (BM25), dense (Qwen3-Embedding), hybrid, and graph-based Personalized PageRank retrieval strategies, and extends NIAH to dynamic, LLM-dependent agentic settings with enforced and variable multi-round reasoning. Key findings: dense retrievers create harder distractors than sparse ones, PPR reranking significantly boosts retrieval, even frontier models like Gemini 2.5 Pro and GPT-5 suffer cascading self-distraction, and models tolerate noisy context "width" better than noisy reasoning "depth," with most models failing to stop early. Code is available on GitHub.

Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation

Authors: Mufei Li, Dongqi Fu, Limei Wang, Si Zhang, Hanqing Zeng, Kaan Sancak, Ruizhong Qiu, Haoyu Wang, Xiaoxin He, Xavier Bresson, Yinglong Xia, Chonglin Sun, Pan Li

Affiliations: Georgia Institute of Technology, Meta AI, University of Illinois Urbana-Champaign, National University of Singapore

Publication: arXiv:2510.07414 (October 2025)

Introduction

Modern long-context LLMs perform well on synthetic "needle-in-a-haystack" (NIAH) benchmarks, but these tests overlook how noisy contexts arise in the real world from biased retrieval and agentic workflows. More realistic evaluation is needed.

Haystack Engineering

A new paradigm to construct realistic noisy long contexts
Captures key real-world factors:
Distraction from heterogeneous biased retrievers
Cascading errors in agentic workflows
Contrasts with "context engineering," which optimizes inputs for best performance

HaystackCraft Benchmark

Built on the full English Wikipedia hyperlink network
Features multi-hop questions
Extends traditional NIAH evaluations in two ways:
Heterogeneous Retrieval-Dependent Haystacks
Dynamic, LLM-Dependent Agentic Context Engineering

Heterogeneous Retrieval Strategies

The benchmark evaluates how different retrieval strategies affect distractor composition, haystack ordering, and LLM performance:

Sparse Retrieval (BM25)
Dense Retrieval (Qwen3-Embedding-0.6B)
Hybrid Retrieval (BM25 + Qwen3-Embedding-0.6B)
Graph-Based Reranking (Personalized PageRank, PPR)

Agentic Context Engineering

Extends NIAH to dynamic, LLM-dependent settings where models refine queries, reflect on past reasoning, and decide when to stop. Two dynamic settings: Enforced Multi-Round and Variable-Round.

Key Findings

Dense retrievers introduce more challenging distractors than sparse ones
Graph-based reranking with PPR significantly improves retrieval effectiveness
Document ordering effects are model-dependent
Even advanced models (Gemini 2.5 Pro, GPT-5) suffer from cascading self-distraction
Models are more robust to noisy long contexts ("width") than to noisy reasoning iterations ("depth")
Most models struggle with appropriate early stopping in variable-round settings

Conclusion

Robust agentic long-context reasoning remains an unsolved challenge. HaystackCraft is established as a valuable testbed for future progress.

Code: <https://github.com/Graph-COM/HaystackCraft>