Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation

Authors: Mufei Li, Dongqi Fu, Limei Wang, Si Zhang, Hanqing Zeng, Kaan Sancak, Ruizhong Qiu, Haoyu Wang, Xiaoxin He, Xavier Bresson, Yinglong Xia, Chonglin Sun, Pan Li

Affiliations: Georgia Institute of Technology, Meta AI, University of Illinois Urbana-Champaign, National University of Singapore

Publication: arXiv:2510.07414 (October 2025)

Introduction

• Modern long-context LLMs perform well on synthetic "needle-in-a-haystack" (NIAH) benchmarks.
• These tests overlook how noisy contexts arise from biased retrieval and agentic workflows.
• More realistic evaluation is needed to capture real-world factors.

Haystack Engineering

A new paradigm for constructing realistic noisy long contexts, capturing:

• Distraction from heterogeneous biased retrievers
• Cascading errors in agentic workflows

This contrasts with "context engineering," which optimizes inputs for best performance.

The HaystackCraft Benchmark

• Built on the full English Wikipedia hyperlink network
• Features multi-hop questions
• Extends traditional NIAH evaluations in two ways:
• Heterogeneous Retrieval-Dependent Haystacks
• Dynamic, LLM-Dependent Agentic Context Engineering

Heterogeneous Retrieval Strategies

The benchmark evaluates how retrieval strategies affect distractor composition, haystack ordering, and LLM performance. Strategies compared:

• Sparse Retrieval (BM25)
• Dense Retrieval (Qwen3-Embedding-0.6B)
• Hybrid Retrieval (BM25 + Qwen3-Embedding-0.6B)
• Graph-Based Reranking (Personalized PageRank - PPR)

Agentic Context Engineering

Extends NIAH to dynamic, LLM-dependent settings where models refine queries, reflect on past reasoning, and decide when to stop. Two dynamic settings:

• Enforced Multi-Round
• Variable-Round

Key Findings

• Dense retrievers introduce more challenging distractors than sparse ones.
• Graph-based reranking with PPR significantly improves retrieval effectiveness.
• Document ordering effects are model-dependent.
• Even advanced models (Gemini 2.5 Pro, GPT-5) suffer from cascading self-distraction.
• Models are more robust to noisy long contexts ("width") than to noisy reasoning iterations ("depth").
• Most models struggle with appropriate early stopping in variable-round settings.

Conclusion

Robust agentic long-context reasoning remains an unsolved challenge; HaystackCraft provides a valuable testbed for future progress.

Code: https://github.com/Graph-COM/HaystackCraft