Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation

Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation

Haystack Engineering: Context Engineering for Heterogeneous and Agentic Long-Context Evaluation

Mufei Li, Dongqi Fu, Limei Wang, Si Zhang, Hanqing Zeng, Kaan Sancak, Ruizhong Qiu, Haoyu Wang, Xiaoxin He, Xavier Bresson, Yinglong Xia, Chonglin Sun, Pan Li

Georgia Institute of Technology, Meta AI, University of Illinois Urbana-Champaign, National University of Singapore

arXiv:2510.07414 (October 2025)

lightbulb
Introduction

Modern long-context LLMs perform well on synthetic "needle-in-a-haystack" (NIAH) benchmarks

These tests overlook how noisy contexts arise from biased retrieval and agentic workflows

Need for more realistic evaluation that captures real-world factors

Traditional needle-in-a-haystack evaluation

architecture
Haystack Engineering

New paradigm to construct realistic noisy long contexts

Captures key real-world factors:

Distraction from heterogeneous biased retrievers

Cascading errors in agentic workflows

Contrast with "context engineering" (optimizing inputs for best performance)

assessment
HaystackCraft Benchmark

Built on full English Wikipedia hyperlink network

Features multi-hop questions

Extends traditional NIAH evaluations in two ways:

Heterogeneous Retrieval-Dependent Haystacks

Dynamic, LLM-Dependent Agentic Context Engineering

compare_arrows
Heterogeneous Retrieval Strategies

Evaluates how different retrieval strategies affect:

Distractor composition

Haystack ordering

LLM performance

Strategies compared:

Sparse Retrieval (BM25)

Dense Retrieval (Qwen3-Embedding-0.6B)

Hybrid Retrieval (BM25 + Qwen3-Embedding-0.6B)

Graph-Based Reranking (Personalized PageRank - PPR)

Comparison of different retrieval methods

psychology
Agentic Context Engineering

Extends NIAH to dynamic, LLM-dependent settings

Simulates agentic operations where models:

Refine queries

Reflect on past reasonings

Decide when to stop

Two dynamic settings:

Enforced Multi-Round

Variable-Round

Agentic workflow with cascading errors

insights
Key Findings

Dense retrievers introduce more challenging distractors than sparse ones

Graph-based reranking with PPR significantly improves retrieval effectiveness

Document ordering effects are model-dependent

Even advanced models (Gemini 2.5 Pro, GPT-5) suffer from cascading self-distraction

Models are more robust to noisy long contexts ("width") than to noisy reasoning iterations ("depth")

Most models struggle with appropriate early stopping in variable-round settings

flag
Conclusion

Robust agentic long-context reasoning remains an unsolved challenge

HaystackCraft established as a valuable testbed for future progress

code
Code available at GitHub

© 2025 Haystack Engineering Research Team