Overview
OOLONG (Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities) is one of the most demanding long-context benchmarks released in late 2025. Unlike Needle-in-a-Haystack style retrieval tests, OOLONG forces models to perform true information aggregation and multi-hop reasoning across very long documents, exposing structural weaknesses in current transformer architectures.
Core Design and Difficulty Sources
- Release: November 4, 2025, arXiv:2511.02817, lead author Andrew Bertsch and colleagues at MIT CSAIL.
- Two subsets:
- OOLONG-Synth: Synthetic tasks adapted from classical in-context learning datasets. Strong controlled variables make it suitable for analyzing verifiable aggregation behavior.
- OOLONG-Real: Built from live Dungeons & Dragons transcript data. Questions require narrative comprehension and multi-character interaction aggregation that cannot be decomposed trivially.
- Task complexity levels:
- Linear OOLONG: Sequential line-level analysis, counting, or aggregation (e.g., sum a property across all characters).
- Quadratic OOLONG-Pairs: The most punishing setting. Requires pairwise processing of every entity in the document (e.g., counting interactions between every character pair). Complexity scales as O(n²), and at 128K+ tokens traditional models collapse.
- Why it is hard:
- Most long-context benchmarks are retrieval-based.
- OOLONG is aggregation-based and demands atomic understanding of every segment before global integration.
- Attention degrades on very long sequences, producing "context rot" and abrupt phase transitions where capability collapses suddenly.
- Outputs are numerically and logically verifiable, removing subjective grading ambiguity.
- On 128K standard OOLONG, several late-2025 frontier models scored below 50% accuracy.
- On OOLONG-Pairs, baseline models—even those with million-token windows—achieved F1 scores between roughly 0.04% and 0.1%, essentially failing.
- Performance continued to degrade up to 263K tokens.
- The result demonstrates that merely enlarging the context window does not guarantee usable long-context reasoning.
- On OOLONG-Pairs: baseline under 0.1% → RLM reaches approximately 58% F1, a gain of more than 50 percentage points.
- At 10M+ token scales, RLM maintains high accuracy while summary- or RAG-based approaches either explode in cost or fail outright.
- Smaller setups such as RLM + GPT-5-mini outperform native GPT-5 on OOLONG by about 114%.
- Mechanism: long context is loaded into a Python REPL; the model emits code to slice, filter, and recursively sub-query the context, fetching only the necessary fragments.
- GitHub: https://github.com/abertsch72/oolong (datasets, evaluation scripts, model outputs).
- Hugging Face: datasets such as
oolongbench/oolong-realare publicly downloadable. - Open-source RLM implementations include https://github.com/alexzhang13/rlm and community forks such as https://github.com/ysz/recursive-llm.
- Community discussion treats OOLONG as a "stress test" for long context; reproductions commonly report near-zero baseline scores on Pairs. RLM is described by some practitioners as a 2026 paradigm shift and is being integrated into agent frameworks for codebase-scale analysis.
- OOLONG paper: arXiv:2511.02817
- Recursive Language Models: arXiv:2512.24601
- Code: https://github.com/abertsch72/oolong
- RLM reference implementation: https://github.com/alexzhang13/rlm
- Dataset:
oolongbench/oolong-realon Hugging Face
Frontier Model Performance
RLM on OOLONG: A Large Jump
The Recursive Language Models (RLM) paper (arXiv:2512.24601, December 31, 2025, Alex Zhang et al., MIT CSAIL) used OOLONG as its central benchmark.
Open Resources and Community Activity (as of January 2026)
Implications
OOLONG challenges the assumption that longer windows equal stronger reasoning. Combined with the neuro-symbolic approach of RLM—precise code-level control plus local sub-model reasoning—it points to a path beyond raw context scaling: teaching models to manage context the way a programmer would.