OOLONG Benchmark: Deep Dive into Long-Context Reasoning and the RLM Breakthrough

Forum topic · ✨步子哥 · 2026-01-20

Summary

OOLONG (Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities), introduced in November 2025 (arXiv:2511.02817) by MIT CSAIL researchers, is a benchmark designed to test long-context information aggregation rather than simple needle-in-a-haystack retrieval. It includes OOLONG-Synth (synthetic in-context learning tasks) and OOLONG-Real (real-world DnD transcript-based tasks), plus a quadratic-complexity variant, OOLONG-Pairs, requiring pairwise entity aggregation. Frontier models perform poorly: many score under 50% on standard OOLONG at 128K tokens, and baselines achieve F1 near 0.04–0.1% on OOLONG-Pairs even with million-token contexts. The follow-up Recursive Language Models (RLM) paper (arXiv:2512.24601) uses OOLONG as a core benchmark, lifting OOLONG-Pairs performance from under 0.1% to 58% F1 by loading context into a Python REPL and recursively generating code to retrieve only relevant segments. Open-source resources include the GitHub repo (abertsch72/oolong) and Hugging Face datasets. Together, these results suggest that simply expanding context windows is insufficient, and that neurosymbolic context management may be the key path forward for long-context reasoning.

OOLONG (full name Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities) is one of the most rigorous long-context AI benchmarks from late 2025. Rather than simply testing needle-in-a-haystack retrieval (like Needle-in-a-Haystack), it forces models to perform genuine information aggregation and multi-hop reasoning, exposing fatal weaknesses in current long-context models. Below is a systematic overview based on the latest results (including the arXiv paper, GitHub, Hugging Face datasets, and community discussions).

Core Design and Extreme Difficulty

Release info: November 4, 2025, arXiv:2511.02817, lead author Andrew Bertsch et al. (MIT CSAIL).
Two subsets:

OOLONG-Synth

OOLONG-Real

Task types:
Linear complexity (standard OOLONG): requires line-by-line analysis, counting, or aggregation (e.g., summing an attribute across all characters).
Quadratic complexity (OOLONG-Pairs): the hardest part! Requires the model to form all pairwise combinations of entities in a document and aggregate (e.g., counting interactions between every pair of characters). Complexity is O(n²); with long contexts (128K+ tokens), traditional models collapse outright — accuracy below 0.1%, essentially no better than guessing.
Why so hard?
Current long-context benchmarks (e.g., S-NIAH) are mostly "retrieval-type": finding one needle is enough.
OOLONG is "aggregation-type": the model must understand every text segment at an atomic level, then integrate globally. Transformer attention suffers "Context Rot" on very long sequences (middle information gets drowned out), leading to a phase transition — the model suddenly goes from "barely coping" to "completely broken."
Outputs are designed to be easily verifiable (numeric/logical answers), avoiding ambiguity in subjective scoring.

Frontier Models Struggle on OOLONG

The paper benchmarked multiple late-2025 frontier models (including GPT-5-class):
On standard OOLONG at 128K context, many models score below 50% accuracy.
On OOLONG-Pairs (quadratic tasks), baseline models — even with million-token contexts — achieve F1 scores near 0.04%–0.1%, essentially complete failure.
Performance continues to collapse even at 263K tokens.
This directly demonstrates that merely stacking context window size is useless: models do not truly "utilize" the full context.

RLM Meets OOLONG: From 0% to 58%

In the follow-up Recursive Language Models (RLM) paper (arXiv:2512.24601, December 31, 2025, by Alex Zhang et al., MIT CSAIL), OOLONG serves as the core evaluation benchmark.
RLM's standout results:
On OOLONG-Pairs: baselines below 0.1% → RLM reaches 58% F1 (a lead of 50+ percentage points).
At larger scales (10M+ tokens), RLM maintains high accuracy, while traditional approaches (summarization + RAG) explode in cost or fail entirely.
Even small models (e.g., RLM + GPT-5-mini) crush native GPT-5 on OOLONG (a 114% improvement).
Mechanism recap: RLM loads the long context into a Python REPL, where the model recursively generates code (slicing, filtering, sub-queries), pulling only necessary segments and avoiding the Context Rot caused by stuffing everything into context.

Open-Source Resources and Community Updates (January 2026)

GitHub: https://github.com/abertsch72/oolong (includes datasets, evaluation scripts, and model outputs).
Hugging Face: datasets such as oolongbench/oolong-real are public and ready to download for testing.
Open-source RLM implementations: several community versions, e.g., github.com/alexzhang13/rlm and github.com/ysz/recursive-llm.
X/Twitter discussions (as of January 20, 2026):
The community widely views OOLONG as "the ultimate interrogation of long context"; many who reproduced the results noted that "traditional models really are near zero on the Pairs task."
RLM is being hailed as "the 2026 paradigm shift"; some have already integrated it into agent frameworks for codebase analysis.
The MIT CSAIL official account and authors (e.g., @a1zhang) are actively engaged, encouraging new model submissions to the OOLONG leaderboard.

What This Means for AI's Future

OOLONG punctures the myth that "long window = strong reasoning," pushing the industry from "stacking tokens" toward "smart context management." Combined with RLM's neurosymbolic approach (precise code control + local reasoning via sub-models), this may be the key path past the current bottleneck — no need for unlimited parameter scaling, just teaching models to "think like programmers."

The MIT one-two punch (OOLONG + RLM) is truly impressive — long-context reasoning looks set to take off in 2026 🚀