Paper: Long-Horizon State Tracking in LLMs — Executing MD5 through 196 Dependent Tool Calls

Forum topic · 小凯 · 2026-09-03

Summary

A new arXiv paper (2509.00005) by Dheeraj Mohandas Pai and Lu Xian tests whether large language models can carry exact intermediate state across long sequences of tool calls. The authors have an LLM execute the MD5 cryptographic hash step by step: 196 dependent tool calls over 64 rounds, carrying four 32-bit words (a, b, c, d) in its own context between calls. Because MD5 is implemented from scratch per RFC 1321 and every call is checked bit-by-bit against a ground-truth trajectory, any failure is pure state-tracking or bookkeeping error, not instruction misinterpretation. gpt-oss-120b, a mixture-of-experts model with ~5.5B active parameters per token, maintained full state across all 196 calls at temperature 0 and returned correct digests in most completed runs. In the strongest setting, every raw tool is replaced by a second LLM, so a driver and a worker compute the entire hash with no exact-arithmetic oracle. Two factors drive success without changing model weights: preserving the model's own reasoning in context each round, and voting across a reasoning-enabled worker to suppress modular-arithmetic mistakes.

Paper Overview

Field: Machine Learning
Authors: Dheeraj Mohandas Pai, Lu Xian
Published: 2026-09-03
arXiv: 2509.00005

Abstract

Long-horizon tasks remain uncommon in large language model (LLM) evaluation, and for a reason: when each step depends on the last, per-step accuracy that looks excellent in isolation decays catastrophically, as errors cascade and the end-to-end failure probability grows sharply with length. Existing agentic benchmarks report end-to-end success but confound this state-tracking difficulty with instruction interpretation, give no control group that isolates it, and are vulnerable to shortcuts such as a hallucinated final answer, so they cannot say why a long run fails. Whether an LLM can carry exact intermediate state across many tool calls at all is itself not well established.

The authors test this cleanly by having the model compute a cryptographic hash, MD5, step by step: a sequence of $196$ dependent tool calls over 64 rounds, carrying four 32-bit words (a, b, c, d) in the model's own context between calls. Interpretation is trivial because MD5 is implemented from scratch (RFC 1321); every call is aligned with a ground-truth trajectory and the digest is checked bit-by-bit, so any failure is pure bookkeeping.

Key Findings

gpt-oss-120b (a mixture-of-experts model with only ~5.5B active parameters per token), at temperature 0 with a short fixed prompt, carried full state across all 196 calls and returned the correct digest in most completed runs.
In the strongest setup, each raw tool is replaced by a second LLM — a driver and a worker compute the entire hash from scratch with no exact-arithmetic oracle in the loop.
Two factors determine success, neither requiring weight changes:
Preserving the model's own reasoning in its context at every round.
Voting across a reasoning-enabled worker to eliminate its modular-arithmetic mistakes.
Residual failures are localized by source, separating state-carrying failures from arithmetic and service failures.

Why It Matters

The benchmark design provides the missing control group for long-horizon agentic evaluation: since instructions are trivial and every step is verifiable against ground truth, observed failures isolate exactly the model's ability to maintain precise intermediate state across many tool calls — a capability orthogonal to instruction following.

--- *Auto-collected on 2026-09-03*