Long-Horizon State Tracking in LLMs: Executing MD5 through 196 Dependent Tool Calls

Forum topic · 小凯 · 2026-09-03

Summary

A paper by Dheeraj Mohandas Pai and Lu Xian (arXiv:2509.00005) tests whether large language models can carry exact intermediate state across long-horizon tool-use sequences. The benchmark asks a model to compute the MD5 cryptographic hash step by step: 196 dependent tool calls over 64 rounds, carrying four 32-bit words (a, b, c, d) in its own context between calls. Because MD5 is implemented from scratch (RFC 1321) and every call is checked bit-by-bit against a ground-truth trajectory, any failure is pure state-tracking bookkeeping rather than instruction interpretation. The authors show that gpt-oss-120b, a mixture-of-experts model with roughly 5.5 billion active parameters per token, maintains full state across all 196 calls at temperature 0 and returns correct digests in most completed runs. In the strongest setup, both driver and worker are LLMs computing the entire hash with no exact-arithmetic oracle. Success depends on two factors that require no weight changes: preserving the model's own reasoning in context each round, and voting across a thinking-enabled worker to suppress modular-arithmetic slips. Failures are attributed by source, separating state-carrying from arithmetic and serving effects.

Overview

Field: Machine Learning Authors: Dheeraj Mohandas Pai, Lu Xian Published: 2026-09-03 arXiv: 2509.00005

Summary

Long-horizon tasks remain uncommon in large language model (LLM) evaluation, and for a reason: when each step depends on the last, per-step accuracy that looks excellent in isolation decays catastrophically, as errors cascade and the end-to-end failure probability grows sharply with length. Existing agentic benchmarks report end-to-end success but confound this state-tracking difficulty with instruction interpretation, give no control group that isolates it, and are vulnerable to shortcuts such as a hallucinated final answer, so they cannot say why a long run fails. Whether an LLM can carry exact intermediate state across many tool calls at all is itself not well established.

This paper tests the question cleanly by having the model compute a cryptographic hash, MD5, step by step: a sequence of 196 dependent tool calls spread over 64 rounds, while carrying four 32-bit words (a, b, c, d) in its own context between each call. Interpretation is trivial because the authors implement MD5 from scratch (RFC 1321), align every call with the ground-truth trajectory, and check the digest bit-by-bit, so any failure is pure bookkeeping.

Key findings

gpt-oss-120b (a mixture-of-experts model with only ~5.5B active parameters per token) carries the complete state across all 196 calls at temperature 0 with a short fixed prompt, and returns the correct digest in most completed runs.
In the strongest setting, every raw tool is replaced by a second LLM, so a driver and a worker compute the entire hash from scratch with no exact-arithmetic oracle in the loop.
Two factors determine success, and neither involves changing weights:
Preserving the model's own reasoning in its context at every round.
Voting across a thinking-enabled worker to eliminate modular-arithmetic slips.
Residual failures are localized by source, separating state-carrying from arithmetic and serving effects.

Long-Horizon State Tracking in LLMs: Executing MD5 through 196 Dependent Tool Calls

Summary

Overview

Summary

Key findings

Links

Tags