Paper: Long-Horizon State Tracking in LLMs: Executing MD5 through Deep Sequential Tool Calls

Forum topic · 小凯 · 2026-09-03

Summary

A new arXiv paper (2509.00005) by Dheeraj Mohandas Pai and Lu Xian tests whether large language models can carry exact intermediate state across long sequences of dependent tool calls. The task: compute the MD5 cryptographic hash from scratch following RFC 1321, requiring 196 dependent tool calls over 64 rounds while maintaining four 32-bit state words (a, b, c, d) in the model's context. Unlike existing agentic benchmarks that confound state tracking with instruction interpretation and are vulnerable to shortcuts like hallucinated answers, this setup uses trivial instructions, ground-truth trajectory alignment, and bit-level digest checks, isolating pure state bookkeeping. The authors show that gpt-oss-120b, a Mixture-of-Experts model with roughly 5.5 billion active parameters per token, carries full state across all 196 calls at temperature 0 and returns correct digests in most completed runs. In the strongest setting, every raw tool is replaced by a second LLM, so a driver and a worker compute the entire hash with no exact arithmetic oracle in the loop. Two factors drive success without changing weights: preserving the model's own reasoning in context each round, and voting across a thinking-enabled worker to filter modular-arithmetic slips.

Paper Overview

Field: Machine Learning Authors: Dheeraj Mohandas Pai, Lu Xian Published: 2026-09-03 arXiv: 2509.00005

Abstract

Long-horizon tasks remain uncommon in large language model (LLM) evaluation, and for a reason: when each step depends on the last, per-step accuracy that looks excellent in isolation decays catastrophically, as errors cascade and the end-to-end failure probability grows sharply with length. Existing agentic benchmarks report end-to-end success but confound this state-tracking difficulty with instruction interpretation, give no control group that isolates it, and are vulnerable to shortcuts such as a hallucinated final answer, so they cannot say why a long run fails. Whether an LLM can carry exact intermediate state across many tool calls at all is itself not well established.

The authors test this cleanly by having the model compute a cryptographic hash, MD5, step by step: a sequence of 196 dependent tool calls over 64 rounds, carrying four 32-bit words (a, b, c, d) in the model's own context between each call. Interpretation is trivial because MD5 is implemented from scratch (RFC 1321); each call is aligned to a ground-truth trajectory and the digest is checked bit-by-bit, so any failure is pure bookkeeping.

Key Findings

gpt-oss-120b, a Mixture-of-Experts model with only ~5.5 billion active parameters per token, carries full state across all 196 calls at temperature 0 with a short fixed prompt, returning correct digests in most completed runs.
In the strongest setting, every raw tool is replaced by a second LLM — a driver and a worker compute the entire hash from scratch with no exact arithmetic oracle in the loop.
Two factors determine success, neither involving weight changes:
Residual failures are localized by source, separating state-carrying from arithmetic and from serving.

--- *Auto-collected on 2026-09-03*