OCR-Memory Explained: Can 'Photographing' Agent Memory Fix the Forgetting Problem?

This post is a deep-dive analysis of OCR-Memory: Optical Context Retrieval for Long-Horizon Agent Memory (Jinze Li, Yang Zhang, Xin Yang, et al.; HKU, University of North Texas, University of Tsukuba, Yonsei University). arXiv:2604.26622v1, April 29, 2026. https://arxiv.org/abs/2604.26622

The Problem

Agents doing long-horizon tasks (e.g., booking flights across a dozen browser steps) forget early reasoning traces when the context window fills up. Existing options all fall short:

• Stuff everything into the prompt — too expensive, and windows are finite.
• Text summarization — summaries lose details the agent may need later.
• Text RAG — semantic retrieval returns 'topically related but logically irrelevant' snippets, and generated text carries hallucination risk.

OCR-Memory takes a fourth path: photograph the agent's full interaction history, let the model 'look' at the images to locate what it needs, then deterministically fetch the original text from an external log.

Key Insight: Vision as a High-Density, Lossless Medium

Building on DeepSeek-OCR (2025), the authors note that a dense page of text can be encoded into far fewer vision tokens without information loss. OCR-Memory reallocates cost from the scarce resource (LLM context) to cheaper ones (disk storage + a small vision encoder).

Architecture: Locate-and-Transcribe

Storage. Each interaction step (user input, reasoning traces, tool calls, environment feedback) is rendered into an image where every text segment is boxed and tagged with a unique numeric index (Set-of-Mark). Memory entries are structured as M = {m_i}, with m_i = (I_i, {s_{i,k}}, π_i): the marked image, the original text segments in an external log, and metadata.

Retrieval. 1. Locate — a DeepSeek-OCR 3B model receives the query plus stored images and outputs a binary relevance vector over segments. No free-text generation, only index selection. 2. Transcribe — selected indices are used for a deterministic database lookup of the literal original text. Zero generation, zero hallucination.

This separation yields 100% retrieval fidelity (the transcribed text always matches what was stored) versus 84.3% for generative text retrieval — though it does not guarantee perfect recall or selection.

Scoring. Token logits are converted to probabilities p_{i,k} = softmax(z(1), z(0)), and a recall-oriented rule keeps all segments above a threshold τ, backfilling with top-K if needed — favoring over-retrieval since missing key evidence is costlier than extra irrelevant segments.

Age-Aware Adaptive Resolution

• Recent 5 steps: high resolution (1024×1024).
• Older history: progressively downsampled to 512×512 thumbnails that retain semantic layout even when text becomes unreadable.
• Active Recall Upsampling: when a relevant segment is spotted in a thumbnail, the original high-res image is re-rendered from the log and used for fine-grained retrieval.

The post notes this is less 'human-like forgetting' and more a tiered cache (L1/L2) strategy — the original log is never destroyed.

Training

• HotpotQA is relabeled: 10 segments rendered per image with SoM marks; the model learns to output a binary support vector.
• Weighted BCE loss (w+ = 2.0, w- = 1.0) penalizes missed relevant segments twice as much as false positives.
• Resolution curriculum: random sampling between 1024×1024 (30%) and 512×512 (70%).
• Frozen vision encoder + LoRA fine-tuning of the language decoder.

Results

Mind2Web (web navigation): Element Accuracy 53.8% vs AWM's 49.1%; Step SR 46.1% and Task SR 4.8% (both SoTA).

AppWorld (API interaction): 58.1% average success (SoTA); hard tasks 30.8% vs 21.4% (retrieval baseline) and 27.2% (AWM) — gains are largest where long-horizon dependencies matter.

Ablations: Removing SoM (free-text generation) triples inference latency and increases hallucination; dynamic resolution achieves near-high-res performance at near-low-res token cost.

Needle-in-a-Haystack: 98.5% accuracy at 4k context, 94.1% at 32k, with sustained >10x compression.

Resource trade-off:

| Resource | Text RAG | OCR-Memory | |---|---|---| | Context tokens/step | 3,980 | 596 (6.7x less) | | Disk per episode | 18KB | 1.47MB (~80x more) | | Retrieval latency | 0.3s | 1.7s (~5.7x more) |

Critical Assessment (Feynman-style)

• '100% fidelity' is honest but limited: it means selected segments are transcribed verbatim, not that every retrieval is correct. Misses and misselections still occur.
• 'Simulating human memory' overstates it: human forgetting is lossy; OCR-Memory's is lossless compression — a caching hierarchy, not hippocampal consolidation.
• Training cost is under-specified: the paper admits fine-tuning is required but omits GPU-hour and data-preparation details — a real deployment barrier.
• Generalization boundary: the approach assumes structured, renderable histories (HTML, API logs). Pure conversation, physical-world robotics, or audio/video streams may not benefit.

Verdict

OCR-Memory's real contributions: (1) separating *locating* (vision model) from *producing* (deterministic DB lookup), eliminating hallucinated retrieval; (2) using the density of visual tokens to bypass the text-context bottleneck; (3) an honest accounting of the storage/latency-for-context trade-off. It does not improve the agent's reasoning itself, handle non-visual modalities, or implement true forgetting — it is a clever engineering compromise for UI-structured agent tasks, not magic.

Reference: Li, J., Zhang, Y., Yang, X., et al. (2026). *OCR-Memory: Optical Context Retrieval for Long-Horizon Agent Memory.* arXiv:2604.26622v1 [cs.CL]. https://arxiv.org/abs/2604.26622