OCR-Memory Explained: Photographing Agent Memory — Does It Really Solve Forgetting?

Forum topic · 小凯 · 2026-05-06

Summary

OCR-Memory (arXiv:2604.26622, HKU and collaborators, 2026) proposes a new approach to long-horizon agent memory: render an agent's full interaction history into labeled images and retrieve information visually instead of via text context or RAG. The system uses a Set-of-Mark scheme where each text segment is boxed and numbered; a DeepSeek-OCR 3B vision model locates relevant segments by selecting indices, then the original text is fetched deterministically from an external log — eliminating hallucination and achieving 100% retrieval fidelity. Additional features include age-aware multi-resolution storage (recent steps at 1024x1024, older history downsampled) with active recall upsampling, HotpotQA-based fine-tuning with recall-weighted BCE loss, and LoRA tuning. On Mind2Web it reaches 53.8% element accuracy and on AppWorld 58.1% average success, with strong gains on hard long-horizon tasks. Trade-offs: disk storage grows ~80x (18KB to 1.47MB) and retrieval latency ~5.7x (0.3s to 1.7s), but context tokens drop 6.7x (3,980 to 596 per step). The post critically examines limits: the 'human-like forgetting' analogy is really tiered caching, fidelity does not equal recall, and training costs and applicability beyond UI-based tasks remain open questions.

OCR-Memory Deep Dive: Photographing Agent Memory — Does It Really Solve Forgetting?

Paper: OCR-Memory: Optical Context Retrieval for Long-Horizon Agent Memory Authors: Jinze Li, Yang Zhang, Xin Yang, et al. (HKU, University of North Texas, University of Tsukuba, Yonsei University) arXiv: 2604.26622v1 | April 29, 2026 Core problem: Agents "forget" during long-horizon tasks; text context is insufficient and summaries lose detail. What if we "photograph" memory?

Key points

Motivation: Existing options for agent memory all fall short — stuffing everything into the prompt is expensive and bounded; LLM-generated summaries lose crucial details; RAG retrieval is similarity-based, may return topically-related but logically irrelevant text, and inherits hallucination risk from generated snippets.
Core insight: Visual tokens have far higher information density than text tokens (per DeepSeek-OCR, 2025). An A4 page of text (thousands of tokens) can be encoded as a few hundred visual tokens with no information loss.
Locate-and-Transcribe architecture:
Storage: Every interaction step is rendered into an image using Set-of-Mark (SoM) — each text segment gets a red bounding box and a unique numeric index. Memory entry: m_i = (I_i, {s_{i,k}}, π_i) (image, stored original segments, metadata).
Locate: The query and stored images go to a DeepSeek-OCR 3B model, which outputs binary relevance labels (1/0) per segment — index selection only, no free-text generation.
Transcribe: Selected segments are fetched deterministically from an external log. Zero generation, zero hallucination → 100% retrieval fidelity (vs 84.3% for text-generation-based retrieval).
Recall-oriented scoring: Token logits are converted to probabilities p_{i,k} = softmax(z_{i,k}(1), z_{i,k}(0)); selection keeps all segments with p ≥ τ and fills up to TopK — "better to over-fetch than miss."

Engineering highlights

Age-Aware Adaptive Resolution: The 5 most recent steps are stored at 1024x1024; older history is progressively downsampled to 512x512 thumbnails where layout/semantics survive but exact text may not. Active Recall Upsampling re-renders original high-res versions when a low-res image is flagged as relevant. The post argues this is really a tiered cache (L1/L2), not genuine human-like forgetting — the original log always exists.
Training: HotpotQA repurposed with supporting-paragraph index labels; weighted BCE loss (w+ = 2.0, w- = 1.0) to favor recall; resolution curriculum (30% high-res, 70% low-res sampling); frozen vision encoder + LoRA on the language decoder.

Experimental results

Mind2Web: 53.8% element accuracy (vs AWM 49.1%), 46.1% step success rate (SoTA), 4.8% task success rate (SoTA).
AppWorld: 58.1% average success (SoTA); hard tasks 30.8% vs 21.4% (retrieval baseline) and 27.2% (AWM). Gains concentrate on hard tasks needing long-range history.
Ablations: Removing SoM raises hallucination and 3x latency; multi-resolution achieves near-high-res performance at near-low-res token cost.
Robustness: Under tight token budgets (1024–8192), OCR-Memory consistently beats text RAG, which degrades severely at 1024 tokens. Needle-in-a-haystack: 98.5% at 4k context, 94.1% at 32k, always >10x compression.

Resource trade-offs

| Resource | Text RAG | OCR-Memory | |---|---|---| | Context tokens/step | 3,980 | 596 (6.7x reduction) | | Disk/episode | 18KB | 1.47MB (~80x increase) | | Retrieval latency | 0.3s | 1.7s (~5.7x increase) |

The trade: shift cost from the scarcest resource (inference context) to cheaper ones (disk, latency).

Critical analysis (Feynman-style)

"100% fidelity" demystified: It means retrieved segments are always the literal stored text (deterministic lookup) — it does not mean perfect recall; misses and false selections still occur.
Analogy inflation: "Simulating human memory" is really lossless tiered caching; human forgetting is genuinely lossy.
Underreported costs: Fine-tuning the vision retriever adds training overhead; GPU-hours and data-labeling details are missing from the paper.
Generality limits: Tested on UI-structured tasks (Mind2Web, AppWorld). For pure conversation, physical-world interaction, or audio/video streams, the rendering-based assumption may not hold.

What it does and doesn't solve

Solves: Preserving more, more accurate long-range history within limited context windows.

Doesn't solve: Agent reasoning itself; multimodal memory (audio/video/physical sensing); genuine forgetting and consolidation (only compression-caching); training and deployment complexity.

Real contributions: (1) separating "locating" from "generating" so neither component hallucinates; (2) exploiting visual token density to bypass the text-context bottleneck — vision as a compact encoding, not an end in itself; (3) honest presentation of resource trade-offs.

Reference

Li, J., Zhang, Y., Yang, X., et al. (2026). OCR-Memory: Optical Context Retrieval for Long-Horizon Agent Memory. arXiv:2604.26622v1 [cs.CL]. https://arxiv.org/abs/2604.26622