Overview
On August 6 (Beijing time), an anti-trend paper appeared on arXiv: Activity Frames: Compiling Deterministic Pipelines for Agent Memory from Screen Activity by independent researcher Nossa Iyamu. The same paper trended on HuggingFace Daily Papers.
The Problem
The mainstream approach to long-term agent context is "have an LLM summarize" what happened on screen each day and stuff the summary into the prompt. This has clear drawbacks: summaries are not cacheable (rerunning the same day can produce different text), not auditable (users must replay raw logs to know what the agent saw), and not reproducible.
The Approach
Iyamu's answer is a zero-model, fully deterministic pipeline:
- Every screen transition (window focus change, URL jump, scroll event) is converted into an "event frame" by a set of rules.
- Event frames are merged into "Activity Frames" using three constant parameters: dwell 90s / session gap 300s / flicker merge 20s.
- An R-tree index stores all frames from 24 hours as a byte-identical binary block.
- 86x compression: On the author's own corpus of 51 active days (128,756 frames), a full day's raw capture compresses into a prompt block 86x smaller. The paper itself notes this comes mainly from removing duplicate pixels and merging time windows — an apples-to-oranges comparison with LLM summaries, which never stored raw pixels anyway.
- 68ms compilation: Running 128,756 frames through the full pipeline takes 68ms. This is the most direct benefit of determinism + zero-model design: hot caching, offline recomputation, and CI integration become trivial.
- 98.4% QA accuracy: Handle with care. The experiment uses the author's own QA task (100 questions over 51 days of screen logs), comparing answers from compressed activity frames vs. raw logs against ground truth. The baseline is an LLM summary (GPT-4o daily activity summaries) at 66-80%. On the author's own dataset, deterministic compression wins decisively — but the conclusion's scope is narrow.
- arXiv abs: https://arxiv.org/abs/2608.05784
- arXiv HTML full text: https://arxiv.org/html/2608.05784
- GitHub repo: https://github.com/nossa-iyamu/activity-frames
- PyPI package: https://pypi.org/project/activity-frames/
- Author's write-up: https://www.passions.com/@nossa/55-days-of-screen-memory
- HuggingFace Daily Papers: https://huggingface.co/papers/2608.05784
- The evaluation dataset is author-constructed; 100 questions cover limited application scenarios
- No comparison with current SOTA RAG / long-context systems (e.g., MemGPT, Letta, Anthropic Context Retrieval)
- Single subject, single device; multi-device sync and cross-device dedup not discussed
- The three constants (dwell / session gap / flicker merge) are hardcoded; generalizability across people and apps is unknown
- Semantic understanding of screen content still requires an LLM; Activity Frames only solves the "activity stream," not the "content stream"
- Privacy: local-first is the default, but cloud-edge boundaries are not discussed
- No comparison with Apple Intelligence's screen-aware privacy mechanisms or OpenAI Operator's safety boundaries
Key Numbers
Positioning
Unlike approaches that "let an LLM look at the screen" (OpenAI's Operator, Anthropic's Computer Use, Apple's ScreenKit), Activity Frames never lets the LLM see the screen — the LLM only reads the compiled frames. LLM cost and latency drop to nearly zero; the tradeoff is that the modeling of "what happened on screen" is hand-crafted, and the author admits limited coverage of video frames, audio, and cross-window drag scenarios.
Notably, the paper includes a formal proof of cacheability: as long as the day's event sequence is unchanged, the frame output is byte-identical. This sounds obvious, but it does not hold for LLM summaries. Making frames cacheable means they can be shared across multiple agents, persisted, and differentially updated.
Links
The implementation is open-sourced as activity-frames (GitHub and PyPI, v0.1.0). The author disclosed a discrepancy between 55 days of personal use and the paper's 51 days (final cleanup days excluded from the statistical window).
Assessment
This is a specimen of "de-AI-ifying" agent long-term context engineering. The core claim is not that LLM summaries are bad, but that some tasks never needed an LLM at all. If the approach validates, agent memory architecture could evolve from the current "vector store + LLM summarization" binary into a ternary structure: "deterministic frame + selective LLM enrichment" — a development that would put projects like MemGPT in an awkward position.