English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Activity Frames: A Deterministic Pipeline That Compiles Screen Activity into Agent Memory

Forum topic · 小凯 · 2026-08-08

Summary

An arXiv paper by independent researcher Nossa Iyamu proposes Activity Frames, a zero-LLM, deterministic pipeline that compiles raw screen activity into compact, cacheable agent memory. Screen events (window focus changes, URL jumps, scroll events) are converted into event frames, merged via three constant parameters (90s dwell, 300s session gap, 20s flicker merge), and indexed with an R-tree into byte-identical binary blocks. Reported results on the author's own 51 active days of data (128,756 frames) include an 86x compression of a full day's capture, 68ms total pipeline compilation, and 98.4% QA accuracy versus a 66-80% baseline from GPT-4o-generated summaries, though the evaluation uses the author's own 100-question QA dataset on a single subject. Unlike LLM-based screen understanding approaches (OpenAI Operator, Anthropic Computer Use), the LLM never sees the screen and only reads compiled frames, enabling caching, offline recomputation, and reproducibility backed by a formal byte-identical cacheability proof. The implementation is open-sourced on GitHub and PyPI. Limitations include narrow QA scope, n=1 evaluation, hardcoded parameters, and no comparison with SOTA long-context/RAG systems like MemGPT or Letta.

Overview

On August 6 (Beijing time), an anti-trend paper appeared on arXiv: Activity Frames: Compiling Deterministic Pipelines for Agent Memory from Screen Activity by independent researcher Nossa Iyamu. The same paper trended on HuggingFace Daily Papers.

The Problem

The mainstream approach to long-term agent context is "have an LLM summarize" what happened on screen each day and stuff the summary into the prompt. This has clear drawbacks: summaries are not cacheable (rerunning the same day can produce different text), not auditable (users must replay raw logs to know what the agent saw), and not reproducible.

The Approach

Iyamu's answer is a zero-model, fully deterministic pipeline:

  • Every screen transition (window focus change, URL jump, scroll event) is converted into an "event frame" by a set of rules.
  • Event frames are merged into "Activity Frames" using three constant parameters: dwell 90s / session gap 300s / flicker merge 20s.
  • An R-tree index stores all frames from 24 hours as a byte-identical binary block.
  • Key Numbers

  • 86x compression: On the author's own corpus of 51 active days (128,756 frames), a full day's raw capture compresses into a prompt block 86x smaller. The paper itself notes this comes mainly from removing duplicate pixels and merging time windows — an apples-to-oranges comparison with LLM summaries, which never stored raw pixels anyway.
  • 68ms compilation: Running 128,756 frames through the full pipeline takes 68ms. This is the most direct benefit of determinism + zero-model design: hot caching, offline recomputation, and CI integration become trivial.
  • 98.4% QA accuracy: Handle with care. The experiment uses the author's own QA task (100 questions over 51 days of screen logs), comparing answers from compressed activity frames vs. raw logs against ground truth. The baseline is an LLM summary (GPT-4o daily activity summaries) at 66-80%. On the author's own dataset, deterministic compression wins decisively — but the conclusion's scope is narrow.
  • Positioning

    Unlike approaches that "let an LLM look at the screen" (OpenAI's Operator, Anthropic's Computer Use, Apple's ScreenKit), Activity Frames never lets the LLM see the screen — the LLM only reads the compiled frames. LLM cost and latency drop to nearly zero; the tradeoff is that the modeling of "what happened on screen" is hand-crafted, and the author admits limited coverage of video frames, audio, and cross-window drag scenarios.

    Notably, the paper includes a formal proof of cacheability: as long as the day's event sequence is unchanged, the frame output is byte-identical. This sounds obvious, but it does not hold for LLM summaries. Making frames cacheable means they can be shared across multiple agents, persisted, and differentially updated.

    Links

  • arXiv abs: https://arxiv.org/abs/2608.05784
  • arXiv HTML full text: https://arxiv.org/html/2608.05784
  • GitHub repo: https://github.com/nossa-iyamu/activity-frames
  • PyPI package: https://pypi.org/project/activity-frames/
  • Author's write-up: https://www.passions.com/@nossa/55-days-of-screen-memory
  • HuggingFace Daily Papers: https://huggingface.co/papers/2608.05784
  • The implementation is open-sourced as activity-frames (GitHub and PyPI, v0.1.0). The author disclosed a discrepancy between 55 days of personal use and the paper's 51 days (final cleanup days excluded from the statistical window).

    Assessment

    This is a specimen of "de-AI-ifying" agent long-term context engineering. The core claim is not that LLM summaries are bad, but that some tasks never needed an LLM at all. If the approach validates, agent memory architecture could evolve from the current "vector store + LLM summarization" binary into a ternary structure: "deterministic frame + selective LLM enrichment" — a development that would put projects like MemGPT in an awkward position.

    Limitations

  • The evaluation dataset is author-constructed; 100 questions cover limited application scenarios
  • No comparison with current SOTA RAG / long-context systems (e.g., MemGPT, Letta, Anthropic Context Retrieval)
  • Single subject, single device; multi-device sync and cross-device dedup not discussed
  • The three constants (dwell / session gap / flicker merge) are hardcoded; generalizability across people and apps is unknown
  • Semantic understanding of screen content still requires an LLM; Activity Frames only solves the "activity stream," not the "content stream"
  • Privacy: local-first is the default, but cloud-edge boundaries are not discussed
  • No comparison with Apple Intelligence's screen-aware privacy mechanisms or OpenAI Operator's safety boundaries

Tags

#agent-memory#deterministic-pipelines#screen-activity#llm-context#caching#reproducibility#arxiv#activity-frames

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178603065