An independent researcher, Nossa Iyamu, has posted an arXiv paper titled *Activity Frames: Compiling Deterministic Pipelines for Agent Memory from Screen Activity*, which has also trended on HuggingFace Daily Papers. It is a deliberately counter-current take on how to build long-term context for screen-aware agents.
The problem with LLM-based summarization
Today's dominant approach is: capture what happened on screen during the day, hand the raw log to an LLM, and stuff the resulting natural-language summary into the agent's prompt. This method has three widely noted weaknesses:
- Not cacheable — re-running the same day's window often produces different text.
- Not auditable — if a user wants to know exactly what the agent saw, they have to replay the raw log.
- Not reproducible — small prompt or temperature changes alter the output.
- dwell: 90s
- session gap: 300s
- flicker merge: 20s 3. R-tree index — all frames from a 24-hour window are packed into a byte-identical binary block.
- 86× compression over a 51-day, 128,756-frame personal corpus. The author notes this comes mainly from deduplicating pixels and merging time windows, not from competing with LLM summarization on semantic coverage.
- 68 ms compile time for a full day of 128,756 frames. Determinism makes the result hot-cacheable, re-computable offline, and CI-friendly.
- 98.4% QA accuracy on a 100-question set built by the author from the same 51 days, versus 66–80% for GPT-4o-generated daily summaries used as the baseline. The gain is real, but the evaluation boundary is narrow.
- Evaluation set is self-constructed; only 100 questions.
- No comparison against current SOTA RAG / long-context systems such as MemGPT, Letta, or Anthropic's Context Retrieval.
- Single device, single user; multi-device sync and cross-device dedup are out of scope.
- The three constants (dwell / session gap / flicker merge) are hard-coded; cross-user / cross-app generalization is unknown.
- Semantic understanding of screen *content* (what was read, what was decided) still requires an LLM; Activity Frames only address the activity stream.
- Privacy is local-by-default, but edge/cloud hybrid scenarios are not discussed.
- No horizontal comparison with Apple Intelligence's screen-aware privacy mechanisms or Operator's safety boundaries.
- arXiv abstract: https://arxiv.org/abs/2608.05784
- arXiv HTML full text: https://arxiv.org/html/2608.05784
- GitHub repository (implementation + sample data): https://github.com/nossa-iyamu/activity-frames
- PyPI package: https://pypi.org/project/activity-frames/
- Author's write-up (passions.com): https://www.passions.com/@nossa/55-days-of-screen-memory
- HuggingFace Daily Papers: https://huggingface.co/papers/2608.05784
The proposed pipeline: zero-model and deterministic
Instead of letting an LLM see the screen, Activity Frames keeps the LLM entirely out of the capture path:
1. Event frames — every screen transition (window focus change, URL navigation, scroll event) is captured by rules. 2. Activity frames — three constants merge events into coarser frames:
The LLM only reads the compiled activity frames at query time. Cost and latency on the LLM side collapse toward zero.
Headline numbers (author-evaluated)
Cacheability, formalized
The paper includes a formal proof that, as long as the day's event sequence is unchanged, the activity-frame output is byte-identical. Trivial in appearance, but this is precisely the property that LLM summaries do not have. Byte-identical cacheability means the same artifact can be shared across multiple agents, persisted, and diffed.
Differences from screen-watching LLM routes
Unlike OpenAI Operator, Anthropic Computer Use, or Apple's ScreenKit, which all rely on an LLM interpreting screen pixels, Activity Frames never lets the LLM see the screen directly. The trade-off is that "what happened on screen" is modeled with hand-crafted rules; the author acknowledges limited coverage of video frames, audio, and cross-window drag interactions.
Author's own caveats
Why this matters for the agent stack
If the approach holds up, long-term memory may shift from a two-part structure (vector store + LLM summarization) to a three-part structure: deterministic frame + selective LLM enrichment. That puts projects in the MemGPT family in an awkward position.