LongNAP Explained: From Reactive AI to Predicting Your Next Action

Forum topic · 小凯 · 2026-05-16

Summary

This post is a detailed Chinese-language breakdown of LongNAP (Long-Context Next Action Prediction), a Stanford-led paper (arXiv:2603.05923) that reframes proactive AI as a computable prediction task: given a user's multimodal interaction history (screenshots, I/O events), predict what they will do next. The authors built NAPsack, a passive data pipeline that continuously captures screenshots and annotates them with VLMs, collecting 1.9M screenshots and 360K action captions from 20 users over 28 days with 75% storage reduction via event-driven capture. LongNAP learns to retrieve relevant past reasoning traces from a personal memory store rather than fine-tune weights, trained with policy gradients where an LLM-as-judge scores semantic similarity between predictions and actual future actions. Results: +79% over supervised fine-tuning, +39% over prompted baselines, with 17.1% absolute alignment (26% after confidence filtering) in an unbounded action space, and +13% generalization to unseen users. The post also covers applications (powerNAP for on-device continual learning, SleepWalk combining prediction with computer-use agents), limitations (VLM annotation noise, small sample, 17.1% absolute accuracy), and the paradigm shift from tool-like to anticipatory AI assistants.

LongNAP: Long-Context Next Action Prediction — A Deep Dive

This is an English translation/summary of a detailed Chinese forum post analyzing the paper LongNAP: Long-Context Next Action Prediction.

Authors: Omar Shaikh, Valentin Teutschbein, Kanishk Gandhi, Yikun Chi, Nick Haber, Thomas Robinson, Nilam Ram, Byron Reeves, Sherry Yang, Michael S. Bernstein, Diyi Yang
Institutions: Stanford University, Hasso Plattner Institute, New York University
Paper: arXiv:2603.05923
Keywords: proactive AI, next action prediction, multimodal behavior prediction, retrieval augmentation, policy gradient, LLM-as-judge

Key points

Problem: Today's AI assistants only respond to explicitly expressed needs. They see prompts but nothing of the context that led to them. The paper turns "proactivity" into a computable task: given a multimodal interaction sequence (screenshots, clicks, sensor data), predict the user's next action — over an unbounded action space.
NAPsack data pipeline (passive supervision): Instead of asking users to label behavior, NAPsack passively observes and lets a VLM annotate afterward:
Continuous screenshots + I/O event recording
Event-driven capture (screenshots only on user actions) cuts storage by 75% (295MB → 76MB)
60-frame segments → VLM generates natural-language action captions
Scale: 20 users × 28 days, 1.9M screenshots, 1,800 hours of screen time, 360K behavior captions, ~15 seconds per event
Source: Stanford's Screenomics project (IRB-approved), covering gaming, shopping, banking, messaging, and social media
Human win-rate vs VLM annotations: 85.8% — good but imperfect, and errors propagate to downstream training
LongNAP architecture: learn to retrieve, not to fit weights:
The authors argue fine-tuning fails at *latent learning* (acquiring information not immediately relevant but useful later), is data-inefficient, and adapts slowly; full-context stuffing is noisy and bounded.
Stage 1 — Reason to Retrieve: from recent context, generate reasoning ("the user just opened a difficult paper review"), then search a memory bank of past observations and reasoning traces ("the user usually asks coauthors to split revisions").
Stage 2 — Reason to Predict: integrate retrieved traces to predict future actions over a horizon.
Feedback loop: good traces are written back to the memory bank (self-improving memory); bad predictions are penalized via policy gradient.
This is a personalized form of RAG — but instead of retrieving external knowledge, it retrieves the user's own past reasoning traces, with a *learned* retrieval policy.
Reward design: Traditional RL needs explicit reward functions. Here the reward is natural — wait to see what the user actually does, then use an LLM-as-judge to score semantic similarity between prediction and reality. This handles delayed rewards, semantic (not exact-match) evaluation, and jointly optimizes retrieval and prediction.

Results

| Comparison | Improvement | |---|---| | vs Supervised Fine-Tuning (SFT) | +79% | | vs Prompted baseline | +39% |

Absolute alignment: 17.1% of predicted trajectories align with real behavior (LLM-judge score ≥ 0.5); 26% after high-confidence filtering. Given thousands of possible actions at any moment, the author argues this is substantial — and the confidence gap suggests the model "knows what it knows."
Cross-user generalization: trained on multiple users, LongNAP beats the best baseline on unseen users by +13%, showing behavioral structure transfers across individuals.

Applications

powerNAP (online continual learning): NAPsack and LongNAP run asynchronously — data is collected, annotated, used for a single training pass, then discarded. The memory never resets and reasoning traces keep accumulating. It can run locally on-device for privacy.
SleepWalk (prediction-driven agent): LongNAP predicts the next action; an off-the-shelf computer-use agent (e.g., Anthropic's Computer Use) executes it — a closed loop of "anticipate + auto-complete."

Paradigm shift

| Generation | AI role | Interaction mode | |---|---|---| | ChatGPT/Claude | Tool | User asks, AI answers | | Computer Use Agent | Operator | User instructs, AI executes | | LongNAP → SleepWalk | Companion/steward | AI observes, anticipates, proactively completes |

The post notes that Apple Intelligence, Google Assistant, and Microsoft Copilot are all pursuing context awareness; LongNAP provides an academic baseline (data pipeline + architecture + evaluation protocol) that industry will rapidly absorb.

Limitations

| Limitation | Detail | |---|---| | Data noise | Imperfect VLM annotations; alignment gap; errors propagate into training | | LLM-judge dependency | Judge bias and variance affect both training and evaluation | | Small sample | 20 users; demographic diversity unknown | | Data vintage | 2021 Screenomics data; modern phone usage patterns differ | | No privacy attack analysis | Risks of maliciously predicting user behavior unexamined | | 17.1% absolute accuracy | In practice >80% of predictions are wrong; fallback strategies needed |

Conclusion

The post frames LongNAP as foundational work for proactive AI — a complete research framework covering task definition (NAP), data infrastructure (NAPsack), model architecture (LongNAP), training (policy gradient + LLM judge), and applications (powerNAP, SleepWalk). Its core insight — learning to retrieve beats learning weights for personalized behavior prediction — suggests future personal AI will not fine-tune huge models per user, but teach models to extract relevant context from the user's own history.

> "Truly proactive AI systems must anticipate what we will do next. This foresight demands far richer information than the sparse signals we type into our prompts — it demands reasoning over the entire context of what we see and do."

References

Shaikh, O. et al. (2026). *LongNAP: Long-Context Next Action Prediction*. arXiv:2603.05923.
Reeves, B. et al. (2021). Screenomics: A new approach to digital life. *Psychological Science*.
Gandhi, K. et al. (2023). SOTOPIA: Interactive evaluation for social intelligence in language agents. *NeurIPS*.
Chan, S. et al. (2022). Data distributional properties drive emergent in-context learning in transformers. *NeurIPS*.
Brown, T. et al. (2020). Language models are few-shot learners. *NeurIPS*.
Lampinen, A. et al. (2025). Latent learning, in-context learning, and the memory-features trade-off. *ICML*.
Zheng, L. et al. (2023). Judging LLM-as-a-judge with MT-bench and Chatbot Arena. *NeurIPS*.