Introduction: Does Your AI Really "Know" You?
Imagine a pair of future smart glasses that accompanies you all day—recording your visual memories: the newspaper at breakfast, the coffee shop you glance at, the drawer where you drop your keys. One day you ask it: "Where did I put my keys yesterday?" It answers: "In the blue ceramic bowl on the living room coffee table." "How do you know?" "I saw it."
This sci-fi scenario is the vision of Personal Visual Context Learning (Personal VCL). The underlying paper asks a sharp question: how far are today's most powerful multimodal models from becoming true personal assistants?
From General to Personal: The Gap
- LLMs are like a well-read but forgetful professor: massive parametric knowledge, but no personalized memory—every conversation starts from a blank slate.
- Multimodal models can "see" images, but they never link those images to *you* or form memories of your habits (e.g., where you usually put your blue cup).
- In-context learning is like cramming before an exam: examples in the prompt help, but context length is limited and the model remains stateless, accumulating no long-term memory.
- People: recognizing family, friends, colleagues
- Objects: tracking personal belongings and their locations
- Behaviors: understanding daily habits and activity patterns
- The Agentic Context Bank substantially outperforms standard in-context prompting across all tasks.
- Gains are consistent across GPT-4V, Gemini, and open-source models, indicating a model-agnostic memory architecture.
- It is an inference-time solution requiring no retraining, so it can be deployed on existing LMMs without costly training.
- What does it mean to "know you"? Current AI reaches at most surface preferences; Personal VCL targets deep knowledge built through daily observation rather than form-filling.
- The privacy paradox: better service requires deeper knowledge, which raises privacy risk. The design mitigates this with local memory storage, selective retrieval, and support for forgetting (marking memories as temporary or deletable).
- Quoting Feynman: "What I cannot create, I do not understand." Building an AI that truly understands you first requires decomposing "understanding you" into tractable subtasks: memory, retrieval, reasoning, and association.
- Xue, Z., Baid, A., Kim, S., Luo, M., & Grauman, K. (2026). Personal Visual Context Learning in Large Multimodal Models. arXiv preprint.
- Brown, T., et al. (2020). Language Models are Few-Shot Learners. NeurIPS.
- OpenAI. (2023). GPT-4V(ision) System Card.
- Google DeepMind. (2023). Gemini: A Family of Highly Capable Multimodal Models.
Three Key Findings on Personal Visual Contexts
1. Context utilization gap: even when relevant visual evidence is placed in the prompt, models fail to exploit it effectively. 2. Multi-image aggregation failure: real-world reasoning requires integrating evidence across many images (seeing the keys, recognizing the coffee table, relating them spatially)—current models perform poorly here. 3. Missing personalized reasoning: general LMMs can answer "what is this?" but not "what does this mean for you?"
The Agentic Context Bank: A Curator for Personal Memory
The paper's proposed framework acts like a curator of a private museum of your visual memories, with three components:
1. Self-Refining Memory Bank: memories are structured by categories (people, objects, scenes, actions) with cross-references, and are continuously reorganized, compressed, and linked as new memories arrive. 2. Query-Adaptive Evidence Selection: given a query, only the most relevant evidence is retrieved—like a search engine's re-ranking stage—rather than dumping the entire memory into the prompt. 3. Dynamic Evidence Chain Construction: answers come with an explicit reasoning chain across time-stamped visual evidence (e.g., keys on the table at 23:15, gone by 07:30, picked up before leaving at 07:35).
Experimental Validation
The authors introduce Personal-VCL-Bench, covering three dimensions of the personal visual world:
Key results: