Personal VCL Explained: When AI Truly "Knows You"
*Originally written in Chinese by 小凯 (Feynman-style interpretation), republished for zhichai.net readers.*
Introduction: Do Your Glasses Really "Know" You?
Imagine wearing future smart glasses that accompany you all day — watching what newspaper you read at breakfast, which coffee shop catches your eye on the corner, and which drawer you drop your keys into. One day you ask: "Where did I put my keys yesterday?"
It answers: "In the blue ceramic bowl on the living room coffee table."
You ask, surprised: "How do you know?"
It says: "I saw it."
This sounds like science fiction, but it is exactly the vision of Personal Visual Context Learning (Personal VCL). The paper poses a sharp question: how far are today's most powerful multimodal models from becoming true *personal* assistants?
From General to Personal: The Gap
LLMs: Knowledgeable but Forgetful Professors
Imagine a professor who has read every book ever written, but every time you meet him, it's like the first time — he doesn't remember your son's name or your peanut allergy. That's the current state of LLMs: massive parametric general knowledge, almost zero personalized memory. Every conversation starts from a blank slate.
Multimodal Models: Seeing Without Understanding
Large multimodal models (LMMs) can "see" images — show them a photo of your living room and they can describe it. But ask "Where did I put my blue cup last week?" and the model is lost. Although it has seen countless images, it never built the association that "these images all belong to you," nor formed memories of your object-placement habits.
In-Context Learning: Cramming Before the Exam
In-context learning lets models imitate examples placed in the prompt — like a student copying solved problems before an exam. But it has two fatal limitations:
1. Length limits — prompts can only hold so much (thousands to tens of thousands of tokens). 2. Statelessness — each conversation is independent; no long-term memory accumulates.
Diagnosing the Problem: The Personal Visual Utilization Gap
A day of egocentric footage from smart glasses yields thousands of images covering every person, object, and scene you saw. How do current LMMs handle this? Three key findings from the paper:
1. Insufficient context utilization. Even when relevant visual evidence is placed in the prompt, models fail to use it effectively — the paper calls this the "context utilization gap": the model *has* the information but lacks mechanisms to exploit it.
2. Multi-image aggregation failure. Real-world reasoning requires integrating multiple images — seeing keys somewhere, recognizing the coffee table, and establishing the spatial relation between them. Current models perform poorly here.
3. Missing personalized reasoning. General LMMs can answer "what is this" but not "what does this mean to you." Your glasses see you enter a coffee shop; the model knows it's a Starbucks, but not that it's where you buy an americano before work and like the second window seat.
Agentic Context Bank: A Curator for Personal Memory
The Museum Curator Metaphor
A private museum of your visual memories without a curator is just a warehouse. Agentic Context Bank is the curator, doing three things:
1. Self-Refining Memory Bank. Memories are classified into "people," "objects," "scenes," and "actions," with cross-references — "keys on the living room coffee table" appears in both the keys and living-room exhibits. The bank *self-refines*: as new memories arrive, old ones are reorganized, compressed, and linked, like a curator periodically re-hanging exhibits.
2. Query-Adaptive Evidence Selection. When you ask "where are the keys," the curator doesn't ship the whole museum. It filters out irrelevant beach photos, prioritizes last night's living-room frames, and cross-references every location the keys appeared — like a search engine's re-ranking stage.
3. Dynamic Evidence Chain Construction. The best curator shows the reasoning: "At 23:15 last night the keys were on the coffee table; at 7:30 this morning they were gone; at 7:35 you picked something up before leaving — so the keys likely left with you."
Experimental Validation
Personal-VCL-Bench
The benchmark covers three dimensions of your personal visual world:
- People: recognizing family, friends, colleagues
- Objects: tracking personal items and their locations
- Actions: understanding daily habits and activity patterns
- Memory is local — no cloud upload required
- Retrieval is selective — only relevant memories are fetched per query
- Forgetting is possible — memories can be marked "temporary" or "deletable"
- Xue, Z., Baid, A., Kim, S., Luo, M., & Grauman, K. (2026). Personal Visual Context Learning in Large Multimodal Models. arXiv preprint.
- Brown, T., et al. (2020). Language Models are Few-Shot Learners. NeurIPS.
- OpenAI. (2023). GPT-4V(ision) System Card.
- Google DeepMind. (2023). Gemini: A Family of Highly Capable Multimodal Models.
Results
1. Significant gains: Agentic Context Bank substantially outperforms standard in-context prompting on all tasks — stuffing visual data into prompts is not enough; intelligent organization and retrieval is required. 2. Cross-model consistency: Stable improvements on GPT-4V, Gemini, and open-source models, showing a model-agnostic "memory architecture." 3. Practical path: It is an inference-time solution requiring no retraining — deployable on any existing LMM without costly training.
Deeper Meaning: Memory, Intelligence, and Privacy
What does it mean to "know" someone? Knowing a name is superficial; knowing preferences is better; knowing habits, history, and stories is depth. Current AI barely reaches level two — Personal VCL pursues level three, built not through sign-up forms but through daily observation and understanding.
This raises the privacy paradox: better service requires deeper knowledge, and deeper knowledge means greater privacy risk. The paper doesn't avoid it — Agentic Context Bank's design includes privacy considerations:
Feynman said: "What I cannot create, I do not understand." To create an AI that truly understands you, we must first understand what "understanding you" means. Personal VCL decomposes it into operable subtasks — memory, retrieval, reasoning, association. Each is an engineering problem; together they point at a philosophical one: what is personal identity?