Shared Memory for AI Agents: From Isolated Ghosts to a Digital Hive

Key points

• Shared-memory architecture: Two agents (chat_agent and research_agent) are built on the same PostgresDb instance and use a common user_id (e.g., john_doe@example.com) with enable_user_memories=True, turning isolated LLMs into a coordinated multi-agent system that remembers one continuous "digital self."
• PostgreSQL as external hippocampus: A local PostgresDb (postgresql+psycopg://ai:ai@localhost:5532/ai) stores vector embeddings of user statements and agent interactions. This overcomes the "eternal present" limitation of LLMs by giving them persistent, queryable long-term memory beyond the context window.
• From text to vector to meaning: Conversations are encoded by models such as text-embedding-ada-002 into 1536-dimensional vectors. Retrieval uses cosine similarity combined with recency and frequency weights:

$$ \text{Memory Relevance} = \alpha \cdot \text{Semantic Similarity} + \beta \cdot \text{Recency} + \gamma \cdot \text{Frequency} $$

Approximate Nearest Neighbor (ANN) algorithms such as HNSW or Faiss reduce search to roughly O(log n), enabling millisecond responses over millions of memories.

• Hive intelligence through specialization: Agents differ only in tools and description. The chat agent has no external tools; the research agent uses DuckDuckGoTools(cache_results=True) for live search. Shared memory acts like a pheromone network, so user preferences learned in conversation immediately influence retrieval-augmented research answers, producing emergent cross-pollination between agents.
• Functional modularity vs. identity unity: This design solves three single-agent pain points: tool overload, context pollution, and role confusion. The framework extends naturally to code_agent (code execution, GitHub) and design_agent (Figma, image generation), forming an AI-native team that shares a whiteboard of project knowledge.
• Selective prompt engineering: Retrieved memories are injected into a structured prompt template, and a system-level memory attention score (softmax of similarity × recency) decides which fragments are included. A dual-layer model distinguishes working memory (current context) from long-term memory (PostgreSQL plus cached search results), balancing latency, token cost, and relevance.
• Privacy and local-first design: Local PostgreSQL deployment keeps memory on-domain, giving users data sovereignty and physical isolation between accounts. Open problems remain: memory decay, conflicting user statements, explainability of memory-driven answers, and alignment with regulations like GDPR's "right to be forgotten."
• Engineering tradeoffs at scale: Each query adds roughly 50 ms (embedding) + 10 ms (vector search) + 5 ms (prompt reranking) ≈ 65 ms. An active user generates ~50 memory entries per day (~10 KB each), or roughly 180 MB/year; 1 million users would require ~180 TB. Mitigations include Redis caching, dedicated vector databases such as Pinecone or Weaviate, async embedding, tiered storage, smaller embeddings like all-MiniLM-L6-v2 (384 dims), and time/frequency-based eviction.
• From memory graph to social memory: Future architectures may evolve into native MemGraphs that traverse multi-hop relations (e.g., quantum cryptography → outdoor communication security), and into team-wide enable_team_memories databases that capture organizational knowledge for onboarding, meeting continuity, and expert departure resilience.
• Ethics and agency: As memory becomes core, new rights frameworks may be needed: memory auditability, editing, automated forgetting, portability between platforms, and inheritance rights. Allowing agents to write self-reflective memories blurs the line between tool and subject, raising questions about AI agency, personality drift across model versions, and the path toward AGI through continuous learning and contextual understanding.
• Illustrative scenario: A user who mentions hiking on Monday and quantum cryptography on Wednesday finds that the chat agent recommends quantum-sensor altimeters on Friday and the research agent tailors its 1000 km QKD breakthrough summary to outdoor security, demonstrating how shared memory produces 1+1>2 personalization across a week of disjoint sessions.

References

1. OpenAI, "Text Embeddings", 2023. 2. Pinecone, "Vector Databases for Machine Learning", 2023. 3. Vaswani et al., "Attention Is All You Need", NIPS 2017. 4. Tulving, "Episodic and Semantic Memory", 1972. 5. Russell & Norvig, "Artificial Intelligence: A Modern Approach", 2021.