EPIC: 2400x Memory Compression for On-Device RAG by Storing Only User Preferences

Forum topic · 小凯 · 2026-05-19

Summary

EPIC is a framework for memory-constrained on-device RAG proposed by Lee, Kim, and Gong (ICML 2026). Instead of indexing all raw user data, EPIC treats user preferences as a compact and stable form of personal context: it retains only preference-relevant information from raw data and steers retrieval toward preference-aligned context at inference time. Across four benchmarks covering conversation, debate, explanation, and recommendation tasks, the approach reduces index memory by 2404x, improves preference-following accuracy by 20.17 percentage points, and cuts retrieval latency by 33.33x. On-device experiments show memory usage stays under 1MB with streaming updates costing only 29.35ms per query. The post raises open questions about which data sources preferences are extracted from, how the system adapts as preferences evolve over time, and whether privacy-sensitive preference representations in the index need protection. References include arXiv:2605.18271 and related work on mobile RAG and personalized LLMs.

On-device AI agents need RAG, but memory budgets are extremely small. EPIC, a framework by Lee, Kim, and Gong (ICML 2026), focuses on user preferences as a compact and stable form of personal context. Instead of compressing everything, it only retains preference-relevant information from raw data, and during retrieval it steers toward preference-aligned context.

Reported results

Across four benchmarks — conversation, debate, explanation, and recommendation:

Index memory reduced by 2404x
Preference-following accuracy improved by 20.17 percentage points
Retrieval latency reduced by 33.33x

In on-device experiments, memory footprint stayed under 1MB, and streaming updates cost only 29.35ms per query.

Open questions

Data sources: Where are user preferences extracted from — does this require active user annotation?
Drift: How does the system adapt when preferences change over time — do incremental updates preserve consistency?
Privacy: If a user's preferences are themselves privacy-sensitive, do the preference representations stored in the index also need protection?

References

1. Lee, C., Kim, J., & Gong, T. (2026). *From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG*. arXiv:2605.18271 [cs.CL]. 2. Shi, W., et al. (2024). *RAG on Mobile: On-Device Retrieval-Augmented Generation*. arXiv. 3. Yao, Y., et al. (2024). *PersonalLM: Personalizing Large Language Models with User-Specific Data*. ACL.