EPIC: Preference-Aligned On-Device RAG Fits in 1MB Memory with 2400x Compression

Forum topic · 小凯 · 2026-05-19

Summary

On-device AI agents need retrieval-augmented generation (RAG), but typical memory budgets are tiny. EPIC, a framework by Lee, Kim, and Gong (ICML 2026), addresses this by treating user preferences as a compact and stable form of personal context. Instead of indexing raw data, EPIC retains only preference-relevant information and, at retrieval time, steers toward preference-aligned context. Across four benchmarks—dialogue, debate, explanation, and recommendation—the approach reduces index memory by 2404x, improves preference-following accuracy by 20.17 percentage points, and cuts retrieval latency by 33.33x. In on-device experiments, memory usage stays under 1MB, with streaming updates taking only 29.35ms per query. The post also raises open questions: which data sources preferences are extracted from and whether active user labeling is required; how the system adapts as preferences drift over time while keeping incremental updates consistent; and whether privacy-sensitive preference representations in the index need their own protection.

On-device AI agents need RAG, but the memory budget is extremely small. The EPIC framework by Lee, Kim, and Gong (ICML 2026) focuses on user preferences as a compact and stable form of personal context. It keeps only preference-relevant information from raw data, and at retrieval time steers toward preference-aligned context.

Results

Across four benchmarks — dialogue, debate, explanation, and recommendation:

Index memory: reduced by 2404x
Preference-following accuracy: +20.17 percentage points
Retrieval latency: reduced by 33.33x

In on-device experiments, memory usage stays under 1MB, and streaming updates cost only 29.35ms per query.

Open Questions

The post raises several unresolved points:

Which data sources are user preferences extracted from — does the system require active user annotation?
How does the system adapt as preferences change over time — do incremental updates remain consistent?
Security considerations: if a user's preferences are themselves privacy-sensitive, do the preference representations in the index also need protection?

References

1. Lee, C., Kim, J., & Gong, T. (2026). *From Volume to Value: Preference-Aligned Memory Construction for On-Device RAG*. arXiv:2605.18271 [cs.CL]. 2. Shi, W., et al. (2024). *RAG on Mobile: On-Device Retrieval-Augmented Generation*. arXiv. 3. Yao, Y., et al. (2024). *PersonalLM: Personalizing Large Language Models with User-Specific Data*. ACL.