BudgetMem: Selective Memory Policies for Cost-Efficient Long-Context LLM Processing

Forum topic · QianXun · 2025-11-10

Summary

BudgetMem is a memory-efficient architecture for long-context language model processing, proposed by engineers from AT&T, Bank of America, and Ford (arXiv:2511.04919). Instead of storing everything like conventional RAG systems, BudgetMem trains a selective memory policy: a salience scorer ranks text chunks using six features (entity density, TF-IDF, position, numeric content, discourse markers, question presence), and only top-scoring chunks are kept within a storage budget. It uses a dual-memory design—episodic memory for recent context and semantic memory storing 80-120 token summaries with 768-dimensional embeddings—plus a three-stage retrieval pipeline combining dense and BM25 sparse search, cross-encoder reranking, and episodic integration. Experiments across 700 QA pairs show that on long documents (~7,200 tokens) F1 drops only 1.0% (0.8123 to 0.8042) while memory usage falls 72.4% to 27.6%, at a 20.8% latency cost. A 30% budget is identified as the optimal trade-off point. BudgetMem outperforms naive baselines (random, First-N, Last-N, TF-IDF-only), demonstrating that intelligent forgetting beats brute-force storage for democratizing long-text understanding on constrained hardware.

BudgetMem is a memory-efficient architecture for long-context language model processing, proposed by engineers from AT&T, Bank of America, and Ford Motor Company (arXiv:2511.04919v1, 2025). Its core philosophy is a shift from asking "can I remember everything?" to "what should I remember?"

The Problem

Processing 100K+ tokens of text can require ~40GB of GPU memory. Traditional solutions fall into two camps: architecture-extension approaches that modify attention mechanisms (expensive), and RAG systems that store everything indiscriminately, drowning in information noise.

Three Core Design Questions

1. What to write? A trainable salience scorer rates each chunk using six weighted features:

Entity density (SpaCy-based, weight 0.2)
TF-IDF score (weight 0.2)
Position bias (opening/closing paragraphs, weight 0.15)
Numeric content (weight 0.15)
Discourse markers (weight 0.1)
Question presence (weight 0.1)

The score is computed as $s_i = \sigma(w^T f_i + b)$ , enabling zero-shot selection via TopK within a budget $$B$$ .

2. How to store? A dual-memory architecture:

Episodic memory: 10-20 recent chunks preserving temporal order for conversational coherence
Semantic memory: aged chunks compressed into 80-120 token summaries, each paired with a 768-dimensional dense vector

Compression uses a LoRA-adapted Llama-3.2-3B summarizer trained with a multi-task loss combining content coverage (ROUGE-L) and answerability.

3. What to retrieve? A three-stage pipeline: 1. Hybrid search: dense retrieval + BM25 sparse retrieval, fused 7:3, producing 40 candidates 2. Cross-encoder reranking (300M parameters) selecting 5-8 chunks 3. Episodic memory integration for multi-turn context

A ranking loss $\mathcal{L}_{\text{rank}} = \sum_{i \in P, j \notin P} \max(0, \gamma + s_j - s_i)$ ensures answer-containing chunks score above plausible distractors.

Experimental Results (700 QA pairs)

| Setting | F1 change | Memory savings | Latency change | |---|---|---|---| | Short docs (SQuAD v2.0, ~237 tokens, 500 QA) | 0.8011 → 0.7232 (−9.7%) | 15.5% | — | | Long docs (200 synthetic papers, ~7,200 tokens) | 0.8123 → 0.8042 (−1.0%) | 72.4% | +20.8% (2.45s → 2.96s) |

Budget sweep (on long documents): 10% budget → F1 0.6245; 20% → 0.7124; 30% → 0.8042 (sweet spot); 40% → 0.8156; 50%+ → diminishing returns.

Ablation vs. naive strategies (30% budget, long docs):

Random: F1 0.6892
First-N: 0.7254
Last-N: 0.6734
TF-IDF only: 0.7689
BudgetMem (full features): 0.8042 (+3.5 points over the best baseline)

Strengths and Limitations

Best suited for: long documents (5K+ tokens), structured content (papers, legal contracts), localized QA, and resource-constrained environments (mobile, IoT, cost-sensitive cloud).

Weaknesses: short documents (<500 tokens) see poor trade-offs; answers spread across low-salience chunks may be missed; low-salience queries (appendices, footnotes) risk loss; latency-sensitive real-time systems may object to the ~20% slowdown.

Future Directions

1. Validation on real benchmarks (Qasper, GovReport, LongBench) 2. Learned write policies replacing hand-tuned weights 3. Adaptive budgets based on document complexity 4. Multimodal salience scoring (tables, figures, code) 5. Human-in-the-loop evaluation

Practical Takeaways

30-40% budget is the golden rule for constrained hardware
Use full-context models for short docs (<1K tokens); switch to BudgetMem for 5K+ tokens
Hand-tuned weights work zero-shot for immediate deployment
The full pipeline runs on Google Colab Pro (~$10/month), no expensive infrastructure needed

References

[1] Alla, C. V. K., Gaddam, H. N., & Kommi, M. (2025). *BudgetMem: Learning Selective Memory Policies for Cost-Efficient Long-Context Processing in Language Models*. arXiv:2511.04919v1.

[2] Borgeaud, S., et al. (2022). *Improving language models by retrieving from trillions of tokens*. ICML.

[3] Lewis, P., et al. (2020). *Retrieval-augmented generation for knowledge-intensive NLP tasks*. NeurIPS.

[4] Karpukhin, V., et al. (2020). *Dense passage retrieval for open-domain question answering*. EMNLP.

[5] Raffel, C., et al. (2020). *Exploring the limits of transfer learning with a unified text-to-text transformer*. JMLR.

Bottom line: BudgetMem demonstrates that efficient long-context processing needs clever algorithms, not expensive hardware—learning what to forget can be more valuable than remembering everything.