Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning

Source: arXiv:2507.22729, July 2025 Authors: Benedikt Roth, Stephan Rappensperger, Tianming Qiu, Hamza Imamović, Julian Wörmann, Hao Shen Category: Embedding models

Overview

This paper addresses how large language models (LLMs) can be converted into strong text embedding models with minimal compute, combining prompt engineering and contrastive fine-tuning rather than expensive pre-training from scratch.

Background and Motivation

In large-scale search, recommendation, and personalization systems, embeddings face long-standing challenges around efficiency, scalability, and understanding user intent. Traditional pipelines separate retrieval, ranking, and generation, which struggles to meet modern demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. This work targets that intersection, aiming to advance the theory and practice of LLM-based embeddings under resource constraints.

Core Contributions

• A unified perspective on adapting decoder-only LLMs into embedding models via prompts plus contrastive learning.
• Clear decomposition of method components (representation learning, retrievers, re-rankers, training strategies) for engineering adoption.
• Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration.
• Open problems identified: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension.

Method Sketch

The typical pipeline follows four stages:

1. Input & representation — encode queries, documents, and context into dense/sparse representations or structured prompts. 2. Core modules — retrievers, re-rankers, planners, memory, and tool interfaces, chained as needed. 3. Learning strategies — supervised fine-tuning, contrastive learning, distillation, reinforcement learning, synthetic data bootstrapping. 4. Inference strategies — single-pass or iterative retrieval, parallel sub-queries, early stopping, and budget control.

Evaluation Context

Relevant benchmarks and metrics in this space include:

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, latency, and token cost.
• Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs.

Exact quantitative results should be verified against the original PDF.

Key Takeaways for Search / Rec / Personalization

1. Architecture: cascaded retrieval + re-ranking + generation remains mainstream, but agentic paradigms make the number and strategy of retrieval steps themselves learnable. 2. Data: high-quality instruction data and click/session logs matter; synthetic data must guard against leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Production: latency, cost, explainability, and safety are hard constraints — do not optimize academic benchmarks alone.

Related Entries

• The Scandinavian Embedding Benchmarks
• A Universal Framework for Compressing Embeddings in CTR Prediction (arXiv:2502.15355)
• Arctic-Embed 2.0: Multilingual Retrieval Without Compromise (arXiv:2412.04506)
• BGE M3-Embedding (arXiv:2402.03216)
• BGE-en-ICL / BGE-ICL: Making Text Embedders Few-Shot Learners (arXiv:2409.15700)
• Beyond Benchmarks: Evaluating Embedding Model Similarity for Retrieval (arXiv:2407.08275)

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |

References

• Original paper: Resource-Efficient Adaptation of Large Language Models for Text Embeddings via Prompt Engineering and Contrastive Fine-tuning, arXiv, July 2025.