Your Prompt Should Do More: How Retrieval Instructions Affect Query Embeddings in Asymmetric Retrieval

Paper Overview

• Field: NLP
• Authors: Amanda Myntti, Jenna Kanerva, Veronika Laippala, Filip Ginter
• Published: 2026-10-07
• arXiv: 2610.10508

Summary

Prompted embedding models have recently received increasing attention, particularly for retrieval, where detailed retrieval instructions are provided as part of the retrieval prompt. Several new datasets and studies have examined this setting, showing that current embedding models often struggle to follow such instructions reliably.

This paper studies the mechanism of how instructions actually affect the representations of retrieval queries in asymmetric retrieval tasks. The authors show that models can fail to follow even simple task instructions when query-side distractors are included in the evaluation. They hypothesize that this behavior is driven by the training setup of current embedding models and their evaluation methodology, and demonstrate that fine-tuning with added query-side distractors leads to significant improvements, while having little impact on other tasks.

Key Takeaways

• Instructions in retrieval prompts may not reliably steer query embeddings under realistic evaluation conditions.
• Including query-side distractors in evaluation reveals failures in instruction following.
• Fine-tuning with query-side distractors substantially improves instruction adherence without hurting other tasks.