A Comprehensive Evaluation of Embedding Models and LLMs for IR and QA Across English and Italian
*Published May 2025 in Advances in Natural Language Processing and Text Mining (MDPI)*
Source: https://www.mdpi.com/2504-2289/9/5/141
Overview
This paper provides a systematic evaluation of embedding models and large language models (LLMs) for information retrieval (IR) and question answering (QA) tasks in both English and Italian. It responds to the English-centric bias of most retrieval benchmarks and examines how well modern dense retrievers and LLM-based QA pipelines transfer across languages.
Key points
- Scope: Benchmarks embedding-based retrievers and LLM-based QA systems across two languages (English and Italian), covering both retrieval quality and answer generation accuracy.
- Motivation: Traditional IR pipelines separate retrieval, ranking, and generation, which struggles to meet the demands of LLM-era natural language interaction, multi-hop reasoning, and real-time knowledge access. Most evaluation resources remain English-centric, leaving cross-lingual generalization poorly characterized.
- Methodological framing: The work follows a pipeline of query/document encoding (dense or sparse representations), core retrieval and reranking modules, and generation, considering learning strategies such as supervised fine-tuning, contrastive learning, and distillation.
- Evaluation dimensions: Typical metrics in this setting include nDCG@10, MRR, Recall@k, Hit@k, and task success rates, compared against baselines such as BM25, dense retrieval, cross-encoder reranking, and retrieval-free LLMs.
- CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents (arXiv:2504.16264)
- Cl2cm: Improving Cross-Lingual Cross-Modal Retrieval
- Cross-Lingual Cross-Modal Retrieval With Noise-Robust Fine-Tuning (IEEE)
- Evaluating Large Language Models for Cross-Lingual Retrieval (arXiv:2509.14749)
- Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval (MM 2024, doi:10.1145/3664647.3680886)
- The Cross-Lingual Cost: Retrieval Biases in RAG over Arabic-English Corpora (arXiv:2507.07543)
Insights for search, recommendation, and personalization
1. Architecture: Cascaded retrieval + reranking + generation remains the dominant pattern, while agentic paradigms increasingly treat retrieval strategy itself as learnable. 2. Data: High-quality instruction data and click/session logs are critical; synthetic data requires safeguards against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online user satisfaction is widening; LLM-as-judge approaches should be cross-validated with human evaluation. 4. Deployment: Latency, cost, interpretability, and safety are hard constraints in production systems and cannot be sacrificed for benchmark gains.
Limitations and future directions
Potential limitations include experiment scale constrained by compute budgets, benchmark mismatch with real user distributions, and unknown cross-lingual generalization beyond the studied language pair. Future work may explore more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation systems.