A Comprehensive Evaluation of Embedding Models and LLMs for IR and QA Across English and Italian (May 2025)

• Source: https://www.mdpi.com/2504-2289/9/5/141
• Published: May 2025, *Advances in Natural Language Processing and Text Mining*
• Topic areas: Multilingual IR, embedding models, LLMs, retrieval-augmented generation (RAG)

Key points

• The paper systematically evaluates embedding models and LLMs for information retrieval (IR) and question answering (QA) across English and Italian, filling a gap in bilingual evaluation of retrieval-augmented systems.
• It situates itself in the LLM-era shift of search system design, where the classical funnel of recall → ranking → generation now must account for inference budgets and action spaces (whether to retrieve, how often, and which tools to call).
• It offers a unified view that decomposes the pipeline into representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms, making components easier to compare and to engineer.

Background and motivation

Neural information retrieval has evolved from BM25 to BERT cross-encoders, bi-encoder dense retrieval, late interaction models, generative retrieval, and LLM-based agents. Each generation balances an efficiency–effectiveness–maintainability triangle:

• Dense retrieval enables millisecond-level recall via approximate nearest-neighbor search but is sensitive to domain shift and long-tail queries.
• Cross-encoders are accurate but cannot precompute document representations.
• RAG and agentic search extend external knowledge access from one-shot retrieval into an iterative, verifiable, plannable process.

The paper addresses how these components behave across two languages, where cross-lingual generalization of embedding quality is a known concern.

Evaluation methodology

Typical benchmarks and metrics covered in this line of evaluation:

• Datasets: MS MARCO, BEIR, Natural Questions, and domain-specific corpora.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, task success rate, latency, and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, and commercial search APIs.
• Ablations: contribution of retrieval steps, reranking depth, and training data scale.

Specific numerical results should be verified against the original PDF; the experimental design logic here is reconstructed from the abstract and public metadata.

Insights for search, recommendation, and personalization

1. Architecture: cascaded retrieve → rerank → generate remains mainstream, but agentic paradigms are making retrieval count and strategy themselves learnable. 2. Data: high-quality instruction data and click/session logs matter as much as model choice; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human assessment. 4. Deployment: latency, cost, explainability, and safety are hard production constraints — optimizing academic benchmarks alone is insufficient.

Cross-lingual relevance

The English–Italian focus connects this work to broader cross-lingual retrieval research, including:

• CLIRudit: Cross-Lingual Information Retrieval of Scientific Documents
• Cross-lingual cross-modal retrieval with noise-robust fine-tuning
• Evaluating Large Language Models for Cross-Lingual Retrieval
• Retrieval biases in RAG over Arabic–English corpora ("The Cross-Lingual Cost")

Limitations and open problems

• Experimental scale may be constrained by GPU budgets; benchmarks may not match real user distributions.
• English-centric data limits certainty about cross-lingual generalization.
• Agentic systems operating over the open web carry security risks.
• Future directions: more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation.

Engineering checklist

| Check | Question | Recommendation | |-------|----------|----------------| | Data | PII in training/index? Versioning? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stop, cache hot queries, async reranking | | Quality | Does offline gain translate to online satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Poisoning/bias from open retrieval? | Source whitelisting, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Route to smaller models, distillation, hybrid sparse+dense |

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |