Better to Ask in English: Cross-Lingual Evaluation of LLMs for Healthcare Queries (WWW 2024)
Overview
This post indexes a WWW 2024 academic paper:
- Title: Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries
- Venue: The Web Conference (WWW) 2024
- DOI / Link: https://dl.acm.org/doi/abs/10.1145/3589334.3645643
- Type: Academic paper (Verticals section)
- Unified framing of retrievers, re-rankers, planners, generators, and feedback mechanisms as composable components.
- Agentic paradigms that treat the number and strategy of retrieval steps as learnable, rather than fixed.
- Open problems including evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual/multimodal generalization — the last being exactly where this paper contributes.
- IR: Information Retrieval
- RAG: Retrieval-Augmented Generation
- LTR: Learning to Rank
- nDCG: Normalized Discounted Cumulative Gain
- Agentic Search: search modeled as sequential decision-making with tool calls
- Gen-IR: Generative Information Retrieval
- Original paper: https://dl.acm.org/doi/abs/10.1145/3589334.3645643
The paper addresses a critical question for deploying LLMs in healthcare: do large language models answer medical queries with the same quality and safety when asked in non-English languages compared to English? Given that most training data and evaluation resources are English-centric, cross-lingual gaps in high-stakes domains like healthcare carry real safety implications.
Context in IR and LLM Systems
The entry situates the paper within modern search, recommendation, and personalization systems, where traditional pipelines that separate retrieval, ranking, and generation struggle to meet LLM-era demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge. Key themes discussed:
Experimental and Evaluation Perspective
Typical evaluation protocols in this space involve datasets such as MS MARCO, BEIR, and Natural Questions, with metrics like nDCG@10, MRR, Recall@k, task success rate, latency, and token cost, benchmarked against BM25, dense retrieval, cross-encoder re-ranking, and retrieval-free LLM baselines. For this specific paper, quantitative results should be taken from the original PDF; this entry is compiled from the abstract and public metadata.
Takeaways for Practitioners
1. Architecture: cascaded retrieval + re-ranking + generation remains mainstream, with agentic control emerging. 2. Data: high-quality instruction data and interaction logs matter; synthetic data risks leakage and distribution shift. English-centric data raises cross-lingual generalization concerns. 3. Evaluation: offline metrics diverge from online satisfaction; LLM-as-judge needs cross-validation with human review. 4. Deployment: latency, cost, interpretability, and safety (e.g., retrieval poisoning, output filtering) are hard constraints for healthcare applications.
Engineering checklist
| Item | Question | Suggestion | |------|----------|------------| | Data | PII in training/index? Versioning? | Partitioned indexes, anonymization, rollback-capable embeddings | | Latency | p99 budget? Retrieval steps? | Cascades with early stopping, query caching, async re-ranking | | Quality | Do offline gains translate online? | Interleaving experiments, human audits, citation checks | | Safety | Open retrieval poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Token/GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense |