Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries (WWW 2024)
This is an English translation and structured summary of a Chinese forum post covering the WWW 2024 paper "Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries."
- Source: https://dl.acm.org/doi/abs/10.1145/3589334.3645643
- Venue: The Web Conference (WWW) 2024
- Category: Verticals (LLMs applied to a specific domain)
- A unified perspective on evaluating LLMs for cross-lingual healthcare question answering.
- A clear decomposition of method components (representation, retriever, reranker, generator, feedback mechanisms) useful for engineering practice.
- Reproducible benchmark and protocol design that lowers the entry cost for follow-up research.
- Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration.
- Explicitly listed open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual/multilingual generalization.
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, non-retrieval LLMs, commercial search APIs.
One-line takeaway
The paper investigates whether LLMs answer healthcare questions reliably when the queries are asked in languages other than English — the title suggests that English queries often yield better results.
Research Background and Motivation
The forum post situates the paper within the challenges facing large-scale search, recommendation, and personalization systems in the LLM era: efficiency, scalability, and understanding user intent. Traditional pipeline approaches that separate retrieval, ranking, and generation struggle to meet modern demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge. This paper addresses a vertical intersection of these concerns — healthcare — and the cross-lingual dimension of LLM reliability.
Core Contributions (as framed by the post)
Method / System Design
Following the post's four-step framing:
1. Input and representation — encoding queries, documents, and user context into dense/sparse representations or structured prompts. 2. Core modules — retrievers, rerankers, planners, memory modules, and tool interfaces, composed serially or in parallel. 3. Learning strategies — supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and synthetic data bootstrapping. 4. Inference strategies — single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control.
Evaluation
Typical datasets and metrics discussed in this research space include:
Key Insights for Search / Recommendation / Personalization
1. Architecture: Cascaded retrieval + reranking + generation remains mainstream, but agentic paradigms are making "whether and how many times to retrieve" itself a learnable decision. 2. Data: High-quality instruction data and click/session logs matter as much as model scale; synthetic data risks knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation against human evaluation. 4. Production: Latency, cost, interpretability, and safety are hard constraints for real deployments — academic benchmarks alone are insufficient.
Limitations and Open Problems
Potential limitations flagged in the post include experiment scale constrained by GPU budgets, mismatch between benchmarks and real user distributions, English-centric training data leading to unknown cross-lingual generalization (the paper's central theme), and safety risks of agent systems on the open web. Future directions include more efficient test-time compute allocation, deeper integration with knowledge graphs and structured databases, and causal/fairness constraints for recommendation systems.
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking-quality metric | | Agentic Search | Modeling search as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |
Actionable Advice for Readers
1. Researchers: Replicate core comparisons; check whether statistical significance and compute cost are reported. 2. Engineers: Extract pluggable components (encoders, rerankers, planners) and assess integration cost with your existing stack. 3. Product managers: Focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.