Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries (WWW 2024)
Paper Metadata
| Field | Content | |-------|---------| | Title | Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries | | Venue | The Web Conference (WWW) 2024 | | Source link | https://dl.acm.org/doi/abs/10.1145/3589334.3645643 | | Resource type | Academic paper | | Category | Verticals (Healthcare × LLM evaluation) |
Overview
This entry indexes a WWW 2024 paper that conducts a cross-lingual evaluation of large language models (LLMs) on healthcare queries. The central research question is whether LLMs give better healthcare answers when queries are posed in English versus other languages — a practically important question, since English-centric training data may cause uneven answer quality across languages, with real consequences for medical information safety.
The surrounding forum entry (largely a template annotation) situates the work in the broader context of LLM-era information retrieval, where traditional pipelines that separate retrieval, ranking, and generation struggle to meet user demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge.
> Note: The original post is primarily a metadata/template entry. Quantitative results should be verified against the paper PDF via the DOI link above.
Typical Evaluation Framing for This Problem
- Task: healthcare QA across multiple languages, comparing answer quality when the same question is asked in English vs. other languages.
- Metrics commonly used: factuality/consistency, human or LLM-as-judge evaluation, task success rate.
- Concerns: cross-lingual generalization, hallucination and safety in the medical domain, and English-centric data bias.
- An interpretable ensemble of graph and language models…
- Automated Query-Product Relevance Labeling using Large Language Models…
- Behavior-driven query similarity prediction based on pre-trained language models…
- Original paper: *Better to Ask in English: Cross-Lingual Evaluation of Large Language Models for Healthcare Queries*, WWW 2024. https://dl.acm.org/doi/abs/10.1145/3589334.3645643
Insights for Search / Rec / Personalization Practitioners
1. Architecture: cascade retrieval + reranking + generation remains mainstream, but agentic paradigms make retrieval strategy itself learnable. 2. Data: high-quality instruction data matters as much as click/session logs; synthetic data requires guarding against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment — academic benchmarks alone are insufficient.
Engineering Deployment Checklist
| Item | Question | Recommendation | |------|----------|----------------| | Data | Does training/index contain PII? Versioning? | Partitioned indexes, anonymization, rollback-capable embeddings | | Latency | p99 budget? How many retrieval steps? | Cascade + early stopping, caching hot queries, async reranking | | Quality | Do offline gains transfer to online CTR/satisfaction? | Interleaving experiments, human audits, citation checks | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query token and GPU cost? | Small-model routing, distillation, hybrid sparse+dense retrieval |
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decision-making and tool use | | Gen-IR | Generative Information Retrieval |