HerHealthEval: A Multilingual, Register-Sensitive Benchmark for Women's Health Communication in LLMs

Forum topic · 小凯 · 2026-09-20

Summary

HerHealthEval is a controlled evaluation framework for assessing how well large language models understand women's-health queries across languages and communicative registers. Each clinical case is provided in matched English, French, and Modern Standard Arabic versions across six communicative forms: canonical, clinical, layperson, indirect/hedged, emotionally concerned, and deliberately under-specified. The first five express the same underlying concern with identical clinical information, while the under-specified form intentionally omits details to test whether a model recognizes when clarification is needed. The authors evaluate a multilingual instruction model and its QLoRA-adapted variants on concern classification, risk calibration, clarification behavior, parsing compliance, and cross-form consistency. Results show that aggregate accuracy can hide safety-critical failures: one multilingual adaptation exhibits 0.994 under-triage in French and Arabic. Controlled re-adaptation using language-invariant risk labels derived from source texts reduces under-triage to 0.572 and 0.558 respectively. The findings indicate that robust multilingual medical evaluation must explicitly test register variation, uncertainty handling, and the provenance and invariance of adaptation labels. Paper: arXiv 2609.20684.

Overview

Field: NLP Authors: Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi, Hana Essam Sayed Ahmed Amrya, Mariam Mousa arXiv: 2609.20684

Large language models are increasingly used in healthcare communication, yet most evaluations emphasize response quality while assuming that the user's concern has been interpreted correctly. HerHealthEval addresses this gap with a controlled evaluation framework for multilingual understanding of women's-health communication.

Framework Design

For each clinical case, HerHealthEval provides matched versions in English, French, and Modern Standard Arabic using six communicative forms:

Canonical
Clinical
Layperson
Indirect / hedged
Emotionally concerned
Deliberately under-specified

The first five express the same underlying concern and retain the same clinical information, whereas the under-specified form intentionally omits relevant details to test whether the model recognizes that clarification is needed.

Evaluation and Findings

The authors evaluate a multilingual instruction model and its QLoRA-adapted variants on:

Concern classification
Risk calibration
Clarification behavior
Parsing compliance
Cross-form consistency

Key findings: aggregate accuracy and consistency can mask safety-critical failures. One multilingual adaptation exhibits 0.994 under-triage in French and Arabic. After controlled re-adaptation using language-invariant risk labels derived from the source texts, under-triage drops to 0.572 (French) and 0.558 (Arabic).

Takeaway

Robust multilingual medical evaluation must explicitly test register variation, uncertainty handling, and the provenance and invariance of adaptation labels.