WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

This post introduces WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data (arXiv:2609.05405), presented as the first LLM benchmark built on real-world wearable device data.

Paper information

• Title: WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data
• Authors: Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda
• arXiv ID: 2609.05405
• Published: 2026-09-04
• Field: NLP / Health AI

Key findings

• 14 LLMs were evaluated on 4,084 multiple-choice questions; accuracy ranged from 19.6% to 72.9% (random baseline: 10%).
• Most models score below 60%, indicating the benchmark is far from solved.
• Models perform worst on cross-signal reasoning — integrating multiple physiological signals.
• Real-world noise and individual variability pose major challenges for current models.

Dataset construction

WearableQA is built on a real, longitudinal, multimodal dataset:
• 200 real users, with up to 500 days of continuous monitoring per person
• Data sources:
• Wearable time series (heart rate, steps, sleep stages, skin temperature, etc.)
• Blood biomarkers (e.g., cholesterol, glucose, vitamin D from lab reports)
• Demographics (age, sex, BMI, etc.)
• Data retains real-world imperfections: missing values from device disconnection, large individual differences, measurement noise, and complex lifestyle interactions.

Question taxonomy: 16 question types

Questions are organized along two dimensions, yielding 2 × 2 × 4 = 16 types: 1. Data reasoning vs. health reasoning — statistical computation on raw data vs. physiological interpretation (e.g., interpreting elevated nocturnal heart rate plus reduced deep sleep). 2. Single-signal vs. cross-signal reasoning — one data type vs. integrating heart rate, sleep, activity, and temperature (e.g., inferring possible illness).

Dual-anchoring framework

To ensure questions have objectively verifiable answers:
• Literature-grounded: questions based on published physiological research findings.
• Population-grounded: statistical analysis over the 200-user dataset ensures answers are statistically significant and verifiable.

Evaluation results

• Best performance: 72.9% (GPT-4o, on 10-way multiple choice)
• Worst performance: 19.6% (near the 10% random baseline)
• Most models: below 60%; median around 45%
• Tested models include proprietary (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro) and open-source (Llama 3, Qwen 2.5, Mistral) LLMs.

Performance degrades along the reasoning ladder:

• Single-signal data reasoning: best (~60–70%)
• Single-signal health reasoning: moderate (~40–55%)
• Cross-signal data reasoning: poor (~35–50%)
• Cross-signal health reasoning: worst (~25–40%)

Why real-world data is hard

• Device noise: motion artifacts in heart-rate sensors, sleep-stage misclassification, temperature sensitivity.
• Individual variability: resting heart rates range from ~40 bpm (athletes) to 60–100 bpm (general population); population-level "normal" ranges may not apply to individuals.
• Temporal dynamics: circadian rhythms, seasonal effects, and distinguishing short-term fluctuations from long-term trends.

Implications

• Wearables are shifting from consumer fitness tracking toward clinical-grade monitoring (e.g., FDA-cleared AFib detection), making AI reasoning reliability critical.
• The 16 question types provide a capability-mapping framework for explainability and for defining where AI can act independently vs. needs clinician oversight.
• Privacy risks remain: even de-identified longitudinal data may be re-identifiable through temporal pattern matching.

Future directions

• Architecture: time-series transformers for irregular sampling, graph networks for physiological correlations, multimodal fusion models.
• Causal reasoning: moving beyond correlation (e.g., does elevated heart rate cause poor sleep, or vice versa?)
• Human-AI collaboration: AI for data integration, pattern detection, and literature retrieval; humans for causal judgment, ethical decisions, and patient communication.

Conclusion

The paper's core message: *having data does not equal having knowledge, and having knowledge does not equal having wisdom*. WearableQA provides a realistic, challenging benchmark with a systematic evaluation framework, reveals the capability boundaries of current LLMs, and reminds us that before letting AI diagnose our health, we must first diagnose the AI.

Reference

• Lee, J. S., Chen, X., Chuang, P., Shenoy, A., Wei, J., Ko, D., Kim, H. J., & Corda, B. (2026). WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data. *arXiv preprint arXiv:2609.05405*.