WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

Forum topic · 小凯 · 2026-09-08

Summary

WearableQA is the first benchmark evaluating LLM health reasoning on real-world wearable device data, built from 200 users with up to 500 days of longitudinal monitoring including wearable time series (heart rate, sleep, steps, skin temperature), blood biomarkers, and demographics. It contains 4,084 multiple-choice questions organized into 16 question types across two dimensions: data vs. health reasoning and single-signal vs. cross-signal reasoning, using a dual-anchoring framework (literature-grounded and population-grounded) to ensure answer verifiability. Evaluation of 14 LLMs, including GPT-4o, Claude 3.5 Sonnet, and Llama 3, shows accuracy ranging from 19.6% to 72.9% against a 10% random baseline, with most models below 60%. Models perform worst on cross-signal health reasoning (roughly 25-40%), revealing that real-world noise, individual variability, and longitudinal context remain major obstacles. The benchmark highlights that current LLMs are far from clinical-grade wearable data interpretation and defines a framework for assessing AI capability boundaries in consumer health applications.

Overview

Paper: WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data Authors: Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda arXiv: 2609.05405 Field: NLP / Health AI

WearableQA is presented as the first benchmark for LLM reasoning over real-world wearable device data, and the results are sobering:

14 LLMs scored between 19.6% and 72.9% on 4,084 multiple-choice questions (random baseline: 10%)
Most models scored below 60%, far from solved
Models performed worst on cross-signal reasoning (integrating multiple physiological signals)
Real-world noise and individual variability remain major challenges

Why Existing Benchmarks Fall Short

Prior health AI benchmarks have three key blind spots:

1. Synthetic vs. real data: Most use synthetic or over-cleaned data lacking device artifacts, motion interference, and individual differences. 2. Static vs. longitudinal data: Health is dynamic — a heart rate value is only meaningful in the context of trends over days, weeks, or years. 3. Simple computation vs. deep reasoning: Real health reasoning requires integrating physiology, medicine, behavioral patterns, and environmental context.

Benchmark Design

Dataset

200 real users, up to 500 days of continuous monitoring each
Data sources: wearable time series (heart rate, steps, sleep, skin temperature), blood biomarkers (cholesterol, glucose, vitamin D), demographics
All real-world imperfections preserved: missing values, measurement noise, large inter-individual variability

16 Question Types

Questions span two dimensions:

Data reasoning vs. health reasoning: statistical computation on raw data vs. physiological interpretation (e.g., identifying likely causes of elevated nighttime heart rate with reduced deep sleep)
Single-signal vs. cross-signal reasoning: one data type vs. integrating heart rate, sleep, steps, and temperature (e.g., inferring possible illness)

2 × 2 × 4 subtypes = 16 question types.

Dual-Anchoring Framework

To ensure questions have verifiable answers:

Literature-grounded: questions based on published physiological research findings
Population-grounded: answers verified statistically against the actual 200-user dataset

Results

Tested models include GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3, Qwen 2.5, Mistral, and others:

Best: 72.9% (10-way multiple choice)
Worst: 19.6%, close to random guessing
Median: roughly 45%

Performance degrades by reasoning type:

| Task type | Approx. accuracy | |---|---| | Single-signal data reasoning | 60–70% | | Single-signal health reasoning | 40–55% | | Cross-signal data reasoning | 35–50% | | Cross-signal health reasoning | 25–40% |

Cross-signal health reasoning — integrating multiple data sources with physiological knowledge — is the weakest capability.

Key Challenges Identified

Device noise: motion artifacts, sleep-stage misclassification, temperature sensitivity
Individual variability: resting heart rate can range from 40 bpm (athletes) to 60–100 bpm (general population); population norms may not apply to individuals
Temporal dynamics: circadian rhythms, seasonal effects, and distinguishing short-term fluctuations from long-term trends

Broader Implications

1. From fitness tracking to clinical decisions: With wearables gaining FDA-cleared features (e.g., atrial fibrillation detection), AI reasoning errors can mean missed diagnoses or false alarms. 2. Explainability: The 16 question types form a capability-mapping framework showing where AI is reliable and where human oversight is required. 3. Privacy and personalization: Longitudinal data risks re-identification, while personalized baselines conflict with privacy constraints.

Future Directions

Architectures better suited to irregularly sampled multi-dimensional time series (time-series Transformers, graph neural networks, multimodal fusion)
Moving from correlation to causal reasoning
Human-AI collaboration: AI for pattern detection and data integration; clinicians for judgment and communication

Conclusion

> Having data is not the same as having knowledge, and having knowledge is not the same as having wisdom.

WearableQA contributes a realistic, challenging benchmark, a systematic 16-type evaluation framework, and a clear picture of current LLM capability boundaries — a reminder that before letting AI diagnose our health, we should first diagnose the AI.

References

Lee, J. S., Chen, X., Chuang, P., Shenoy, A., Wei, J., Ko, D., Kim, H. J., & Corda, B. (2026). WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data. *arXiv preprint arXiv:2609.05405*.