WearableQA: A Benchmark Testing Whether LLMs Can Reason Over Real-World Wearable Health Data

Forum topic · 小凯 · 2026-09-08

Summary

WearableQA is the first benchmark for evaluating large language model (LLM) health reasoning over real-world wearable device data. Built from 200 real users with up to 500 days of continuous monitoring, the dataset combines wearable time series (heart rate, steps, sleep, skin temperature), blood biomarkers, and demographics. It contains 4,084 multiple-choice questions organized into 16 types along two dimensions: data reasoning vs. health reasoning, and single-signal vs. cross-signal reasoning. Questions use a dual-anchoring framework (literature-grounded and population-grounded) to guarantee objective, verifiable answers. Evaluations of 14 LLMs, including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3, and Qwen 2.5, show accuracy ranging from 19.6% to 72.9% (random baseline: 10%), with most models below 60%. Models perform worst on cross-signal health reasoning (~25-40%), revealing that real-world noise, individual variability, and longitudinal dynamics remain major challenges. The benchmark defines clear capability boundaries for AI in consumer-to-clinical health monitoring.

WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data

Paper: WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data (arXiv:2609.05405) Authors: Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda Field: NLP / Health AI

Overview

WearableQA is the first benchmark targeting real-world wearable device data for LLM health reasoning. The results are sobering:

14 LLMs scored between 19.6% and 72.9% on 4,084 multiple-choice questions (random baseline: 10%)
Most models scored below 60% — far from solved
Models perform worst on cross-signal reasoning (integrating multiple physiological signals)
Real-world noise and individual variability pose major challenges

Why Existing Benchmarks Fall Short

1. Synthetic vs. real data: Most benchmarks use clean, synthetic or de-identified data, lacking real-world artifacts (motion interference, poor sensor contact, environmental effects). 2. Static vs. longitudinal data: Health is dynamic — a single heart-rate snapshot is meaningless; "elevated nighttime heart rate persisting for two weeks" is not. 3. Simple calculation vs. deep reasoning: True health reasoning requires integrating physiology, medicine, behavior, and environmental context.

The WearableQA Dataset

200 real users, up to 500 days of continuous monitoring each
Sources: wearable time series (heart rate, steps, sleep stages, skin temperature), blood biomarkers (cholesterol, glucose, vitamin D), and demographics
All real-world imperfections retained: missing values, measurement noise, large inter-individual differences

4,084 Questions in 16 Types

Two dimensions structure the question taxonomy:

Data reasoning vs. health reasoning: statistical computation on raw data vs. physiological interpretation (e.g., "Why is the user's nighttime heart rate elevated while deep sleep decreased?")
Single-signal vs. cross-signal reasoning: one data type vs. integrating heart rate, steps, sleep, and temperature (e.g., "Is the user possibly getting sick?")

16 types = 2 (data/health) × 2 (single/cross-signal) × 4 (subtypes).

Dual-Anchoring Framework

Literature-grounded: questions based on published physiological findings, with answer options drawn from known medical associations
Population-grounded: statistical significance verified against the actual 200-user dataset, ensuring objectively verifiable answers

Results

Tested models include GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3, Qwen 2.5, and Mistral:

| Task type | Approximate accuracy | |---|---| | Single-signal data reasoning | 60–70% (best) | | Single-signal health reasoning | 40–55% | | Cross-signal data reasoning | 35–50% | | Cross-signal health reasoning | 25–40% (worst) |

Even the best model (GPT-4o, 72.9%) is far from clinically usable performance.

Why Real-World Data Is So Hard

Device noise: motion artifacts, sleep-stage misclassification, temperature sensitivity
Individual variability: a resting heart rate of 45 bpm may be normal for an athlete, abnormal for others
Temporal dynamics: circadian rhythms, seasonal effects, short-term fluctuations vs. long-term trends all require different interpretation

Implications

Consumer to clinical: As wearables gain FDA-cleared features (e.g., AFib detection), AI errors — false negatives or false positives — carry real medical consequences.
Explainability: The 16 question types define a framework for mapping AI capability boundaries, showing where human oversight is needed.
Privacy and personalization: Longitudinal data risks re-identification; population-based "normal ranges" are often meaningless for individuals, requiring personal baselines that conflict with privacy goals.

Future Directions

1. Better architectures: time-series Transformers, graph neural networks, multimodal fusion models for irregular physiological signals 2. Causal reasoning: moving beyond correlation (does a higher heart rate cause worse sleep, or vice versa?) 3. Human-AI collaboration: AI handles data integration and anomaly detection; physicians handle causal judgment and patient communication

Conclusion

> Having data does not mean having knowledge, and having knowledge does not mean having wisdom.

WearableQA's contributions: a realistic, challenging benchmark; a systematic 16-type evaluation framework; a clear map of current LLM limitations; and direction for future research. Before letting AI diagnose our health, we need to diagnose AI's capabilities first.

Reference: Lee, J. S., et al. (2026). WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data. *arXiv preprint arXiv:2609.05405*.