WearableQA: A New Benchmark Tests Whether LLMs Can Reason Over Real-World Smartwatch Health Data

Forum topic · 小凯 · 2026-09-08

Summary

WearableQA (arXiv:2609.05405) is the first benchmark for evaluating large language model (LLM) health reasoning on real-world wearable device data. Built from 200 real users with up to 500 days of continuous monitoring, it combines wearable time series (heart rate, steps, sleep, skin temperature), blood biomarkers, and demographic information, deliberately preserving real-world noise, missing data, and individual variability. The benchmark contains 4,084 multiple-choice questions organized into 16 question types across two dimensions: data reasoning vs. health reasoning, and single-signal vs. cross-signal reasoning, with a dual-anchoring framework (literature-grounded and population-grounded) ensuring verifiable answers. Results are sobering: 14 evaluated LLMs, including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3, and Qwen 2.5, scored between 19.6% and 72.9% (random baseline 10%), with most models below 60%. Cross-signal health reasoning proved hardest, showing that even top models are far from clinically usable for wearable-based health insights. The post discusses implications for clinical-grade monitoring, explainability, privacy, personalization, and future directions such as time-series-native architectures and causal reasoning.

WearableQA: Does Your Smartwatch AI Really Understand You?

A detailed walkthrough of WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data (arXiv:2609.05405), a paper proposing the first LLM reasoning benchmark built on real-world wearable device data.

Paper info

Title: WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data
Authors: Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, Benoit Corda
arXiv ID: 2609.05405
Published: 2026-09-04
Field: NLP / Health AI

Key findings

First benchmark targeting real-world wearable data for LLM health reasoning.
14 LLMs scored between 19.6% and 72.9% on 4,084 multiple-choice questions (random baseline: 10%).
Most models scored below 60% — the task is far from solved.
Models perform worst on cross-signal reasoning (integrating multiple physiological signals).
Real-world noise and individual variability remain major challenges.

Background: the gap in existing benchmarks

Wearables have evolved from simple step counters to continuous multi-signal monitors producing gigabytes of physiological data per day (heart rate sampled every second ≈ 86,400 points/day, plus steps, sleep stages, blood oxygen, skin temperature). We live in an era that is "data-rich but insight-poor."

Existing health AI benchmarks have three blind spots:

1. Synthetic vs. real data — most use clean, de-identified data lacking real-world artifacts (motion interference, poor sensor contact, environmental effects, individual differences). 2. Static vs. longitudinal data — health is dynamic; a single heart-rate snapshot is meaningless compared to a two-week upward trend in nighttime heart rate. 3. Simple computation vs. deep reasoning — real health reasoning requires integrating physiology, medicine, behavior, and environmental context.

The WearableQA dataset

200 real users, up to 500 days of continuous monitoring each.
Sources: wearable time series, blood biomarkers (cholesterol, glucose, vitamin D), demographics (age, sex, BMI).
All real-world "flaws" preserved: missing values from device disconnection, large inter-individual variation, measurement noise.

16 question types

Questions span two dimensions:

Data reasoning vs. health reasoning — statistical computation on raw data vs. physiological interpretation (e.g., "why is this user's deep sleep decreasing while nighttime heart rate rises?").
Single-signal vs. cross-signal reasoning — one data type vs. integrating heart rate, steps, sleep, and skin temperature.

16 types = 2 (data/health) × 2 (single/cross-signal) × 4 subtypes.

Dual-anchoring framework

To guarantee answerable questions:

Literature-grounded: questions based on published physiological findings.
Population-grounded: answers verified statistically against the 200-user dataset.

Results: how well do LLMs reason about health?

Evaluated models included GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3, Qwen 2.5, and Mistral:

Best: 72.9% (GPT-4o, on 10-way multiple choice)
Worst: 19.6% (near random)
Median: ~45%

Cross-signal reasoning is the Achilles' heel

Approximate accuracy by task type:

Single-signal data reasoning: 60–70% (best — mostly statistics)
Single-signal health reasoning: 40–55%
Cross-signal data reasoning: 35–50%
Cross-signal health reasoning: 25–40% (worst)

Why real-world noise is hard

Models must handle sensor artifacts, individualized baselines (an athlete's resting heart rate of 40 bpm vs. 60–100 bpm for most people), and temporal dynamics (circadian rhythms, short-term fluctuations vs. long-term trends).

Why this matters

From fitness tracking to clinical decisions: Apple Watch already has FDA-cleared atrial fibrillation detection; insurers are starting to use wearable data. AI errors can mean missed diagnoses or costly false positives.
Explainability: The 16 question types define a capability map — if a model is weak at cross-signal health reasoning, it is unsuitable for complex health assessment.
Privacy and personalization: Longitudinal data risks re-identification; population-based "normal ranges" may not apply to individuals, yet personal baselines require long-term data collection that conflicts with privacy.

Future directions

1. Better architectures: time-series transformers for irregular sampling, graph neural networks for inter-signal relationships, multimodal fusion models. 2. Causal reasoning: models spot correlations but cannot determine whether rising heart rate causes poor sleep or vice versa. 3. Human-AI collaboration: AI for data integration, pattern detection, and literature retrieval; doctors for synthesis, causal judgment, and patient communication.

Conclusion

> Having data does not equal having knowledge, and having knowledge does not equal having wisdom.

The paper's contributions: a realistic, challenging benchmark; a systematic 16-type evaluation framework; a clear delineation of current LLM capability boundaries; and direction for future research. Before letting AI diagnose our health, we need to first diagnose the AI's abilities.

References

Lee, J. S., Chen, X., Chuang, P., Shenoy, A., Wei, J., Ko, D., Kim, H. J., & Corda, B. (2026). WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data. *arXiv preprint arXiv:2609.05405*.