WearableQA: A Benchmark Testing Whether LLMs Can Reason Over Real-World Smartwatch Health Data

Forum topic · 小凯 · 2026-09-08

Summary

WearableQA (arXiv:2609.05405) is the first benchmark for evaluating LLM health reasoning over real-world wearable device data, built from 200 real users monitored for up to 500 days, combining wearable time series (heart rate, steps, sleep, skin temperature), blood biomarkers, and demographics. It contains 4,084 multiple-choice questions organized into 16 types along two dimensions: data vs. health reasoning, and single-signal vs. cross-signal reasoning, with a dual-anchoring framework (literature-grounded and population-grounded) ensuring answer validity. Testing 14 LLMs including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3, Qwen 2.5, and Mistral, accuracy ranged from 19.6% to 72.9% against a 10% random baseline, with most models below 60%. Cross-signal health reasoning proved hardest (roughly 25-40%), revealing that noisy, individualized, longitudinal wearable data remains far beyond current LLM capabilities for clinical use. The forum post discusses implications for consumer-to-clinical wearable health AI, explainability, privacy, and future directions like causal reasoning and human-AI collaboration.

This forum post discusses the paper WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data (arXiv:2609.05405, published 2026-09-04, NLP/Health AI), by Ji Soo Lee, Xilun Chen, Pierce Chuang, Ashish Shenoy, Jason Wei, Dohwan Ko, Hyunwoo J. Kim, and Benoit Corda.

Key findings

First benchmark targeting LLM reasoning over real-world wearable device data.
14 LLMs tested on 4,084 multiple-choice questions scored from 19.6% to 72.9% accuracy (random baseline: 10%).
Most models scored below 60% — the problem is far from solved.
Models performed worst on cross-signal reasoning (integrating multiple physiological signals).
Real-world noise and individual variability pose major challenges.

Motivation: rich data, poor insight

Modern wearables continuously capture heart rate (86,400 samples/day at 1 Hz), steps, sleep stages, blood oxygen, skin temperature, and more. Yet existing health-AI benchmarks have three blind spots:

1. Synthetic vs. real data — most benchmarks use clean synthetic or de-identified data, lacking real-world artifacts (motion interference, poor device contact, environmental effects). 2. Static vs. longitudinal data — health is dynamic; a heart-rate value is meaningless without comparison to the user's own history. 3. Simple calculation vs. deep reasoning — real health reasoning requires integrating physiology, medicine, behavior, and environmental context.

Dataset and question design

200 real users, up to 500 days of continuous monitoring each.
Sources: wearable time series, blood biomarkers (cholesterol, glucose, vitamin D, etc.), demographics.
4,084 questions in 16 types = 2 (data vs. health reasoning) × 2 (single- vs. cross-signal) × 4 subtypes.
Dual-anchoring framework ensures answerability:
*Literature-anchored*: questions grounded in published physiological findings.
*Population-anchored*: statistically verified against the actual 200-user dataset.

Evaluation results

Models tested included GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3, Qwen 2.5, and Mistral. Approximate accuracy by category:

| Task type | Accuracy | |---|---| | Single-signal data reasoning | ~60–70% (best) | | Single-signal health reasoning | ~40–55% | | Cross-signal data reasoning | ~35–50% | | Cross-signal health reasoning | ~25–40% (worst) |

Even the best model (GPT-4o at 72.9% on 10-way multiple choice) falls well short of clinical usability. Real-world challenges include device noise, individual baselines (an athlete's resting heart rate can be ~40 bpm vs. 60–100 bpm for others), and temporal dynamics (circadian rhythms, seasonal effects, short-term fluctuations vs. long-term trends).

Why it matters

Wearables are shifting from consumer fitness tracking toward clinical-grade monitoring (e.g., FDA-cleared AFib detection), making AI reasoning reliability critical — false negatives may miss serious disease; false positives trigger unnecessary procedures.
The 16 question types double as an explainability/capability-mapping framework, clarifying which tasks AI can handle independently and which need physician oversight.
Raises unresolved issues of privacy (longitudinal data can be re-identified) and personalization (population norms may not apply to individuals).

Future directions

Architectures better suited to irregularly sampled multivariate time series (time-series Transformers, graph neural networks, multimodal fusion).
Moving from correlational to causal reasoning.
Human-AI collaboration: AI for data integration, pattern detection, and anomaly flagging; doctors for synthesis, causal judgment, and patient communication.

The post closes with the reflection: *having data is not the same as having knowledge, and having knowledge is not the same as having wisdom* — before letting AI diagnose our health, we need to first diagnose the AI.

Reference

Lee, J. S., Chen, X., Chuang, P., Shenoy, A., Wei, J., Ko, D., Kim, H. J., & Corda, B. (2026). WearableQA: A Benchmark for Health Reasoning over Real-World Wearable Data. *arXiv preprint arXiv:2609.05405*.