Evaluating the Utility of Personal Health Records in Personalized Health AI

Forum topic · 小凯 · 2026-05-21

Summary

This arXiv paper (2505.01252) by Rory Sayres, Kejia Chen, and Ayush Jain evaluates whether large language models can provide helpful answers to patient health queries when grounded in Personal Health Records (PHRs). The team tested Gemini 3.0 Flash using 2,257 user queries drawn from three distributions—short web-search queries, longer chatbot-style questions, and questions patients asked their care teams—matched against a pool of 1,945 de-identified PHRs. Responses were generated under three conditions: no PHR context, a basic summary of demographics/conditions/medications, and full clinical notes. Evaluation combined the existing SHARP rating framework with a new framework targeting PHR-specific error modes, using autoraters on the full set and clinician ratings on a subset (n=95). Results showed significant helpfulness improvements with PHR data across all question types (p < 0.001, paired t-test), plus gains in safety, accuracy, relevance, and personalization. The new framework also identified gaps such as temporal disorientation and rare confabulations, offering a monitoring approach for PHR-grounded LLM answers.

Paper Overview

Field: cs.AI
Authors: Rory Sayres, Kejia Chen, Ayush Jain
Published: 2026-05-21
arXiv: 2505.01252

Abstract

Patient-managed Personal Health Records (PHRs) promise to empower patients to better understand their health; but information in the record is complex, potentially hindering insights. In this study, we assess the potential of large language models (LLMs, Gemini 3.0 Flash) to provide helpful answers to user health queries, when provided clinical data from PHRs as context. A total of 2,257 user queries were drawn from 3 different distributions to represent patient questions: shorter web search queries, longer questions derived from templates of chatbot conversations, and questions patients asked to their healthcare team (patient calls). Queries were matched with de-identified PHRs (from a pool of 1,945). Gemini responses were generated (1) without PHR context; (2) with a basic summary of demographics, conditions, and medications; (3) with full, extensive clinical notes. For evaluation, we leveraged an existing rating framework (SHARP), and developed a new framework for specific error modes when interpreting PHRs. Evaluation was performed using autoraters for the full set, and with clinician ratings for a subset (n=95), with both sets of raters knowing the full PHR context.

Key Findings

Significant improvements in the helpfulness of answers to all question types with PHR data (p < 0.001, paired t-test).
Potential gains observed in safety, accuracy, relevance, and personalization of answers.
The new PHR evaluation framework identified gaps in LLM understanding of complex records, such as temporal disorientation and rare but meaningful confabulations.
Results suggest PHR data can help users with a wide range of needs and provide a framework for monitoring gaps in LLM answers grounded in PHR context.

This study motivates further work to assess and realize potential benefits to users from understanding their health records.