Shun Liao, Paolo Di Achille, Jiang Wu, Silviu Borac, Jonathan Wang, Xin Liu, Eric Teasley, Lawrence Cai, Yuzhe Yang, Yun Liu, Daniel McDuff, Hao-Wei Su, Brent Winslow, Anupam Pathak, Shwetak Patel, James A. Taylor, Jameson K. Rogers, Ming-Zher Poh
5 min
The authors investigate whether a smartphone-based system can accurately and equitably monitor heart rate (HR) and resting heart rate (RHR) passively—without requiring specialized wearable hardware—by utilizing the front-facing camera during routine, everyday smartphone interactions.
The researchers developed a system called PHRM (Passive Heart Rate Monitoring), which uses remote photoplethysmography (rPPG) to detect subtle changes in facial skin color associated with the cardiac cycle. The system processes 8-second video clips captured automatically when a user unlocks their phone.
The pipeline includes:
The system was trained on over 225,000 videos and validated against electrocardiogram (ECG) and wearable tracker data from over 200 participants, specifically ensuring representation across diverse skin tones and lighting conditions.
Longitudinal RHR tracking is a vital biomarker for cardiovascular health and mortality, but it has historically been limited to those who own and consistently wear dedicated fitness trackers. By enabling passive monitoring through the ubiquitous smartphone, this technology lowers the barrier to entry for health tracking. Furthermore, by demonstrating equitable performance across diverse skin tones, the study addresses a critical limitation in existing optical health-monitoring technologies, paving the way for more inclusive and accessible digital health tools.
Resting heart rate (RHR) is a key biomarker of cardiovascular health and mortality1–3, but passively tracking it longitudinally generally requires a wearable device, limiting its availability. Here we present passive heart-rate monitoring (PHRM), a deep-learning system that uses facial video-based photoplethysmography for passive measurements of heart rate (HR) and RHR during everyday smartphone interactions. Our system was developed using 192,353 videos from 485 participants and validated on 162,546 videos from 211 participants in laboratory and free-living conditions, representing, to our knowledge, the largest validation study of its kind. PHRM outperformed state-of-the-art methods on our benchmarks. Compared with reference electrocardiograms, PHRM achieved a mean absolute percentage error (MAPE) lower than 10% for HR measurements across three skin-tone groups of light, medium and dark pigmentation, meeting industry accuracy standards; MAPE for each skin-tone group was non-inferior versus the others. Daily RHR measured by PHRM had a mean absolute error of less than five beats per minute, compared with a wearable HR tracker, and was associated with known risk factors for cardiovascular disease. These results highlight the potential of smartphones for enabling passive and equitable monitoring of heart health. To facilitate further research, we publicly release a large, annotated smartphone video dataset along with a pre-trained HR model. A machine-learning model that uses smartphone cameras to measure heart rate in the background during normal daily phone use and subsequently estimate resting heart rate could make it easier for people to monitor heart health.
Alex: That's a meaningful distinction. So the system is grabbing short clips every time someone unlocks their phone. How does it go from those scattered, imperfect snippets to a reliable daily reading?
Sam: It works in two stages. First, there's what they call "confidence gating." Before any clip is used, the system evaluates its quality. If the footage is too shaky, or the lighting is too poor, that clip is simply discarded. It only keeps the measurements it trusts.
Alex: And then it averages the ones that pass?
Sam: Not quite a simple average. The valid measurements are fed into what's called a Kalman filter—a tool borrowed from engineering that's very good at combining noisy, intermittent data into a smooth, stable estimate. Think of it like a weather forecast that doesn't just look at today's temperature, but weighs all the recent readings to give you the most reliable picture. Over the course of a day, many small snapshots add up to a solid resting heart rate figure.
Alex: So it's not one clean measurement—it's an accumulation of evidence built up through ordinary phone use. How did they verify it was actually accurate?
Sam: They compared the system's output against readings from a standard clinical heart rate monitor. The key metric was how far off the smartphone estimate was from the real value, on average. Across all the skin tones and lighting conditions they tested, that error stayed below ten percent—which the researchers consider a meaningful indicator of reliability for a passive, uncontrolled setting.
Alex: And the Monk skin tone scale was how they categorized the diversity of people in the dataset?
Sam: Correct. The Monk scale is a standardized way of classifying a broad spectrum of skin tones, developed specifically to be more representative than older systems. Using it meant the researchers could confirm the technology wasn't just working for one segment of the population—it held up across the full range they tested.
Alex: There's something quietly significant about that. The whole premise is that the system works precisely because it doesn't ask you to do anything differently. Your phone is already in your hand dozens of times a day.
Sam: That's the key insight. Most health monitoring requires a deliberate action—putting on a device, sitting still, pressing a button. This approach turns the ordinary act of checking your phone into a continuous, background health check. The paper suggests that kind of passive, unobtrusive monitoring could make heart rate tracking accessible to people who would never use a dedicated wearable—which, depending on the population, could be most people.
Alex: It's a reminder that sometimes the most practical solutions are the ones that fit into life as it already is, rather than asking people to change their habits. Thanks for listening to ResearchPod.