Consumer devices now boast advanced sensors for comprehensive fitness and wellbeing tracking. We previously launched sleep sensing on the Nest Hub, leveraging Soli radar technology to analyze sleep patterns near the bedside
Our new research, published in “UWB Radar-based heart rate monitoring: A transfer learning approach”, reveals that ultra-wideband (UWB) technology, increasingly common in mobile phones, can effectively perform radar-based heart rate measurement. While UWB excels in applications like secure vehicle unlocking and precise item location, its radar sensing potential remains largely unexplored. This study showcases leveraging existing UWB hardware for vital sign monitoring, specifically heart rate (HR) measurement.
Detecting heart rate contactlessly with UWB radar, mirroring mobile phone capabilities, through a deep learning ML model.
Consumer Device Radar Sensors
Promising radar systems for vital sign measurement in consumer devices include millimeter wave frequency-modulated continuous wave (mm-wave FMCW) and impulse-radio ultra-wideband (IR-UWB) radar. Google’s prior work on sensing sleep and motion gestures, utilizing the Soli radar platform, employed FMCW technology. This provided us with extensive datasets, studies, and machine learning algorithms, including those for heart rate monitoring with FMCW radar.
Concurrently, UWB, a versatile technology gaining significant traction and appearing in numerous current mobile phone models and other consumer devices, possesses radar capabilities. The radar potential of UWB has been largely untapped; current applications primarily focus on non-radar functions like localization, tracking, vehicle unlocking, and data transfer.
Conquering Contactless Sensing Challenges
Contactless heart rate detection via radar faces challenges due to subtle heartbeat-induced chest wall movements being easily obscured by more significant breathing and body motions. Radar's inherent 3D spatial resolution, utilizing both distance and direction, allows for precise measurement zone definition around a person's torso. This capability isolates chest reflections while disregarding stationary background objects or external movements. Furthermore, its high temporal resolution samples signals rapidly (up to 200Hz), capturing even the faint, rapid motion of a heartbeat. We engineered a novel method that optimally exploits these unique 2D spatio-temporal radar signal properties for highly accurate heart rate measurement.
Bridging Different Radar Types
We explored transferring learned features from FMCW radar, benefiting from extensive existing datasets and studies, to UWB radar. These two radar systems operate on fundamentally different physical principles. Mm-wave FMCW transmits a continuous sinusoidal wave with a linearly increasing frequency, while UWB transmits extremely short pulses lasting picoseconds to nanoseconds. This research marks the first demonstration of successfully transferring learned features between radar types for vital sign measurement. We selected heart rate as our initial task due to its high utility and inherent complexity.
High-level architecture illustrating model transfer.
Developing Advanced Deep Learning for Radar Heart Rate Detection
We engineered a novel deep learning framework to accurately model the intricate spatio-temporal dynamics within radar signals for HR estimation. The architecture commences with a 2D ResNet, processing input data where one axis represents time and the other spatial measurements. This initial phase extracts features from fine-grained spatio-temporal patterns generated by chest wall movements.
Subsequently, the model collapses the spatial dimension through average pooling. The resultant feature set feeds into a 1D ResNet, specifically designed to analyze the signal solely along the temporal dimension. This second stage identifies the longer-range, periodic patterns characteristic of a heartbeat from the features previously extracted.
When trained on our FMCW dataset, the model achieved a mean absolute error (MAE) of 0.85 beats per minute (bpm) for heart rate measurement. This result significantly surpasses previous state-of-the-art performance on this dataset, reducing the prior error rate by half.
Comparing previous state-of-the-art performance with our model on FMCW radar data, including MAE with 95% confidence interval and mean absolute precision error (MAPE).
Representative example of overnight session performance on the test set. The top plot shows model performance (blue) versus ground truth (orange). The middle plot indicates body position, and the bottom plot displays estimated user distance from the radar.
Transferring Learned Features to Ultra-Wideband Radar
We then conducted a study collecting UWB radar data, alongside electrocardiogram (ECG) and photoplethysmogram (PPG) for ground truth heart rate. The UWB radar sensor was positioned simulating typical mobile phone usage: on a table in front of the user or on their lap. The UWB radar dataset comprised 37.3 hours of data, significantly smaller than the 980 hours in the FMCW dataset. Due to the UWB configuration closely mimicking mobile phone capabilities and its lower bandwidth, its range resolution was considerably lower than the FMCW dataset.
To optimize our model for transfer to the UWB dataset, we retrained it after implementing preprocessing steps to align the mm-wave FMCW radar data more closely with the target IR-UWB data, effectively reducing its range resolution. We then fine-tuned this model on the IR-UWB dataset, achieving an MAE of 4.1 bpm and a mean absolute percentage error (MAPE) of 6.3%, a 25% improvement over the baseline error rate. Our baseline performance on UWB radar, using the best model trained from scratch on our UWB dataset, was 5.4 bpm MAE and 8.4% MAPE. Through transfer learning, our UWB radar achieved compliance with Consumer Technology Association standards for heart rate measurement in consumer devices, requiring an accuracy of up to 5 bpm MAE and 10% MAPE.
Comparing baseline and transfer learning model performance on IR-UWB radar data, including MAE with 95% confidence interval and MAPE.
Representative examples of model performance (blue) versus ground truth (orange) for three selected participants from the test set for sessions with the radar on the table (a) and at the participant's lap (b).
Ensuring Accuracy Across Diverse Scenarios
We meticulously analyzed model performance across various scenarios and user conditions within each dataset to guarantee accuracy and reliability. For both radar types, we observed consistent heart rate measurement performance in adequately represented situations. For instance, the FMCW radar, which gathered data during overnight sleep sessions, maintained performance across different sleep positions and even during transitions between them. Similarly, with UWB radar, heart rate measurement proved equally accurate for both tested device positions relative to the user: on a table in front of them or in their lap. For comprehensive details on this subgroup analysis and other findings, consult the full research paper.
The Broader Impact: Everyday Health Monitoring
Heart rate measurement offers crucial insights into an individual's cardiovascular status and physiological responses across various health conditions, making it vital for a spectrum of health, fitness, and wellness applications. This demonstrated capability for heart rate measurement could represent a significant step towards utilizing mobile devices to capture even more complex and subtle health signals from the heart and major blood vessels.
While wearable devices like fitness bands and rings have popularized continuous health and fitness monitoring, the ability to measure heart rate contactlessly using consumer-grade radar sensors expands this technology's accessibility to a much broader audience of smartphone users. Our study focused on heart rate during sleep (FMCW) and in common phone-holding positions (UWB). As technology advances, continuous monitoring could integrate seamlessly into daily routines across various settings.
Implications for Future Devices
This research advances contactless heart rate measurement using consumer devices, particularly as ultra-wideband (UWB) technology becomes more prevalent in mobile phones. Although this study did not involve direct real-world testing with mobile phones, it establishes critical foundational knowledge for such future applications.
A key finding demonstrates the successful adaptation of a model trained on one radar type (FMCW) for another (UWB) to measure heart rate. This transfer learning approach signifies a major advancement, suggesting a more efficient development pathway. Future research can leverage foundational knowledge from existing, extensive datasets for new devices, accelerating the integration of such features into consumer products by reducing the need for extensive data collection for each new hardware iteration.