5th Year Master's Thesis Presentation - Tianyue Ruby Sun
— 11:00am
Location:
In Person
-
Gates Hillman 8102
Speaker:
TIANYUE RUBY SUN
,
Master's Student
Computer Science Department
Carnegie Mellon University
https://www.linkedin.com/in/ts3/
Remote photoplethysmography (rPPG) enables contactless measurement of physiological signals such as heart rate and respiratory rate from videos, offering a practical alternative to traditional contact-based sensor measurements. Recent deep learning methods have achieved strong rPPG accuracy, but these approaches often depend on controlled settings and struggle to generalize to real-world environments with motion and varying lighting. These limitations are in part due to the reliance on techniques such as manual parameter tuning and the need for large labelled datasets that are often captured under clean conditions.
This research thesis presents an exploration of the end-to-end rPPG pipeline. The primary contribution is a novel spatiotemporal architecture for rPPG that combines DINOv2, a vision transformer, and Chronos, a time series model. This represents the first multimodal rPPG framework that leverages a combination of spatial and temporal representations from foundation models for physiological measurement. The two foundation models are kept frozen, and a lightweight prediction head is trained.
The proposed model achieves strong performance on the synthetic SCAMPS dataset for heart rate estimation, establishing benchmarks for future rPPG research. On real-world datasets, including PURE and UBFC-rPPG, the model demonstrates effective learning of blood volume pulse (BVP) waveforms and heart rate estimation, despite the increased errors reflecting the difficulty of more challenging conditions. Extensions of the model to respiratory rate illustrate the generalizability of the architecture across different physiological measurement tasks. Overall, the results show that foundation models can improve rPPG robustness and generalization, offering a promising path towards practical rPPG systems with applications in inpatient monitoring, telehealth, and emergency response.
In addition to model development, this thesis analyzes components of the full rPPG pipeline, including signal processing and ground truth extraction. It is shown that common signal processing methods applied to the same BVP signal can lead to discrepancies in the estimation of the scalar heart rate value. Moreover, the method of obtaining the ground truth from data affects the reported performances. These insights motivate the need to further discuss reliable signal processing and evaluation procedures to ensure reliable comparisons and interpretations of rPPG methods.
Thesis Committee
Artur Dubrawski (Chair)
Laszlo Jeni
Additional Information
For More Information:
amalloy@cs.cmu.edu