The heart rate (HR) signal is so weak in remote photoplethysmography (rPPG) and ballistocardiogram (BCG) that HR estimation is very sensitive to face and body motion disturbance caused by spontaneous head and body movements as well as facial expressions of subjects in conversation. This article proposed a novel multimodal quasi-contactless HR sensor to ensure the robustness and accuracy of HR estimation under extreme facial poses, large-motion disturbances, and multiple faces in a video for computer-aided police interrogation. Specifically, we propose a novel landmark-based approach for a deep facial region of interest (ROI) tracker and face pose constrained Kalman filter to continuously and robustly track target facial ROIs for estimating HR from face and head motion disturbances in rPPG. This motion-disturbed rPPG signal is further fused with a minimally disturbed BCG signal by the face and head movements via a bank of notch filters with a recursive weighting scheme to obtain the dominant HR frequency for final accurate HR estimation. To facilitate reproducible HR estimation research, we synchronously acquire and publicly share a multimodal data set that contains 20 sets of ECG and BCG signals as well as uncompressed, rPPG-dedicated videos from ten subjects in a stable state and large-motion state (MS) without and with large face and body movements in a sitting position. We demonstrate through experimental comparisons that the proposed multimodal HR sensor is more robust and accurate than the state-of-the-art single-modal HR sensor solely with rPPG- or BCG-based methods. The mean absolute error (MAE) of HR estimation is 7.13 BPM lower than the BCG algorithm and 3.12 BPM lower than the model-based plane-orthogonal-to-skin (POS) algorithm in the MS.

中文翻译：

---

使用具有深部ROI跟踪器和姿态约束卡尔曼滤波器的BCG融合远程PPG进行运动鲁棒性多模态心率估计

在远程光电容积描记法（rPPG）和心搏描记图（BCG）中，心率（HR）信号是如此微弱，以至于HR估算对于由自发的头部和身体运动以及谈话对象的面部表情引起的面部和身体运动干扰非常敏感。本文提出了一种新颖的多模式准非接触式HR传感器，以确保在极端面部姿势，大动作干扰和多面人脸的视频中，用于计算机辅助警察审讯的HR估计的鲁棒性和准确性。具体来说，我们针对深层面部关注区域（ROI）跟踪器和面部姿态约束卡尔曼滤波器提出了一种新颖的基于地标的方法，以连续，稳健地跟踪目标面部ROI，以从rPPG中的面部和头部运动障碍中估算出HR。通过一系列具有递归加权方案的陷波滤波器，通过脸部和头部运动，将此运动受干扰的rPPG信号与最小干扰的BCG信号进一步融合，以获得递归加权方案的主导HR频率，以进行最终的精确HR估计。为了促进可重现的HR估计研究，我们同步获取并公开共享一个多模式数据集，该数据集包含20组ECG和BCG信号以及来自十个处于稳定状态和大运动状态（MS）的未经压缩的rPPG专用视频坐着时没有面部和身体较大的动作。通过实验比较，我们证明了所提出的多模式HR传感器比仅基于rPPG或BCG的方法所具有的最新技术的单模式HR传感器更加强大和准确。人力资源估计的平均绝对误差（MAE）为7。