1.

Introduction

Photoplethysmography imaging (PPGI) is a non-invasive and non-contact technology that can measure and monitor human vital signs such as the heart rate (HR), respiratory rate (RR), heart rate variability (HRV), and blood oxygen saturation (SpO₂) by measuring the minute changes in the skin color due to the heartbeat with a normal camera.¹ PPGI technology has been attracting a lot of attention from researchers because of its several advantages. First, it does not cause any discomfort to the people and also not interfere with their activities. Next, it allows measurement of human cardiac activities with only a smartphone, which requires a low cost to implement. The amount of hemoglobin changes in the microvascular network in the dermis due to the systole and diastole of the heart. At the same time, the absorption of light also changes according to the amount of hemoglobin, and it makes a variation in the intensity of the reflected light. Based on this fact, PPGI technology extracts the pulse signal from the image of the exposed skin. Therefore, PPGI technology requires not only sufficient knowledge of the light reflection mechanism on the skin and the transformation relationship between the color spaces but also the various signal processing techniques.

In Refs. 2–5, the PPGI-based HR measurement techniques studied in recent years have been outlined and systematically described by dividing them into the various categories from the different perspectives. Most of the techniques for measuring vital signs by PPGI can be seen as the signal extraction techniques by the linear and nonlinear combination of the color channel signals. In other words, only the method of obtaining the combination coefficients of the color channel signals is different. The methods by combining the R, G, and B channels can be classified into two categories: (1) blind source separation (BSS)-based method and (2) prior knowledge-based method.

Since Ref. 6 applied the independent component analysis (ICA) to the PPGI-based HR measurement, many methods based on BSS have been employed.^7–14 However, the BSS-based methods have some limitations such as the order uncertainty, the requirement for the signal length, and the computational complexity; therefore, a series of problems can be raised in ensuring the automatism and real-time performance.

The methods based on prior knowledge can be divided into the following four classes. The first class is one that uses the individual color channels in the different color spaces. The sRGB color space is based on three basic colors including red, green, and blue, and through the different combinations of these three colors, any color in nature can be synthesized. The sRGB color space is commonly used in computer systems, cameras, and videos, and thus, it is the most widely employed in HR measurement based on PPGI. Reference 15 revealed that in the sRGB color space, the G channel had the most pulse information, and the R and B channels also had a certain amount of the pulse information. After that, several studies^16,17 measured HR using the G channel in the sRGB color space. In Ref. 18, a study on the effectiveness of YUV color space in HR measurement based on PPGI was conducted. The YUV color space consists of a luma component (Y) and two chrominance components (U: B-Y, V: R-Y), and it is used as a default for the Android OS. By using the Android SDK, the captured frames can be stored in a data buffer with YUV format, so it can be considered as being practically more efficient in processing YUV data than RGB data. Comparing the channel signals between the sRGB and the YUV color spaces, it was found that using the V channel of the YUV color space had a better performance than either using all the channels of the sRGB color space or applying ICA or PCA to all the channels of the sRGB color space.¹⁸ In the HR measurement based on PPGI, the CIE Lab color space is also used. It is composed of a lightness component ($\text{L}$) and two chromatic components ($a*$ and $b*$), in which the Euclidean distance between two different colors is similar to the color difference perceived by the human eye.¹⁹ In Ref. 20, the channels in the sRGB and the CIE Lab color spaces were compared and analyzed. As a result, it was proved that $a*$ channel performed better than $b*$ channel. In addition, it was shown that the signal of the $a*$ channel had more pulse information than the G channel of the sRGB color space. In Ref. 21, comparisons of the R, G, B, and hue color channels were conducted. As a result, the hue channel showed the best performance among the four color channels. Meanwhile, in Ref. 22, the channel signals for the seven color spaces (sRGB, HSL, HSV, HIS, XYZ, CIE XYZ, and CIE YUV) were compared, and their performances were compared with ICA. The result verified that using the hue channel of HSV/HSL/HIS could measure HR more accurately than using all channels of the seven color spaces or ICA. HSL/HSV/HSI color spaces are cylindrical-coordinate color systems, where H is hue, S means saturation, L stands for lightness, V denotes value, and I represents intensity. Also, in all three color spaces, the hue channel is the same and the saturation channel is different. Variations in the sRGB color space depend on the color of the object as well as on the intensity of the reflected light from the surface, whereas hue channel does not depend on lightness.²² These characteristics show that hue channel is suitable for HR measurement. The second class includes the methods using the chrominance. In Ref. 23, a study on HR measurement using YCbCr color space was performed. The YCbCr color space consists of a luminance component (Y) and two chroma components [the blue-difference chroma component (Cb) and red-difference chroma component (Cr)], which separates the intensity and chroma information from the color information. Based on this, it was shown that Cb and Cr channels in the YCbCr color space contained a certain amount of pulse information and these two channels had the anti-phase nature, which resulted in a good SNR.²³ In addition, it was suggested that the quality of the signal could be improved by the chrominance in the RGB color space, of which G-B had the best SNR, and this chrominance signal was also used in Ref. 24. Moreover, in some papers,²⁵ HR was measured by using the $-{\text{R}}_n+2·{\text{G}}_n-{\text{B}}_n$, which was the intersection of the projection planes used in Refs. 24 and 26. The third one is based on the model. Model-based methods include the chrominance-based method (CHROM),²⁶ the blood volume pulse signature method (PBV),²⁷ and the plane-orthogonal-to-skin based method (POS).²⁴ According to the experimental results of Ref. 24, the overall performance of POS was better than the G,¹⁵ G-R,²⁸ ICA,⁷ PCA,⁸ CHROM, PBV, and 2SR.²⁹ A detailed explanation of the model-based methods can be found in Ref. 24. The other methods except for three methods mentioned above belong to the fourth class. In Ref. 30, the G channel in the intensity normalized RGB color space was used, whereas in Ref. 31, the change of blood concentration due to arterial pulsation as a pixel quotient in log space was defined and it was employed to improve the quality of the signal. Obviously, prior knowledge-based methods extract the pulse signals by considering the prior knowledge of the color vectors, thus the computational complexity of the algorithm can be reduced, and it can be considered that the extracted signals contain the enough pulse information. However, most studies for measuring the vital signs did not perform the comprehensive analysis on the color channel combination of the color spaces, but selected a color space based on the analysis of the relative merits for the color channel combination in two or more color spaces, in which the process for HR estimation was performed.

In this paper, we first use the public database UBFC-RPPG and PURE to compare and analyze 13 color channel combinations that can be considered to have the potential abilities for the artifact reduction, among the color channel combinations in the different color spaces. As a result, for the two databases, it is verified that the quality of the pulse signal can be significantly improved by the simple color space transformation from the sRGB space to the intensity normalized RGB space and the Cb+Cr combination in the YCbCr color space contains the most pulse information. In addition, it is also revealed that the plane created by the two chroma channel Cb and Cr in the YCbCr color space is orthogonal to the skin-tone direction [1 = (1, 1, 1) in the temporally normalized RGB space], which has the same characteristics as the POS plane suggested in Ref. 12. Considering the results mentioned above, this paper proposes a modified POS to extract the pulse signals by projecting the intensity normalized RGB signals onto the CbCr plane and the fine-tuning on the CbCr plane. With the experimental results of Ref. 24, we conducted the performance comparison between the proposed method and POS by using two public databases, UBFC-RPPG³² and PURE.³⁰

2.

Materials and Methods

2.2.

Extracting Raw Signal in sRGB Color Space from the Video Data

The process for extracting the raw signal of the sRGB color space from the video data can be divided into two steps: (1) detection and tracking of the region of interest (ROI) and (2) spatial averaging.

First, the subject’s front face was detected using a detector³³ that combines the modified linear support vector machine and the histograms of oriented gradient, after which the predictive model ( http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2) that provided 68 facial landmarks was applied to the detected front face to obtain the ($x,y$) coordinates of the subject’s facial landmarks in the different postures. In this paper, considering the fact that forehead and cheeks in the exposed facial skin have more pulse information,¹⁷ we selected both cheek areas enclosed by 2, 3, 4, 5, 31, 35, 11, 12, 13, 14, and 27 among the estimated 68 facial landmarks as the ROI (see Fig. 1).

Fig. 1

Selection of the facial ROI: (a) arrangement of 68 facial landmarks and (b) participant’s facial ROI.

Second, the raw RGB signals ${\mathit{C}}_{\mathrm{PPGI}}^i(t)$ were obtained by performing the spatial averaging for each fame through the individual R, G, B channels of the selected ROI and concatenating them according to the frame order.

Eq. (1)

$${\mathit{C}}_{\mathrm{PPGI}}^i(t)=\frac{\sum _{x,y\in {\mathrm{\Omega }}_{\mathrm{fROI}}}{\mathit{C}}^i(x,y,t)}{|{\mathrm{\Omega }}_{\mathrm{fROI}}|},i\in \{\mathrm{R},\mathrm{G},\mathrm{B}\},$$

where ${\mathit{C}}^i(x,y,t)$ is the pixel value at the coordinates ($x,y$) in the $i$ channel at the time $t$ and $|{\mathrm{\Omega }}_{\mathrm{fROI}}|$ represents the area of the facial ROI. Like other sensors, the sensors for the sRGB color camera also cause the quantization noise. If a sufficient amount of the sensors for the RGB camera contributed to obtaining a facial ROI image, this quantization noise can be neglected by the spatial averaging in Eq. (1).

3.

Results

In this paper, the public databases UBFC-RPPG and PURE were used to compare the combination methods mentioned in Sec. 2.3. First, the raw RGB signals were extracted by the method mentioned in Sec. 2.2, then the signals in the different color spaces were obtained by using the combination methods mentioned in Sec. 2.3, after which the SNRs of the signals were calculated by the method mentioned in Ref. 26. Finally, the performance comparison of the combination methods was conducted by using the average SNR and the box-plot of SNR. In the same way, POS and modified POS were compared. The equation for calculating the SNR of the signal described in Ref. 26 is as follows:

3.1.

Comparison of Combination Methods

Figures 2 and 3 show the box-plots of the SNRs for the pulse signals obtained by each combination method mentioned in Sec. 2.3, for the public databases UBFC-RPPG and PURE, and Table 1 lists their average SNRs. As can be seen in Table 1 and Fig. 2, for the public database UBFC-RPPG, the combination of Cb and Cr in the YCbCr color space (i.e., Cb+Cr), P_C combination in the temporally normalized RGB color space, G channel (${\mathrm{G}}_n$) in the intensity normalized RGB color space have better SNR than the other combinations, where their average SNRs are 2.10, 2.09, and 1.99, respectively. Meanwhile, from Table 1 and Fig. 3 for the public database PURE, it can be noted that the values of average SNRs for the signals obtained by Cb+Cr combination in the YCbCr color space and P_C combination in the temporally normalized RGB color space are 5.41 and 5.02, respectively, which indicates that they have better SNRs than the others.

Fig. 2

SNR comparison of the color channel combinations for the public database UBFC-RPPG.

Fig. 3

SNR comparison of the color channel combinations for the public database PURE.

Table 1

Comparison of the average SNRs for the color channel combinations.

	R	G	B	G_B	P_C	Gn	Q	hue	Cb	Cr	Cb+Cr	a*	b*
UBFC-RPPG	$-4.27$	$-0.78$	$-6.18$	0.72	2.09	1.99	0.84	$-5.99$	$-0.51$	$-0.41$	2.10	1.63	$-1.39$
PURE	$-0.57$	1.00	$-3.68$	4.00	5.02	4.79	$-3.12$	4.23	3.01	$-2.34$	5.41	4.59	$-0.60$

3.2.

Comparison of POS and Modified POS

Figure 4 depicts the box-plots of the SNRs for the pulse signals obtained by POS and modified POS for the public databases UBFC-RPPG and PURE, and Table 2 lists their average SNRs. As can be shown in Table 2 and Fig. 4, the average SNRs of the modified POS for the two public databases UBFC-RPPG and PURE are 4.61 and 6.18, respectively, and the average SNRs of POS for the same two public databases are 3.54 and 2.51, respectively, which points out that the performance of the modified POS is better than that of POS.

Fig. 4

SNR comparison of POS and modified POS: (a) UBFC-RPPG and (b) PURE.

Table 2

Comparison of the average SNRs for POS and modified POS.

	POS	Modified POS
UBFC-RPPG	3.54	4.61
PURE	2.51	6.18
Overall	3.03	5.13

Overall: mean value of the average SNRs of the two public databases for the compared methods.

In Fig. 5, the signals extracted by applying POS and modified POS to the data of the 35th participant in the public database UBFC-RPPG are plotted in the time and frequency domains. As can be seen in Fig. 5, it is obvious that the performance of the modified POS is better than that of POS.

Fig. 5

Comparison of the pulse signals extracted by POS and modified POS in the time and frequency domains: (a) POS and (b) modified POS.

4.

Discussion

In this section, we comparatively and comprehensively analyze the methods of combining the color channels in the different color spaces and the proposed method through the results in Sec. 3. We also describe our opinions, the limitations of the proposed method and the future research directions.

In this paper, research was conducted in two areas. First, using two public databases, UBFC-RPPG and PURE, the effects of the color spaces and the color channel combinations on the measurement of the vital signs were analyzed through a comparative experiment of 13 single color channels and color channel combinations showing the relatively good HR estimation performances. As can be seen in Table 1, Figs. 2 and 3, the average SNRs of the G channel in the sRGB color space were $-0.78$ and 1.0, which indicated that G channel contained the most pulse information, followed by R and B. This result is the same as Ref. 15. The signals obtained by the chrominances (i.e., G_B and P_C) in the temporally normalized RGB color space had better SNRs than all channels in the sRGB color space. In particular, the average SNRs of the pulse signals extracted by the P_C combination were 2.09 and 5.02, which were significantly more improved than the G channel of the sRGB color space. In fact, in the temporally normalized RGB color space, the P_C color vector was one that lies in the intersection between the POS plane and the CHROM plane, so the P_C combination could reduce the artifacts to a certain degree. In addition, in the intensity normalized RGB color space, the average SNRs of the G channel signals were 1.99 and 4.79, which were significantly improved compared to the R, G, and B channels of the sRGB color space. This is because intensity normalization reduces the effect of global illumination changes and increases the effect of color hue.³⁰ The signal obtained by the pixel quotient was slightly improved with an average SNR of 0.84 in the public database UBFC-RPPG, but on the contrary, the quality was deteriorated to $-3.12$ in the public database PURE. This implies that the performance of this method depends on the measurement conditions such as environment, camera, etc. In the YCbCr color space, a certain amount of the pulse information was included in the Cb and Cr channels. Furthermore, the average SNRs of Cb+Cr were 2.1 and 5.41, which could be considered that the quality of the signal was significantly improved. This is related to the fact that the pulse information is included in both the Cb and Cr channels, and at the same time, the artifacts in both channels have the anti-phase nature. In the CIE Lab color space, the average SNRs for $a*$ channel were 1.63 and 4.59, which meant that $a*$ channel had more pulse information than $b*$. This is due to the fact that blue and yellow in $b*$ have a small skin penetration depth, so the performance is lower than $a*$. This result is the same as Ref. 20. Comprehensive analysis of the color channels and the combination methods mentioned in this paper shows that for both of the public databases, the signal obtained by the Cb+Cr combination has the most pulse information, followed by P_C. The SNRs of ${\mathrm{G}}_n$ and $a*$ are also relatively good. On the other hand, the performance of the hue channel is relatively good with an average SNR of 4.23 for the public database PURE, but on the contrary, it is very poor with an average SNR of $-5.99$ for the public database UBFC-RPPG, which indicates that the hue’s performance varies depending on the measurement conditions. Second, the improvement for the quality of the pulse signal was analyzed based on the dual-color space transformation and the compensation of the two color vectors. Obviously, the SNR of the pulse signal was significantly improved through the transformation from the sRGB space to the intensity normalized RGB color space (i.e., scaling), and the Cb+Cr combination in the YCbCr color space had the best SNR among the 13 combination methods. Meanwhile, in the YCbCr color space, the CbCr plane is one orthogonal to the normalized skin tone direction, which has the same characteristics as the POS plane. In consideration of these characteristics, we proposed a modified POS that combines the dual-color space transformation (sRGB space -> intensity standardized RGB space -> YCbCr space) and the fine-tuning, and compared its performance with POS. As can be seen in Table 2 and Fig. 4, the average SNR of the modified POS for all used databases was 5.13, which was better than POS with the average SNR of 3.03. In particular, the signal quality was significantly improved in the database PURE considering the motion, which implies that the effect is more remarkable in the condition that the artifacts exist.

To fairly evaluate the proposed method and other previous methods, we only described the core algorithm for the HR measurement but did not discuss the pre-processing or post-processing algorithms. In addition, only two relatively representative public databases were used for the comparative experiments, but it would be better to use the databases considering the more complex motion artifacts, the various illumination intensities and the dynamic illumination variations with the several kinds of cameras so that the comparative analysis gets more universal results. However, as we know, there is no public database that takes into account all of these possible practical conditions, and we will proceed with the more detailed measurement experiments in the future. Also, since each color space has the different characteristics, vital signs will be estimated more accurately if data fusion technology is applied based on the concrete analysis of the different color spaces using the sufficient databases.

5.

Conclusion

In this paper, the methods of combining the color channels based on the prior knowledge of the color vectors were comprehensively compared by using two public databases, UBFC and PURE, and a modified POS was proposed. The comparison results showed that a simple color space transformation or a color channel combination could significantly improve the signal quality. In addition, it was verified that the Cb and Cr channel in the YCbCr color space could replace the two projection axes of the conventional POS, and the average SNR of the modified POS for the all databases was 69.3% improved compared to POS. At the same time, unlike the methods that do not use prior knowledge about color channel vectors such as BSS, the proposed method has the advantage of the lower calculational complexity. These advantages of the proposed method will be of great help in the practical use of the PPGI-based measurement for vital signs.