Title Page

Contents

Abstract 14

1. Introduction 16

2. Related Works 24

2.1. Skin Surface-Related Biosignals Measurement 30

1) Contact devices based Skin surface-Related Biosignals Measurement 30

2) Non-Contact Skin surface-Related Biosignals Measurement 31

2.2. Cardiovascular-Related Biosignals Measurement 34

1) Contact devices based Cardiovascular-Related Biosignals Measurement 35

2) Non-Contact Cardiovascular-Related Biosignals Measurement 40

2.3. Respiratory-Related Biosignals Measurement 46

1) Contact devices based Respiratory-Related Biosignals Measurement 46

2) Non-Contact Respiratory-Related Biosignals Measurement 51

2.4. Mental-Related Biosignals Measurement 55

1) Contact devices based Mental-Related Biosignals Measurement 55

2) Non-Contact Mental-Related Biosignals Measurement 59

3. Pre-processing for Biosignal Measurement 62

3.1. Face Detection Stabilization 62

3.2. Optimal ROI Determination 68

3.3. Color System Conversion 70

4. Biosignal Measurement and Diagnosis using Facial Skin Images 74

4.1. Skin Surface Biosignal Measurement and Diagnosis 82

4.1.1. Skin Moisture Measurement Method 82

4.1.2. Skin Oil Measurement Method 83

4.1.3. Skin pH Measurement Method 84

4.1.4. Skin Temperature Measurement Method 85

4.1.5. Body Temperature Measurement Method 87

4.1.6. Skin Surface Biosignal Diagnosis 88

4.2. Cardiovascular Biosignal Measurement and Diagnosis 89

4.2.1. Pulse wave Extraction Method 90

4.2.2. Pulse Rate Measurement Method 94

4.2.3. Deep Learning based Pulse Rate Measurement Method 96

4.2.4. Blood Pressure Measurement Method 97

4.2.5. Paradoxical Pulse Measurement Method 106

4.2.6. Blood Volume Measurement Method 108

4.2.7. Hemoglobin Measurement Method 109

4.2.8. Blood Glucose Measurement Method 111

4.2.9. Blood Viscosity Measurement Method 113

4.2.10. Cardiovascular Biosignal Diagnosis 114

4.3. Respiratory Biosignal Measurement and Diagnosis 116

4.3.1. Respiratory signal Extraction Method 116

4.3.2. Respiration rate Measurement Method 119

4.3.3. Lung Function Measurement Method 121

4.3.4. Oxygen Saturation Measurement Method 126

4.3.5. Respiratory Biosignal Diagnosis 128

4.4. Mental Biosignal Measurement and Diagnosis 129

4.4.1. Stress Index Measurement Method 130

4.4.2. Depression disorder Index Measurement Method 136

4.4.3. Dementia Index Measurement Method 140

4.4.4. Mental Biosignal Diagnosis 141

5. Experiments and Implementations 142

5.1. Skin Surface Biosignal Measurement Experiment 143

5.1.1. Experimental results of Skin Moisture/Oil/pH Measurement 143

5.1.2. Experimental results of Skin Temperature Measurement 143

5.1.3. Experimental results of Body Temperature Measurement 144

5.2. Cardiovascular Biosignal Measurement Experiment 145

5.2.1. Experimental results of Pulse rate 145

5.2.2. Experimental results of Pulse rate using CNN & RNN 152

5.2.3. Experimental results of Blood pressure 153

5.2.4. Experimental results of Paradoxical pulse 158

5.2.5. Experimental results of Blood volume 158

5.2.6. Experimental results of Hemoglobin 159

5.2.7. Experimental results of Blood glucose 160

5.2.8. Experimental results of Blood viscosity 162

5.3. Respiratory Biosignal Measurement Experiment 163

5.3.1. Experimental results of Respiration rate 163

5.3.2. Experimental results of Lung function 168

5.3.3. Experimental results of Oxygen saturation 171

5.4. Mental Biosignal Measurement Experiment 172

5.4.1. Experimental results of Stress Index 172

5.4.2. Experimental results of Depression Index 178

5.4.3. Experimental results of Dementia 183

5.5. Integrated-Biosignal Measurement Application 185

6. Conclusions 188

References 193

논문요약 215

Table 1. Summary of BioSignal Measurement Methods 75

Table 2. Regression Formula of Skin moisture 83

Table 3. Regression Formula of Skin oil 84

Table 4. Regression Formula of Skin pH 85

Table 5. Regression Formula of Skin temperature 86

Table 6. Regression Formula of Body temperature 88

Table 7. Skin moisture/oil/pH Diagnosis 89

Table 8. Body Temperature Diagnosis 89

Table 9. Blood volume measurement formula 109

Table 10. Regression Formula of Hemoglobin before and after meal 110

Table 11. Regression Formula of fasting(or postprandial) Blood glucose 112

Table 12. Cardiovascular Diagnosis 115

Table 13. Regression Formula of FEV and PEF (Blow) 124

Table 14. Regression Formula of FEV, FVC and PEF (Non-blow) 126

Table 15. Respiratory Diagnosis 128

Table 16. Average of LF Activity by age 132

Table 17. Pulse-based Stress Index Measurement Formula 134

Table 18. Respiratory-based Stress Index Measurement Formula 135

Table 19. Activity-based Stress Index Measurement Formula 136

Table 20. Activity-based Depression Index Measurement Formula 139

Table 21. The dementia-related frequency band 140

Table 22. Mental Diagnosis 141

Table 23. MAPE (%) of skin moisture/oil/pH 143

Table 24. MAPE (%) rate for each measurement area 144

Table 25. MAPE (%) for body temperature of skin temperature and biosignal 145

Table 26. Comparative results of the FPR on stable and unstable datasets 147

Table 27. Comparative results of the FPR on indoor dataset 149

Table 28. Comparative results of the FPR on external dataset 150

Table 29. MAPE (%) of the short-time FPR on indoor dataset 151

Table 30. MAPE (%) of the FPR according to distance 152

Table 31. Six areas of ROI 152

Table 32. Pulse rate measurement result (using CNN & RNN) 153

Table 33. Comparative results of SBP and DBP on stable and original ROI 155

Table 34. Comparative results of BP on internal dataset 157

Table 35. Comparative results of BP on external dataset 157

Table 36. MAPE (%) of gradient and blood volume 159

Table 37. MAPE (%) of hemoglobin by color data 160

Table 38. MAPE (%), MAE of blood glucose 161

Table 39. MAPE (%) of spring constant K and blood viscosity 162

Table 40. Comparative results of FRR on stable and unstable dataset 164

Table 41. Comparative results of the FRR on indoor dataset 166

Table 42. Comparative results of the FRR and other FRR on external dataset 167

Table 43. MAPE (%) of the short-time FRR on indoor dataset 168

Table 44. MAPE (%) of the lung function (Blow) 169

Table 45. MAPE (%) of the lung function (Non-blow) 170

Table 46. Average of MAPE (%) about Blow and Non-blow 171

Table 47. Comparative results of the SpO₂ 171

Table 48. Comparative results of the pulse wave-based pSI, sSI and SI 172

Table 49. Comparative results of the respiratory signal-based pSI, sSI and SI 173

Table 50. Comparative results of frontal Lobe features-based pSI, sSI and SI 174

Table 51. Classification results of Stress levels 175

Table 52. Classification Accuracy for Each Segment of Stress levels 175

Table 53. Classification Accuracy Result of Stress levels 176

Table 54. Weighted Average between Right Frontal Lobe and Respiratory Signal based Classification Accuracy for Each Segment of Stress levels 177

Table 55. Weighted Average between Right Frontal Lobe and Respiratory Signal based Classification Accuracy Result of Stress levels 177

Table 56. Classification Accuracy Results for Each Segment and All Stress Levels Using Weighted Average 178

Table 57. Comparative results of the Depression Index 178

Table 58. Comparative results of frontal Lobe features-based DDI 179

Table 59. Classification results of Depression levels 180

Table 60. Classification Accuracy for Each Segment of Depression levels 180

Table 61. Classification Accuracy Result of Depression levels 181

Table 62. Weighted Average between Right and Left Frontal Lobe based Classification Accuracy for Each Segment of Depression levels 182

Table 63. Weighted Average between Right and Left Frontal Lobe based Classification Accuracy Result of Depression levels 182

Table 64. Classification Accuracy Results for Each Segment and All Depression using Weighted Average 183

Table 65. Comparison of ISO Certification Criteria with Facial Skin Image-Based Biosignal Measurement Results 189

Fig. 1. Contact Devices-based Biosignal Measurement 25

Fig. 2. Different types of wearable devices 26

Fig. 3. Evolution of smart-phones over time 27

Fig. 4. Paradigm Transformation with the Development of Digital Video 28

Fig. 5. Biosignal Measurement Process Using Color Data of Facial and Parm skin Region of Interest (ROI) 29

Fig. 6. Examples of Wood's Lamp Usage 32

Fig. 7. Skin temperature measurement of the anterior body using Infrared Thermal Camera 33

Fig. 8. Block diagram of PPG device operation (a) A transimpedance amplifier stage that converts light intensity to an amplifier output voltage and (b) The... 36

Fig. 9. PPG signal containing Asystole event 38

Fig. 10. Pulse arrival time (PAT) and pulse transit time (PTT) 39

Fig. 11. Flowchart of the Pulse wave extraction and Pulse Rate measurement using the facial images based on ICA 41

Fig. 12. General contactless bio-signal measurement environment 42

Fig. 13. Participant posture and position of sensors to collect BP DB 44

Fig. 14. Overview of measurement of PTT and iPTT ((a) Contact method, (b) Non-contact method) 45

Fig. 15. Comparison of feature-based (red) and filter-based (green) techniques for extraction of exemplary respiratory signals during Inhalation and Exhalation 48

Fig. 16. Absorption spectra of hemoglobin 49

Fig. 17. Flowchart of the Respiratory signal extraction and Respiration Rate measurement using RGB color data of the facial images 52

Fig. 18. Flowchart of the Respiratory signal extraction and Respiration Rate measurement using RGB color data of the chest 52

Fig. 19. Illustration for traditional R and B channels method (AC components, where ACRed & ACblue are the standard deviations (SD) of the RGB data and...[이미지참조] 54

Fig. 20. Interval tachogram of 256 consecutive RR values in a normal subject at supine rest (a) and after head-up tilt (b). The HRV spectra are shown,... 57

Fig. 21. Typical Diastole series of PPG signal selected in each group(MDDSI+, MDDSI-, and CONT) 58

Fig. 22. Stress Classification using Facial Features, ECG, Voice 60

Fig. 23. Depression Classification using Pulse signal of facial images 61

Fig. 24. Expended set of Haar-like features: (a) edge features, (b) line features, (c) center-surround features. Face detection based on (d) the edge feature of... 65

Fig. 25. X-coordinate variability of ROI 66

Fig. 26. ROI stabilization process 66

Fig. 27. Calibrated x-coordinate variability of ROI (n＝15) 67

Fig. 28. (a) Blood vessel and region of interest (red rectangle) of the facial artery. (b) Detected cheek ROI (green rectangle) from the facial video 69

Fig. 29. The extracted Cg signal recorded for 60s. 73

Fig. 30. Frequency Domain Distribution for BioSignal Measurement 76

Fig. 31. Simple block diagram of skin moisture/oil/ph analysis 82

Fig. 32. Simple block diagram of skin temperature 86

Fig. 33. Simple block diagram of Body temperature analysis 87

Fig. 34. Flowchart of Pulse wave Extraction 90

Fig. 35. Pulse-Frequency domain of the Cg signal (Fig. 29) 91

Fig. 36. Illustration of pulse wave extraction process 93

Fig. 37. Pulse wave comparison(PPG(blue), skin image(red)) 93

Fig. 38. Peak detection of pulse wave 94

Fig. 39. RR interval calculation of the pulse wave 95

Fig. 40. CNN based pulse rate classification pipeline 96

Fig. 41. RNN based pulse rate estimation pipeline 97

Fig. 42. Flowchart of systolic & diastolic blood pressure measurement 98

Fig. 43. Selection of two different ROIs for blood pressure measurement 99

Fig. 44. Peak detection and phase extraction of roi₁ and roi₂ 100

Fig. 45. Relationship dataset collection between the distance of camera-user and the detected face width 101

Fig. 46. Regression Analysis of Face width and Pixcel Distance 102

Fig. 47. Distance between roi₁ and roi₂ 103

Fig. 48. baPWV measurement 104

Fig. 49. PTT calculation using Pulse wave of roi₁ and roi₂ 105

Fig. 50. Pulse Wave Analysis in Inhalation and Exhalation Intervals 107

Fig. 51. Gradient calculation process 108

Fig. 52. Simple block diagram of Hemoglobin analysis 109

Fig. 53. Simple block diagram of Blood glucose analysis 111

Fig. 54. Blood viscosity calculation process 113

Fig. 55. Flowchart of Respiratory signal Extraction 117

Fig. 56. Respiration-Frequency domain of the Cg signal (Fig. 29) 118

Fig. 57. Illustration of respiratory signal extraction process 119

Fig. 58. Peak detection of respiratory signal 120

Fig. 59. PP interval calculation of the respiratory signal 120

Fig. 60. Simple block diagram of FVC, PEF, FVC(FEV₁, FEV₂ and FEV₃) analysis (Blow) 123

Fig. 61. Simple block diagram of FVC, PEF, FVC(FEV₁, FEV₂ and FEV₃) analysis (Non-blow) 125

Fig. 62. Flowchart of Activity Calculation 130

Fig. 63. A comparative analysis of changes in autonomic nervous system activity between healthy people (left) and depressed patient (right) 137

Fig. 64. Analysis of PCA application of frequency data for normal people and dementia patients (x-axis real part, y-axis imaginary part) 141

Fig. 65. Comparison of FPR calculation results by stable and unstable methods 146

Fig. 66. Graphs comparing FPR (Proposed) with other FPR methods (STFT, ACF, ICA-BPF) 148

Fig. 67. Comparison of BP calculation results by stable and original ROI 154

Fig. 68. Comparison of BP measurement results using PWV and PTT 156

Fig. 69. Comparison of FRR Results by face detection stabilization 163

Fig. 70. Comparison of FRR (Proposed) and other FRR (STFT, ACF, ICA-BPF) Graph 165

Fig. 71. Dementia index (x-axis) and correlation coefficient (y-axis) calculated for normal people and dementia patients 184

Fig. 72. Complexion state app (Measurement result screen) 185

Fig. 73. Lung function Measurement (Blow) 186

Fig. 74. Chart and management of Measured biosignal 187