The Study of Facial Muscle Movements for Non-Invasive Thermal Discomfort Detection via Bio-Sensing Technology. Part I: Development of the Experimental Design and Description of the Collected Data

Abstract

In the time of climate change, as heat waves become a more regular occurrence, indoor thermal comfort is an important factor in day to day life. Due to such circumstances, many researchers have focused their studies on finding an effective solution that will not only enable thermal comfort, but also increase satisfaction within the indoor environment and, as a result, productivity. The fast development of the biometrical field encouraged the study focused on the investigation of how bio-markers, in combination with artificial intelligence algorithms, can be collected within an experimental setting to create a new approach for non-invasive thermal discomfort detection. The developed experimental design provides synergy between automatic facial coding, pulse, and galvanic skin response measurements via iMotions software in a controlled environment. The iMotions software has built-in machine vision algorithms, and with Shimmer sensors and a post-processing tool through Affectiva AFFDEX, is able to collect facial action data through detection of the facial muscle movements and various bio-markers. The Zero Emission Building (ZEB) Test Cell laboratory was used as the control environment and transformed to imitate an office space for the data collection campaign at NTNU in Trondheim. The given experimental design provides an opportunity to create an immense database with bio-markers that are linked to the subcortical level of the brain, indoor parameters, and direct feedback on the comfort level of occupants within an office-like environment. In total, 111 data collection sessions were registered with iMotions. The discomfort button was pressed 240 times and 1080 planned indoor comfort evaluation surveys were held during experiment. The discomfort button was pressed 49 times to indicate that participant felt discomfort due to low temperature and 52 due to high temperature. Collected data revealed a big deviation in the discomfort temperature values for experiment participants with respect to performed temperature ramps. While it is common to use the same predefined temperature range for facility management, it became clear that the complexity of the task is greater and should not be approached on a human computational level. Implementation of AI can potentially provide higher value accuracy within thermal discomfort detection and enable unique personal user experience at the workplace.

1. Introduction

The indoor environment (IE) has become the leading space where people spend their time. IE can provide protection from outdoor conditions and prevent overheating or overcooling. Indoor Environmental Quality (IEQ) is a general criterion for IE and has a direct link to occupant satisfaction with indoor parameters (e.g., temperature, visibility, levels of background noise) [1]. A number of studies have proven that there is a strong association between IEQ and the health + productivity of the building occupants [2,3,4,5,6]. Furthermore, effective IEQ may reduce energy consumption and prevent peak loads in the grid [7,8,9,10].

The following research is focused on development of the experimental design for the indoor thermal comfort (ITC) measure of IEQ. According to ASHRAE Standard 55, thermal comfort (TC) can be defined as a condition of mind that expresses satisfaction with the thermal environment and is assessed by subjective evaluation [11]. However, an occupants’ perception of the thermal environment differs since ITC is determined by a number of personal parameters, which define heat interchange within a person and the built indoor conditions [12]. Variables such as age, activity level, clothing insulation, humidity and other influences have a direct influence on the TC of a person [12,13,14].

The ITC is usually evaluated by the following indices: Predicted Mean Vote (PMV) and Predicted Percentage of Dissatisfied (PPD) [15]. The PMV model proposed by Fanger contains a seven point scale, which is also attributed as the seven-point ASHRAE thermal sensation scale [15] (see Figure 1).

The PPD represents a quantitative measurement of the TC for a number of people within a defined temperature zone [16]. Both scales were adopted by ASHRAE Standard 55, ISO 7730 and other international standards [11,17].

With technological innovation and the development of new algorithms [18], non-invasive bio-sensing approaches for characterisation of the personal TC have been investigated in a number of studies [19,20,21,22,23,24,25]. The majority of these studies used thermal cameras or regular cameras in combination with post-processing algorithms to extract local body temperature data from the images/videos. The given temperature values were combined with other indoor/outdoor parameters and the results from comfort surveys. In addition to skin temperature data, a number of biometrical parameters were collected such as: Heart Rate, Body Mass Index (BMI), Sweat Rate, Electroencephalogram (EEG), Electrocardiogram (ECG) and various other data (see Figure 2).

All of the data collected was used in varying combinations as sets of predictors under the assumption that they can potentially reveal the thermal state of a person. The k-Nearest Neighbor, Support Vector Machine, Random Forest and many other supervised learning algorithms were implemented to build non-invasive TC prediction models, whose target functions are to predict the state of a person’s comfort/discomfort.

A number of studies have presented limitations with respect to the volume of collected data points, sufficient number of participants in the sample group and other issues associated with the creation of an appropriate database. In addition, only a few articles discussed their approach with significant problems like overfitting during implementation of the machine learning algorithms [19,27,28,29,30,31]. A combination of the following factors had a major part in the prevention of the model’s deployment in real life. As it was elaborated in Marchenko and Temeljotov-Salaj [26], there is a great opportunity to utilize a novel equipment, in combination with artificial intelligence algorithms to develop non-invasive methods for thermal discomfort detection. Literature review suggested that it can be potentially helpful to use the connection between the thermoregulation of the body and its interaction with the subcortical level of the brain (or otherwise known as the lower brain) [26]. The overall research was divided into two stages: the thermal comfort study; the study of facial muscle movements for non-invasive thermal discomfort detection via bio-sensing technology (see Figure 3).

Based on the aforementioned arguments, the given paper deals with Part I: Stage 2, and seeks to answer the following question: How can biometrical data, in combination with artificial intelligence algorithms, be collected within an experimental setting to design a new approach for non-invasive thermal discomfort detection?

2. Experimental Design Approach

Based on the outcomes from a literature review within the field [26], it was determined to focus on the bio markers which are associated with the subcortical section of the brain. One of the common visible indicators for the general emotional state of a person, is a facial expression [26]. The facial muscle movements are controlled by the facial nerve, which is directly connected to the brainstem and motor cortex [32,33,34,35]. Since the brainstem belongs to the subcortical part of the brain and it is responsible for involuntary muscle movements, it can be inferred that involuntary muscle movements may also contain information with respect to a person’s TC or point out the events of thermal discomfort.

The amygdala controls the release of stress-related hormones (such as cortisol, thyroid hormone and corticotropin-releasing hormone) into the blood stream. The heart rate, skin conductance, respiration, and other observable behaviours are also controlled by the amygdala’s functioning.

Facial expressions are generated by facial muscle movement. The facial nerve is connected to the brainstem and motor cortex [36]. It also branches out to different muscles located on the face. Traditionally, emotions are divided by two categories: voluntary and involuntary [37,38,39]. Since the brainstem is the oldest part of the brain area—it is responsible for involuntary/unconscious facial expressions and the motor cortex is specifically active in consciously controlled/intentional facial expressions.

In 1988, the facial feedback hypothesis was proposed by Fritz Strack [40,41,42] and his research team:

“The facial feedback hypothesis postulates that selective activation or inhibition of facial muscles has a strong impact on the emotional response to the stimuli”

Based on this, people have a number of facial expressions which can be correlated with different types of emotions [43,44,45]. Moreover, there is is an uncountable number of different facial expressions, only the following categories of emotions may be tracked at current stage of development within field: joy, anger, surprise, fear, contempt, sadness and disgust [46]. The given categories of emotions do not depend on age, ethical background or other parameters [47,48].

The facial expressions can be collected and processed [49,50,51] via Facial electromyography (fEMG), The Facial Action Coding System (FACS) [47,52] and Automatic Facial Coding (AFC) [53,54,55,56]. After all benefits and limitations were evaluated (see

Table 1. Benefits and limitations of the methods for facial expressions processing.

Method	Benefits	Limitations
fEMG	non-invasive method	needs electrodes and other equipment
	has no language dependency	electrode placement may contribute to experiment subject awareness regarding measurements being collected
	doesn’t require cognitive effort	sensitive to motions
	able to measure even subtle facial muscle activity	analysis is expensive because it can only be performed by a specialist with biosensor processing skills
FACS	very reliable	requires high quality video resolution
	non-intrusive	requires a trained expert to perform score evaluation
	contains a 5 step intensity rating	expensive with respect to the time needed for expert video processing
AFC	reliable	requires high quality video resolution
	non-invasive	requires acomputer vision specialist who is also familiar with FACS
	easy to use	requires a number of pre- and post-processing stages

), the AFC method, which is based on artificial intelligence algorithms was chosen for collection of the facial muscle movement data.

The quantitative method was chosen based on the objectives of previous research, which provided a foundation for the framework of facial muscle movements data collection in combination with other biometrical parameters for potential non-invasive thermal discomfort detection at the workplace.

Experimental design for the given research should fulfill the following steps:

• Define the research problem and research questions
• Define the population of interest
• Describe the needs for sampling
• Design of the experiment

Since the main goal of experimental design is to make sure that data collected during the experiment will answer predefined research question(s), it is important to define those questions in advance [57].

The next step of a successful experimental design is to define the population that fits the purpose of the research. The sample from which data is going to be collected should, as much as possible, reflect variability within the actual population [58].

The experimental design should be seen as a model, which consists of controlled, uncontrolled and fixed variables. The given model should be developed in a way that best fits the research objective. The anticipated result is to create a layout of the design, which will reflect all important structural components and the envisioned data analysis.It should also contain the following sections: definition of the experimental unit, variable types and design structure. The experimental unit, or as it is also called a sampling unit, is the smallest unit of the analysis for which data is going to be collected within an experiment. The data collection campaign should consider and clearly define the main four variable types: background, constant, uncontrollable and primary. Background variables are those which can be measured, yet can not be controlled. As opposed to constant variables, which can be controlled and measured. These variables are going to be held constant throughout the data collection campaign due to reasons predefined by the scope of the study. Primary variables are independent variables which are a possible source of variation—they are controlled by the research.

The data collection equipment should be placed in the most efficient locations and should not be moved throughout the duration of data collection. Equipment should be configured to the predefined manufacturer settings as most appropriate.

The treatment structure should account for a number of factors that will be studied and the conclusion(s) that will be performed. In the majority of cases, the experimental design needs experimental units to be divided into treatments either randomly or randomly with constraints.

Data collection should strictly follow a defined protocol. The created data collection protocol should be detailed, structured and easy to follow since the researcher(s) will usually be collecting data with laboratory workers and in scheduled shifts. So, it is essential to check that data is consistently collected and not reshaped.

The last section of the experimental protocol should contain a list of assumptions and limitations within the study. Each possible source of error should be discussed and evaluated.

3. Methodology

The given research is focused on designing the proper experiment for utilization of biometrical data in combination with artificial intelligence, in order to provide a new approach for non-invasive thermal discomfort detection at the workplace.

The study was performed at The Norwegian University of Science and Technology (NTNU). Due to the need of bio metrical data collection, a notification form/request was submitted to the Norwegian Center for Research Data (NSD). Each NSD application should contain information about what type of data will be collected and how it will be saved. It is important to be conscientious about the collection of personal data. It should be stored safely and encrypted in order to prevent links to the internet or access by third parties, which might make it public. Each type of personal data should be discussed and the need for collection should be proved. The research team applied to NSD when the final version of the experimental design was completed. The ethical committee evaluated application within a few weeks and approved it. The trial data collection sessions started in September 2019.

Several sensing modalities, including the automatic facial expression detection, pulse monitoring, and galvanic skin response, were introduced and included in this experiment. The collected database is going to be preprocessed for better understanding of its variance, elimination of the missing data and bugs which can bring bias into the future data processing. In order to see whether there are clear patterns in facial muscle movements with respect to temperature profile, unsupervised learning cauterization will be performed. Based on the cauterization results, the collected biomarkers will be synergistically used to monitor the subject’s thermal discomfort by creating a feature vector. Afterwards, each feature vector is going to be paired with a recorded outcome (e.g., moment of discomfort). As result, sorted database will be perfect for any algorithm which trains on supervised learning principle. General framework of the initial data flow and data types presented at Figure 4.

3.1. Sampling Strategy

To define an appropriate dimension for the sample, it was decided to refer to the statistics accumulated from a Systematic Literature Review [26] within the field (see

Table 2. Overview of the amount of people involved in each processed study and their age.

Reference:	Publication Type	Year	Number of People	Age
Cheng et al. [19]	Journal article	2019	16	20–29
Ueda et al. [61]	Journal article	1997	62	22–36
Cheng et al. [62]	Journal article	2017	16	20–29
Ueda et al. [63]	Journal article	1997	11	29–41
Matalucci et al. [64]	Journal article	2017	12	18–36
Chaudhuri et al. [27]	Conference Paper	2018	20	21–25
Bermejo et al. [65]	Journal article	2012	3	NaN
Lee et al. [66]	Journal article	1998	13	NaN
Lopez et al. [20]	Conference Paper	2018	1	NaN
Lopez et al. [67]	Conference Paper	2016	5	20–29
Zhai et al. [68]	Conference Paper	2017	20	21–26
Vesely and Zeiler [24]	Conference Paper	2014	6	NaN
Li et al. [23]	Journal article	2019	10	NaN
Ghahramani et al. [69]	Journal article	2016	15	NaN
Salamone et al. [70]	Journal article	2018	8	33–61
Cosma and Simha [28]	Journal article	2019	24	NaN
Katić et al. [29]	Journal article	2018	2	29
Li et al. [30]	Journal article	2018	12	22–27
Chaudhuri et al. [71]	Journal article	2018	20	21–25
Pavlin et al. [72]	Journal article	2017	10	27–28
Yang et al. [73]	Conference Paper	2019	22	23–26
Choi and Yeom [74]	Journal article	2017	15	30–39
Barrios and Kleiminger [75]	Conference Paper	2017	7	23–30
Cosma and Simha [76]	Journal article	2018	30	20–42
Lu et al. [31]	Journal article	2019	2	NaN
Ghahramani et al. [77]	Journal article	2018	10	NaN
Burzo et al. [78]	Conference Paper	2014	14	22–35

). The average number of participants per study was 14. Even though women are assumed to express facial actions more often than men [59], the study performed by McDuff et al. [60] showed that general expressiveness is predefined by emotional status of the person. Due to described conditions, it was agreed to not differentiate gender of participants and in general to make sure that there is not less than 14 participants involved in the study.

Participation in the test cell experiment was voluntary. Posters with an invitation to attend the experiment for a minimum of 1 session and a maximum 8 sessions were disseminated across the University campus. Scheduling was also flexible. Each participant could agree to a suitable time to fit into their schedule. In general, the main concern was that participants should not be scheduled more than 2 times in a row to avoid generic answers.

3.2. Experiment Location and Facility

In order to recreate an office-like environment, the experiment was conducted at SINTEF (SINTEF is one of Europe’s largest independent research organisations, which has developed solutions and innovation for society and customers all over the world. For more information please follow the link: https://www.sintef.no/en/this-is-sintef/)

and NTNU’s Zero Emission Building (ZEB) Test Cell Laboratory on the Gløshaugen campus in Trondheim [79]. This is a PASSYS instrumented test cell that is comprised of two thermal chambers (Chamber A and Chamber B) inserted in the same guard room, which have one wall hosting a non-operable window exposed to the outdoor climate. Both chambers have internal dimensions of approximately 4.36 m × 2.50 m × 3.39 m [width × depth × height]. The interior of the test cell is furnished with typical office furniture: desk, a chair, a mouse and a large PC monitor, which served as the main work screen for participants. For our experiment a Chamber B was used (see Figure 5).

The detailed overview of the materials used to construct ZEB laboratory and their specifications are presented in work of Cattarin et al. [79]

4. Results

Based on the literature review in the field and chosen methodology, the following experimental design is developed as a model which contains controlled, uncontrolled, fixed and blocked variables. Detailed specification of the variables are described in the following steps:

• Step: preparation
  One subject is located in the test cell with thermally comfortable conditions for 1 h. Meanwhile instruments are going to be assembled on body parts. Administration of the Preassessment thermal comfort questionnaire. Data collection on:
  • Comfort assessment questionnaire; CLO; Fat percentage % Height [m]; Weight [kg]; Age; Gender; Amount of time spent in Norway
  Instruction of the experiment subjects and signing of the consent form.
• Step: experiment
  • Air velocity (fixed variable); Relative humidity (no control); Black globe temperature [deg.C] (variable); Air temperature (degrees C) (variable); Galvanic skin response (variable); Hart rate (variable); Facial Expressions (variable); Event of discomfort
• Blocked variables:
  • CLO; Position of the desk

4.1. Facial Mapping and Bio-Sensing Equipment

The iMotions software was chosen to facilitate the AFC method for facial muscle movement data collection. It is a facial muscle movement tracking software (see Figure 6) that is able to recognize different emotions of a person by collecting data on each involuntary facial expression.

To get more precise data from the iMotions software we used high-definition camera Logitech C925e Webcam (Height × Width × Depth: 1.2 in (29 mm) × 5.0 in (126 mm) × 1.3 in (32 mm)) for accurate processing of the input video. The camera is placed on 60 cm distance to participant’s face. The face and position detection is performed via face detection algorithms (such as SIFinder (SIF) or University of Surrey (UniS) and others [80,81,82]) when image/video of the person is acquired (see Figure 7). After the face is detected and allocated into the so-called “box”, the feature extraction algorithm performs marking of the facial landmarks (e.g., nose, corners of the eyes and mouth etc.) and classifies them. Afterwards, all facial landmarks are fed into classification algorithms as inputs which decode features into Action Code units. The overview of the graph with decoded features is presented at Appendix A: Figure A2.

While main parameters of the Shimmer (level of the charge, intensity of signal and other) are presented on the left side of software interface, the live data flow from the sensors is presented in form of the graphs on the right side.

The Affectiva AFFDEX is a part of iMotions’ software and is used for post-processing of the collected data (see Figure 8). The given program is able to detect 7 basic emotions including: valence, engagement, attention, joy, surprise, fear and anger [83,84]. The Affectiva has 14 facial expression metrics and is able to detect head orientation, which contributes to detection of yaw, pitch, roll and others.

Another biometrical piece of equipment is the Consensys GSR Kit, which contains an Optical Pulse Sensor Ear Clip (earlobe), two GSR Finger Electrodes, two Biophysical Leads, one Optical Pulse Sensor (finger), a Wrist Strap and the Shimmer3 GSR (see Figure 9).

The main function of the GSR unit is to measure galvanic skin response, which is also known as electro-dermal resistance or skin resistance, between two reusable electrodes attached to the fingers of one hand. Another electrode, which is also connected to the Shimmer device, is an optical pulse sensor that can be placed on the finger or earlobe. It is preferred that any person who participates in the experiment, uses slow hand movements in order to prevent sensors from moving and creating outliers in the data set. An example of sensor placing can be observed in Figure 10.

4.2. Monitoring of IE Parameters

The Test Cell control and monitoring system is a custom made Real Time LabVIEW application, using cRIO technology from NI. It allows for monitoring of about 500 parameters, such as temperatures, CO₂ level, indoor lighting, outdoor weather and much more. Data can be seen in real time on the computer screens in the experiment control room. Data, both measured values, set points and derived values is stored in a database, as well as stored locally.

The air temperature inside test cell was measured by three Pt 100 sensors, which were placed on adjustable tripod. Given sensors can measure temperature on different levels for better monitoring of the temperature next to the experiment participant (see Figure 11 ). Pt 100 has measurement range from −5 ${}^{\circ }$C to 60 ${}^{\circ }$C and accuracy from −0.3 ${}^{\circ }$C to +0.3 ${}^{\circ }$C. The globe sensor was placed in the center of test cell to measure operative temperature. Given sensor is also Pt 100 and has same accuracy range as sensors placed on tripod.

By the sensors which were installed at the wall we measured relative humidity, air temperature and CO₂ (Figure 12). The indoor relative humidity was measured with range from $0\%$ to $100\%$ and accuracy from $-5\%$ to $5\%$. Temperature measured with range from 0 ${}^{\circ }$C to 50 ${}^{\circ }$C and accuracy from −8 ${}^{\circ }$C to +8 ${}^{\circ }$C.

4.3. Equipment Installation and Utilization during the Experiment

All communication with the participant in the test cell was performed via Skype messenger using a PC which was located in the right corner of the worktable. Participants had an open Skype window with chat options, so they could communicate in case they had any questions or they needed to have a toilet break. The same computer was used to present the participant with surveys, and featured a red button that the participant could click to stop the experiment due to discomfort. Reports from surveys were stored in an organized structure with timestamps and ID numbers.

The experiment control room is situated next to the guard room of the ZEB Test Cell laboratory (see Figure 13). It is equipped with a main computer, used for controlling and monitoring the guard room and both cells. Data from both cells can be monitored from here, and historic data can be accessed using the database. An additional PC was installed to operate the other data collection software such as Shimmer and iMotions.

Visualising data from the cell in real time proved important for the operator of the experiment, for better control of ramp parameters. A researchers view of the data was developed to facilitate this. A screenshot of this can be seen in Figure 14.

In total, 4 heating and 4 cooling ramps were created. Each ramp has its steady rate of increase or decrease in temperature per hour:

• heating 1.4 ramp = +1.4 ${}^{\circ }$C per hour
• heating 2.2 ramp = +2.2 ${}^{\circ }$C per hour
• heating 3.4 ramp = +3.4 ${}^{\circ }$C per hour
• heating 4.4 ramp = +4.4 ${}^{\circ }$C per hour

• cooling 1.4 ramp = −1.4 ${}^{\circ }$C per hour
• cooling 2.2 ramp = −2.2 ${}^{\circ }$C per hour
• cooling 3.4 ramp = −3.4 ${}^{\circ }$C per hour
• cooling 4.4 ramp = −4.4 ${}^{\circ }$C per hour

4.4. Temperature Ramps Implemented during Experiment

Operation of temperature ramps were implemented on the main computer using LabVIEW. In total, 4 heating and 4 cooling ramps were developed and randomly distributed within each experiment [11]. Normally, the temperature in the cell is regulated to reach a certain temperature, and maintain it. For this experiment a new approach was developed, where the derivative of the cell temperature was kept constant, giving us a steady rate of change (increasing or decreasing). This worked well for heating, but was more challenging when cooling. Cooling in the cell is done using a heat exchanger in the ventilation ducts, which gets cold water from a heat pump. The heat pump would not deliver a constant rate of cold water, making the regulating difficult. Since outside weather and temperature also varied throughout the experiment period, manual adjustments to ramp parameters were necessary to ensure the most ideal linear ramp.

A graph of the temperature ramp was saved for each experiment, and some samples are presented in Figure 15 and Figure 16.

4.5. Comfort Evaluation and Survey Structures

In total, 4 hard-copy surveys were developed for the given experiment: Survey1 was used to collect data on age, gender, height, weight of the person and the duration of time spent in Norway; Survey2 focused on collection of the CLO value and other parameters of indoor comfort (e.g., temperature, air humidity and other); Survey3 contained questions regarding sources of the discomfort and possibility to decrease it; and Survey4 contained questions designed to sum-up the total satisfaction experienced with the environment. Study participants were asked to answer these surveys during experiment participation.

Survey3, was generated every time the discomfort button was pressed. It contains sections dedicated to each aspect of the office environment. Each section contained 3 types of scale: (1) Thermal sensation, (2) Thermal Preference and (3) Thermal acceptability (see Figure 17). The participants were asked to mark the scale under each question with a pencil. This approach enabled all answers to be transformed into a numeric variable instead of categorical, if needed [85].

4.6. Action Protocol

An action protocol was developed in order to synchronize steps of the experiment and define what is not allowed to do in the test cell (see

Table 3. Action protocol.

Time	Procedure	Comments
8:30	Participants are expected to arrive. They are invited to the acclimatization area, asked to take off non-needed clothes i.e., jacket (it will be saved in special space during experiment).	Participants should not adjust their clothes during the experiment.
8:35	Instructions will be provided to participants. Afterwards they need to sign a consent form. IMPORTANT: Participants should NOT take part in the experiment if they do not agree to sign the consent form.
8:55	Survey is handed out to experiment participants. They have 5 min to fill in answers. In case they do not fill in all of the questions—they can take the survey to the experiment room and finalize it there. Afterwards, the survey should be placed on the corner of the desk.
9:00	Start of the experiment. Participant asked to get inside test room and start to work on his/her regular tasks. During given period it is not allowed to change clothes, open windows or walk around the room. Participant should be allocated within given experiment space.	Participants should use adjustment (discomfort) BUTTON which is situated at their tables to indicate event of discomfort.
12:30	Lunch brake. Opportunity to drink coffee will be provided. Participant can leave test space for 1 hour. During this time, window in the room will be opened to provide fresh air, completed questionnaires will be collected. It is forbidden to perform sport activities or do any other actions which increase heart rate (e.g., running, jumping or smoking).	During this period, a new study should be generated with unique ID since some participants would stay for only half of the day and a new person would arrive for the afternoon.
13:30	Lunch brake is over. Opportunity to take one cup of coffee with you and return to the test cell. It is important that person’s don’t forget to visit the WC before the experiment starts to avoid interruptions.	Participants should use the adjustment (discomfort) BUTTON which is located at their tables to indicate an event of discomfort.
16:30	The end of the experiment, opportunity to drink more coffee, finalize surveys and agree on the schedule for the next meeting. Person should not lose code which was assigned to the participant at the beginning of the experiment	All collected data within a given day of experiment should be collected on the hard drive and backup files should be generated.

).

4.7. The Overview of Collected Data

During September 2019–January 2020, 38 participants took part in the experiment, among which 29 were female and 9 male participants. The majority of participants have been living in Norway more than 5 years (see Figure 18). All participants were either students or exchange students, professors or researchers at NTNU.

The following age groups were represented:

• 20–30 years old—27 participants
• 30–40 years old—8 participants
• 40–50 years old—2 participants
• 60–70 years old—1 participant

In total, 111 data collection sessions were registered with iMotions. The discomfort button was pressed 240 times and 1080 planned indoor comfort evaluation surveys were held during experiment. In Survey1, the experiment participants were asked to rank the three physical features that in their opinion are the most important in making a workplace a pleasant one. The survey results revealed that TC and air characteristics are in the first place for a pleasant workplace environment for the majority of experiment participants (see

Table 4. Rank of the three physical features which are the most important for pleasant workspace.

Physical Feature	1st Place	2nd Place	3rd Place
ViewFromTheWindows	2	0	0
AcousticComfort	6	11	6
OfficeLayout	1	1	2
VisualComfort	4	4	5
AirQuality	14	7	7
Cleanliness	2	1	4
ThermalComfort	15	10	5
Privacy	3	1	5

).

The discomfort button was pressed 49 times to indicate that the participant felt discomfort due to low temperature and 52 due to high temperature (see Figure 19). Collected data revealed a big deviation in the discomfort temperature values for experiment participants with respect to performed temperature ramps. While it is common to use the same predefined temperature range for facility management, it became clear that the complexity of the task is greater and should not be approached on a human computational level. Implementation of AI can potentially provide higher value accuracy within thermal discomfort detection and enable unique personal user experience at the workplace.

5. Discussion

Indoor thermal comfort is a complex parameter which consists of different physiological, psychological and functional aspects. Until now, researchers have tried to develop various strategies to ensure that the indoor environment is comfortable and, as such increase the satisfaction and productivity of occupants [2,3,4]. Among those strategies, which are mainly based on direct user survey feedback, non-invasive TC detection by implementation of the bio-sensing technology provides a solution from a different angle. Several studies investigated various sets of biological parameters in combination with usual indoor/outdoor parameters as well as system parameters [19,20,22]. The skin temperature has shown to be the most frequent criterion for thermal comfort prediction. Some studies directly extracted images with a thermal camera and others collected regular photos/videos with post-processing afterward [28,86]. While other studies were using face mapping and temperature extraction from the defined regions via temperature sensors [61,69].

In general, the main assumption behind the data accumulation from images of the skin surface, heart bit, systolic + diastolic pressure and other biological data is that the our body is expected to to balance conditions via set of biological features defined by nature in order to be in comfort. So, it should be possible to track biological changes within the body that occur before the actual event of thermal discomfort. Due to the fast development of novel technology in the field of biometrical data collection, it is feasible to conclude that a more structured approach should be applied for the bio-markers selection. Moreover, a constructive background must be provided with respect to why certain biometrical variables were chosen. Only few studies discussed points of validity for chosen biometrical parameters [30,69].

A thorough literature review of the field found that no one has used facial muscle movement data as a feature for evaluation of TC. The given experimental design is a novel approach to resolve the long-standing concern of indoor thermal discomfort events, and enables further investigation into the application of such novel technology for real-life solutions [26].

Studies that applied iMotions before were mostly focused on the monitoring and evaluation of user’s feedback, interaction, level of attention or consumer preference in marketing purposes [87,88,89,90,91]. While given frameworks are convenient since the Affectiva AFFDEX post processing algorithm provides results which can be directly used for the next stages of the study evaluation, we have a bigger ambition for iMotions use. The presented experimental design provides abnormal framework for iMotions software and hardware modules, which involves post processing of the extracted data before it’s directly used with machine learning algorithms or other type of processing.

6. Conclusions

The experimental design, which was developed in the framework of PhD research, suggests how novel technologies may be implemented in the field of thermal discomfort detection. Tracking of the involuntary facial muscle movements within a built environment is a big step forward since such biometrical data has a direct link to the subcortical level of the brain, which is also responsible for thermal regulation of the body. By synthesizing bio-markers, it is possible to create a more logically structured database. A generated database is used for the implementation of the artificial intelligence and machine learning algorithms in the final stage of non-invasive thermal discomfort detection.

The article presents a developed framework on how facial muscle movement data can be extracted via iMotions software, which is a synergy between FACS and AFC with its built-in machine vision algorithms and post processing tool Affectiva AFFDEX. It also provides an approach for synchronised real time biometrical data collection via Shimmer equipment that directly interacts with iMotions software during the data collection process.

The software enables synchronization of the data from Shimmer device and actual facial expressions. While iMotion synchronization feature is useful and provides a nicely organized file for each session, it doesn’t allow the direct inclusion of temperature data from other sensors. Due to the given circumstances, the time step for a number of variables needs to be readjusted at a post-processing step in order to create the proper feature vector afterword.

With development of the technology, it is foreseen to use Electromyography and many other features in future data collection campaigns. The given hardware will allow tracking of brain wave activity and to determine which parts of the brain are active during indoor temperature changes and at the moment of discomfort.

The experiment has shown there is a general curiosity with the development of the thermal comfort field and a desire to actively participate in data collection campaigns. While the issue of concern was that people would not feel comfortable with facial muscle movement data collection, since they would need to be continuously monitored via camera. It turned out, that in the world of smartphones that use facial scanning to unlock, people have become accustomed to this technology since we provide our facial biometrical data many times a day. The given insight suggests that a majority of people will not feel discomfort if a new non-invasive thermal discomfort detection model were deployed in the workplace environment.