Skip to main content
SpringerLink
Log in

A knowledge discovery and visualisation method for unearthing emotional states from physiological data

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

In this paper we propose a knowledge discovery and visualisation method for unearthing emotional states from physiological data typically available from wearable devices. In addition we investigate the viability of using a limited set of wearable sensors to extract decision tree rules which are representative of physiological changes taking place during emotional changes. Our method utilised a fusion of pre-processing and classification techniques using decision trees to discover logic rules relating to the valence and arousal emotional dimensions. This approach normalised the signal data in a manner that enabled accurate classification and generated logic rules for knowledge discovery. Furthermore, the use of three target classes for the emotional dimensions was effective at denoising the data and further enhancing classification and useful rule extraction. There are three key contributions in this work, firstly an exploration and validation of our knowledge discovery methodology, secondly successful extraction of high accuracy rules derived from physiological data and thirdly knowledge discovery and visualisation of relationships within-participant physiological data that can be inferred relating to emotions. Additionally, this work may be utilised in areas such as the medical sciences where interpretable rules are required for knowledge discovery.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Adnan MN, Islam MZ (2016) Knowledge discovery from a data set on dementia through decision forest. In: The 14th Australasian Data Mining Conference: AusDM 2016. CRPIT, pp 1–8

  2. Adnan MN, Islam MZ (2017) ForEx++: a new framework for knowledge discovery from decision forests. Aust J Inform Syst 21:1–20

    Google Scholar 

  3. Adolphs R (2002) Neural systems for recognizing emotion. Curr Opin Neurobiol 12:169–177. https://doi.org/10.1016/S0959-4388(02)00301-X

    Article  Google Scholar 

  4. Aydin SG, Kaya T, Guler H (2016) Wavelet-based study of valence–arousal model of emotions on EEG signals with LabVIEW. Brain Inform 3:109–117

    Article  Google Scholar 

  5. Banaee H, Ahmed MU, Loutfi A (2013) Data mining for wearable sensors in health monitoring systems: a review of recent trends and challenges. Sensors 13:17472–17500

    Article  Google Scholar 

  6. Beda A, Jandre F, Phillips D, Giannella-Neto A, Simpson DM (2007) Heart-rate and blood-pressure variability during psychophysiological tasks involving speech: influence of respiration. Psychophysiology 44:767–778. https://doi.org/10.1111/j.1469-8986.2007.00542.x

    Article  Google Scholar 

  7. Bruno P, Melnyk V, Völckner F (2017) Temperature and emotions: Effects of physical temperature on responses to emotional advertising. Int J Res Mark 34:302–320. https://doi.org/10.1016/j.ijresmar.2016.08.005

    Article  Google Scholar 

  8. Calvo RA, D'Mello S (2010) Affect detection: an interdisciplinary review of models, methods, and their applications. IEEE Trans Affect Comput 1:18–37

    Article  Google Scholar 

  9. Wood M (2014) Taking tracking to the extreme. New York Times. Retrieved from https://nyti.ms/335sY9i

  10. Conor A (2018) Smartwatches will remain the most popular wearables into 2022. Retrieved from https://www.wareable.com/smartwatches/smartwatches-most-popular-wearable-2022-idc-6840

  11. Costadopoulos N, Islam MZ, Tien D (2019a) Data mining and knowledge discovery from physiological sensors. In: Paper presented at the pervasive technologies related to assistive environments (PETRA), Rhodes, Greece, June 5–7, 2019

  12. Costadopoulos N, Islam MZ, Tien D (2019b) Discovering emotional logic rules from physiological data of individuals. In: Paper presented at the international conference on machine learning and cybernetics (ICMLC), Kobe, Japan, July 7–10, 2019

  13. Costadopoulos N, Islam MZ, Tien D (2019c) Using Z-score to Extract Human Readable Logic Rules from Physiological Data. In: Paper presented at the (Accepted/In press) 11th IEEE international conference on knowledge and systems engineering (KSE) Da Nang, Vietnam, October 24–26, 2019

  14. Critchley HD (2002) Review: electrodermal responses: what happens in the brain. Neuroscientist 8:132–142. https://doi.org/10.1177/107385840200800209

    Article  Google Scholar 

  15. Human respiratory system (2020). In: Encyclopædia Britannica. Retrieved from https://academic-eb-com.ezproxy.csu.edu.au/levels/collegiate/article/human-respiratory-system/117582#

  16. Drazin S, Montag M (2012) Decision tree analysis using Weka. Retrieved from http://wwww.samdrazin.com/classes/een548/project2report.pdf

  17. Fallen E (2000) Hidden rhythms in the heart rate record: a primer on neurocardiology. Clin Invest Med 23:339–394

    Google Scholar 

  18. Fang Y, Zhou D, Li K, Ju Z, Liu H (2019) Attribute-driven granular model for EMG-based pinch and fingertip force grand recognition. IEEE Trans Cybern

  19. Fitbit versa the Apple Watch (2020) https://search-proquest-com.ezproxy.csu.edu.au/docview/2331562403?accountid=10344. Accessed 01 Jan 2020

  20. Fletcher S, Islam MZ (2017) Measuring rule retention in anonymized data-when one measure is not enough. Trans Data Priv 10:175–201

    Google Scholar 

  21. Geng L, Hamilton HJ (2006) Interestingness measures for data mining: a survey. ACM Comput Surv (CSUR) 38:9

    Article  Google Scholar 

  22. Google (2019) Google Dataset Search. https://toolbox.google.com/datasetsearch/

  23. Hochman D (2015) Does the Spire stress tracker actually work? Forbes. Retrieved from https://www.forbes.com/sites/davidhochman/2015/06/21/does-the-spire-stress-tracker-actually-work/2/#60bf44215b48

  24. Holmes G, Donkin A, Witten IH (1994) Weka: a machine learning workbench. Paper presented at the proceedings of ANZIIS'94-Australian New Zealand intelligent information systems conference

  25. Hong L, Cai J (2010) The application guide of mixed programming between MATLAB and other programming languages. 2010

  26. Hui T, Sherratt R (2018) Coverage of emotion recognition for common wearable biosensors. Biosensors 8:30

    Article  Google Scholar 

  27. IDC (2020) Earwear and wristbands drive first quarter growth in the worldwide wearables market says IDC. IDC, Melbourne

    Google Scholar 

  28. Jain A, Nandakumar K, Ross A (2005) Score normalization in multimodal biometric systems. Pattern Recogn 38:2270–2285

    Article  Google Scholar 

  29. Jenke R, Peer A, Buss M (2014) Feature extraction and selection for emotion recognition from EEG. IEEE Trans Affect Comput 5:327–339

    Article  Google Scholar 

  30. Jirayucharoensak S, Pan-Ngum S, Israsena P (2014) EEG-based emotion recognition using deep learning network with principal component based covariate shift adaptation. Sci World J 2014

  31. Khatchadourian R (2015) We know how you feel. The New Yorker. Retrieved from https://www.newyorker.com/magazine/2015/01/19/know-feel

  32. Kimberly H (2016) Thermoregulation. Healthline Media. https://www.healthline.com/health/thermoregulation. 2018

  33. Koelstra S, Mühl C, Soleymani M, Lee J-S, Yazdani A, Ebrahimi T, Pun T, Nijholt A, Patras I (2012) DEAP: a database for emotion analysis using physiological signals. IEEE Trans Affect Comput 3(1):18–31. https://doi.org/10.1109/T-AFFC.2011.15

    Article  Google Scholar 

  34. Lan Z, Sourina O, Wang L, Liu Y (2016) Real-time EEG-based emotion monitoring using stable features. Int J Comput Graph 32(3):347–358. https://doi.org/10.1007/s00371-015-1183-y

    Article  Google Scholar 

  35. Marta Z (2017) The warmth of friendship, the chill of Betrayal. Sci Am Mind 28:66. https://doi.org/10.1038/scientificamericanmind0317-66

    Article  Google Scholar 

  36. Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn JF (2013) Disfa: a spontaneous facial action intensity database. IEEE Trans Affect Comput 4:151–160

    Article  Google Scholar 

  37. McGarry K (2005) A survey of interestingness measures for knowledge discovery. Knowl Eng Rev 20:39–61

    Article  Google Scholar 

  38. Metz R (2014) Using your ear to track your heart. MIT Technology Review. Retrieved from https://www.technologyreview.com/2014/08/01/171915/using-your-ear-to-track-your-heart/

  39. Pfister T, Robinson P (2011) Real-time recognition of affective states from nonverbal features of speech and its application for public speaking skill analysis. IEEE Trans Affect Comput 2(2):66–78. https://doi.org/10.1109/T-AFFC.2011.8

    Article  Google Scholar 

  40. PhysioNet (2019) The research resource for complex physiologic signals. MIT Laboratory for Computational Physiology. https://physionet.org

  41. Picard V, Healey (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23:1175–1191. https://doi.org/10.1109/34.954607

    Article  Google Scholar 

  42. Picard R (2003) Affective computing: challenges. Int J Hum Comput Stud 59:55–64. https://doi.org/10.1016/S1071-5819(03)00052-1

    Article  Google Scholar 

  43. Picard RW (1995) Affective computing. In: Paper presented at the MIT Media Laboratory Perceptual Computing Section Technical Report

  44. Quinlan JR (1993) C4.5: programs for machine learning. Morgan Kaufmann Publishers Inc, San Francisco

    Google Scholar 

  45. Rhodes BJ, Minar N, Weaver J (1999) Wearable computing meets ubiquitous computing reaping the best of both worlds. In: The Third International Symposium on Wearable Computers (ISWC '99), San Francisco, CA, October 18–19 1999 1999. San Francisco, CA, pp pp. 141–149

  46. Ruiz-Rodríguez J (2013) Innovative continuous non-invasive cuffless blood pressure monitoring based on photoplethysmography technology. Intensive Care Med 1618–1625

  47. Salzberg SL (1994) C4.5: Programs for machine learning by j. ross quinlan. Morgan Kaufmann Publishers, Inc., 1993 Machine Learning 16:235–240

  48. Sarkar S, Bhoi AK, Savita G (2012) Fingertip pulse wave (PPG signal) analysis and heart rate detection. Int J Emerg Technol Adv Eng 2:404–408

    Google Scholar 

  49. Sarker R, Abbass H, Newton C (2002) Introducing data mining and knowledge discovery. Heuristic and optimization for knowledge discovery. IGI Global, Pennsylvania, pp 1–12

    Google Scholar 

  50. Smets E et al (2018) Large-scale wearable data reveal digital phenotypes for daily-life stress detection. NPJ Digital Med 1:67

    Article  Google Scholar 

  51. Soleymani M, Koelstra S, Patras I, Pun T (2011) Continuous emotion detection in response to music videos. In: Paper presented at the Face and Gesture

  52. Soleymani M, Lichtenauer J, Pun T, Pantic M (2012) A multimodal database for affect recognition and implicit tagging. IEEE Trans Affect Comput 3:42–55

    Article  Google Scholar 

  53. Wang D, Shang Y (2013) Modeling physiological data with deep belief networks. Int J Inform Edu Technol (IJIET) 3:505

    Google Scholar 

Download references

Acknowledgements

We wish to thank Professor Terry Bossomaier from the (School of Computing and Mathematics, Charles Sturt University) for his advice on the pre-processing steps and strategies for validating our methodology.

Author information

Authors and Affiliations

  1. School of Computing and Mathematics, Charles Sturt University, Bathurst, Australia

    Nectarios Costadopoulos, Md Zahidul Islam & David Tien

Authors
  1. Nectarios Costadopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Md Zahidul Islam
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David Tien
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nectarios Costadopoulos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Costadopoulos, N., Islam, M.Z. & Tien, D. A knowledge discovery and visualisation method for unearthing emotional states from physiological data. Int. J. Mach. Learn. & Cyber. 12, 843–858 (2021). https://doi.org/10.1007/s13042-020-01205-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-020-01205-4

Keywords

Search

Navigation