Deception in the eyes of deceiver: A computer vision and machine learning based automated deception detection

https://doi.org/10.1016/j.eswa.2020.114341Get rights and content
Under a Creative Commons license
open access

Highlights

  • Deception is detected through nonverbal behaviour from video interviews.

  • Real-time deception detection through facial micro-movements using machine learning.

  • Facial micro-movements are ranked on distinctiveness towards deceptive behaviour.

  • Results show eye related micro-movements more significant for deception detection.

Abstract

There is growing interest in the use of automated psychological profiling systems, specifically applying machine learning to the field of deception detection. Several psychological studies and machine-based models have been reporting the use of eye interaction, gaze and facial movements as important clues to deception detection. However, the identification of very specific and distinctive features is still required. For the first time, we investigate the fine-grained level eyes and facial micro-movements to identify the distinctive features that provide significant clues for the automated deception detection. A real-time deception detection approach was developed utilizing advanced computer vision and machine learning approaches to model the non-verbal deceptive behavior. Artificial neural networks, random forests and support vector machines were selected as base models for the data on the total of 262,000 discrete measurements with 1,26,291 and 128,735 of deceptive and truthful instances, respectively. The data set used in this study is part of an ongoing programme to collect a larger dataset on the effects of gender and ethnicity on deception detection. Some observations are made based on this data which should not be interpreted as scientific conclusions, but pointers for future work. Analysis of the above models revealed that eye movements carry relatively important clues to distinguish truthful and deceptive behaviours. The research outcomes align with the findings from forensic psychologists who also reported the eye movements as distinctive for the truthful and deceptive behavior. The research outcomes and proposed approach are beneficial for human experts and has many applications within interdisciplinary domains.

Keywords

Deception detection
Credibility assessment
Facial micro-gestures
Nonverbal behavior analysis
Eye movements
Psychological profiling

Cited by (0)

Dr. Wasiq Khan is a Senior Lecturer in Artificial Intelligence (AI) & Data Sciences within the Department of Computer Science at Liverpool John Moores University, UK. Wasiq received his B.Sc. in Mathematics, Physics and M.Sc. in Computer Science from Pakistan. He further received an MSc in AI followed by a Ph.D. in AI & Speech Processing from Bradford University in 2015, UK. Wasiq is research active within the domain of AI, Deep/Machine learning, Speech processing, and Video/Image analysis. He has been working as a lead researcher/Co-Investigator on various large-scale research projects in collaborations with academia & industry. He has been publishing research outcomes in high impact Journals and conferences. He is an active reviewer of various top ranked Journals (including IEEE transactions and IEEE Access), UKRI/EPSRC research grants and chairing conference sessions. Along with the teaching roles and Ph.D supervisions, Wasiq has established academic citizenship within the domain of AI and Data Science while he is also Fellow of Higher Education Academy, UK and Member of IEEE, Computational Intelligence Society.

Dr. Keeley Crockett is a Reader in Computational Intelligence in the School of Computing, at Manchester Metropolitan University in the UK. She gained a BSc Degree (Hons) in Computation from UMIST in 1993, and a PhD in the field of machine learning from the Manchester Metropolitan University in 1998. She is A Senior Fellow of the Higher Education Academy. She leads the Computational Intelligence Lab that has established a strong international presence in its research into Adaptive Psychological Profiling including an international patent on “Silent Talker”. She is currently a member of the IEEE Task Force on Ethical and Social Implications of Computational Intelligence. She has 22 PHD completions and externally examined 8 PhDs. Her main research interests include fuzzy decision trees, semantic text based clustering, conversational agents, fuzzy natural language processing, semantic similarity measures, and AI for psychological profiling. Currently the Principal Investigator (MMU) on the H2020 funded project iBorderCtrl: Intelligent Smart Border Control, CI on H2020 Grant “Populism and Civic Engagement: a fine-grained, dynamic, forward-looking response to the negative impacts of populist movements (PaCE)”, and CI on UK KTP with Service Power. She is a member of the IEEE WIE Leadership committee, Chair of the IEEE CIS Webinars, Vice-Chair of the IEEE Women into Computational Intelligence. She is Student Activities co-chair for IEEE WCCI 2020. She has authored over 120 peer reviewed publications.

Dr. O’Shea holds concurrent affiliations as a Senior Lecturer in Computer Science at Manchester Metropolitan University. He offers 18 years of experience to his role as co-founder and consultant for the Silent Talker team. Dr. O’Shea helped develop, test, and patent the proprietary Silent Talker software. He has earned degrees in Chemistry and Artificial Intelligence. He is Editorial Board Member for the International Journal of Intelligent Defense Support Systems. He organized and chaired international conferences on Agent and Multi-Agent Systems. With more than 60 publications in international scientific Journals, book chapters and peer-reviewed conferences, Dr. O’Shea is a leading expert in adaptive psychological profiling, dialogue systems and computational intelligence. He is also an expert in applying these systems both in English and with resource-poor languages such as Arabic, Urdu, and Thai. Dr. O’Shea has led or contributed to projects funded by Horizon 2020. He is an enthusiastic science communicator including developing young scientists through Research Placements funded by the Nuffield Foundation. He is currently a co-investigator on the H2020 funded project iBorderCtrl – Intelligent Smart Border Control. He is a member of the IEEE and a lifetime gold member of the Knowledge Engineering Society.

Prof. Abir H. is a professor of Machine Learning and a member of the Applied Computing Research Group at the Faculty of Engineering and Technology. She completed her PhD study at The University of Manchester (UMIST), UK in 2000 with a thesis title Polynomial Neural Networks for Image and Signal Processing. She has published numerous referred research papers in conferences and Journal in the research areas of Neural Networks, Signal Prediction, Telecommunication Fraud Detection and Image Compression. She has worked with higher order and recurrent neural networks and their applications to financial, physical, e-health and image compression techniques. She has developed with her research students a number of recurrent neural network architectures. Her research has been published in a number of high esteemed and high impact journals such as the Expert Systems with Applications, PloS ONE, Electronic Letters, Neuro-computing, and Neural Networks and Applications. She is a PhD supervisor and an external examiner for research degrees including PhD and MPhil. She is one of the initiators and chairs of the Development in e-Systems Engineering (DeSE) series, most notably illustrated by the IEEE technically sponsored DeSE International Conference Series.

Dr. Bilal Khan. is senior researcher at the University of California Los Angeles. He also co-founded and currently serving as the chief analytics officer at Noria Water Technologies, a Los Angeles based technology company. Bilal has master’s degrees in computer science (PK) and Pervasive Computing (Birmingham City University, UK), Ph.D. in cooperative vehicular networks using game theory and artificial intelligence (University of Bradford, UK, 2012) and the fellowship of the UK higher education commission. His current areas of expertise are the use of data analytics in the areas of membrane-based water treatment, heat exchangers, nanotechnology and environmental risk assessment. Dr. Bilal was recently nominated as 2019 young professional by the US Water and Wastes Digest for his IP protected work for digital advancements in water treatment and power industries. Bilal has active memberships of various prestigious scientific organizations (American Water Works Association, American Chemical Society, American Institute of Chemical Engineers, US-EU Nanotechnology Roadmap, US Nano Working Group). Dr. Bilal is also an active reviewer of reputed journals (including Nature Scientific Data, Environmental Science and Technology, Nanoscale) and has chaired technical sessions in the fields of Nanotechnology and Water treatment.