Skip to main content
SpringerLink
Log in

Toward emotional interactive videogames for children with autism spectrum disorder

  • Long Paper
  • Published:
Universal Access in the Information Society Aims and scope Submit manuscript

Abstract

Technology and videogames have been proven as motivating tools for working attention and complex communication skills, especially in children with autism spectrum disorder (ASD). In this work, we present two experiences that used interactive games for promoting communication and attention. The first game considers emotions in order to measure children’s attention, concentration and satisfaction, while the second uses tangible tabletops for fostering cognitive planning. The analysis of the results obtained allows to propose a new study integrating both, in which the tangible interactive game is complemented with the emotional trainer in a way that allows identifying and classifying children’s emotion with ASD when they collaborate to solve cognitively significant and contextualized challenges. The first application proposed is an emotional trainer application in which the child can work out the seven basic emotions (happiness, sadness, fear, disgust, anger, surprise and neutral). Further, a serious videogame is proposed: a 3D maze where the emotions can be captured. The second case study was carried out in a Special Education Center, where a set of activities for working cognitive planning was proposed. In this case, a tangible interactive tabletop was used to analyze, in students with ASD, how the communication processes with these interfaces affect to the attention, memory, successive and simultaneous processing that compose cognitive planning from the PASS model. The results of the first study, suggest that the autistic children did not act with previous planning, but they used their perception to adjust their actions a posteriori (that explains the higher number of collisions). On the second case study, the successive processing was not explored. The inclusion of the mazes of case study 1 to a semantic rich scenario could allow us to measure the prior planning and the emotions involved in the maze game. The new physiological sensors will also help to validate the emotions felt by the children. The first study has as objective the capability to imitate emotions and resolve a maze without semantic context. The second study organized all the actions from a semantic context close to users. The attention results presented by the second study are coherent with the first study and complement it showing that attention can be receptive or selective. In the first study case, the receptive attention was the focus of analysis. In the second case, both contributed to explain and understand how it can be developed from a videogame.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Rivière, A.: El desarrollo y la educación del niño autista. In: C. Coll, J. Palacios, A. Marchesi (orgs.) Desarrollo Psicológico y Educación: necesidades educativas especiales y aprendizaje escolar, III. Alianza Editorial (1993)

  2. Friedrich, E.V.C., Suttie, N., Sivanathan, A., Lim, T., Louchart, S., Pineda, J.A.: Brain–computer interface game applications for combined neurofeedback and biofeedback treatment for children on the autism spectrum. Front. Neuroeng. 1, 15–21 (2014). https://doi.org/10.3389/fneng.2014.00021

    Article  Google Scholar 

  3. Tomasello, M.: The Cultural Origins of Human Cognition. Harvard University Press, Harvard (1999)

    Google Scholar 

  4. Baron-Cohen, S.: From attention-goal psychology to belief-desire psychology: The development of a theory of mind and this dysfunction. In: Baron-Cohen, S., Tager-Flusberg, H., Cohen, D.J. (eds.) Understanding Other Minds: Perspectives from Autism. Oxford University Press, New York (1993)

    Google Scholar 

  5. Piper, A., O’Brien, E., Morris, M., Winograd T.: SIDES: a cooperative tabletop computer game for social skills development. In: 20th Conference on Computer Supported Cooperative Work, pp 1–10 (2006)

  6. Battocchi, A., Ben-Sasson, A., Esposito, G., Gal, E., Pianesi, F., Tomasini, D., Venuti, P., Weiss, P., Zancanaro, M.: Collaborative puzzle game: a tabletop interface for fostering collaborative skills in children with autism spectrum disorders. J. Assist. Technol. 4(1), 4–14 (2010)

    Article  Google Scholar 

  7. Passerino, L.M., Santarosa, L.M.C.: Autism and digital environments: processes of interaction and mediation. Comput. Educ. 51(1), 385–402 (2008)

    Article  Google Scholar 

  8. Chen, W.: Multitouch tabletop technology for people with autism spectrum disorder: a review of the literature. Proc. Comput. Sci. 14, 198–207 (2012)

    Article  Google Scholar 

  9. Vygotsky, L.: Mind in Society: The development of Higher Psychological Processes. Ed. M. Cole. Harvard University Press, Cambridge (1978)

    Google Scholar 

  10. Abt, C.: Serious Games. The Viking Press, New York (1970)

    Google Scholar 

  11. Bul, K.C.M., Doove, L.L., Franken, I.H.A., Oord, SVd, Kato, P.M., Maras, A.: A serious game for children with Attention deficit hyperactivity disorder: Who benefits the most? PLoS One 13(3), e0193681 (2018). https://doi.org/10.1371/journal.pone.0193681

    Article  Google Scholar 

  12. Zakari H.M., Ma M., Simmons D.: A review of serious games for children with autism spectrum disorders (ASD). In: Conference: 5th International Conference, SGDA, Berlin, Germany (2014). https://doi.org/10.1007/978-3-319-11623-5_9

  13. Anwar, Md., Rahman, M., Ferdous, S.M., Anik, S.A., Ahmed, S.I.: A computer game based approach for increasing fluency in the speech of the autistic children. In: 11th IEEE International Conference on Record, Advanced Learning Technologies, pp. 17–18. (2011)

  14. Sáenz de Urturi, Z., Méndez Zorrilla, A., García Zapirain, B.: Serious game based on first aid education for individuals with autism spectrum disorder (ASD) using android mobile devices. In: Conference Record, 16th International Conference on Computer Games, pp. 223–227 (2011)

  15. Fridenson-Hayo, S., Berggren, S., Lassalle, A., Tal, S., Pigat, D., Meir-Goren, N., O’Reilly, H., BenZur, S., Bölte, S., Baron-Cohen, S., Golan, O.: ‘Emotiplay’: a serious game for learning about emotions in children with autism: results of a cross-cultural evaluation. Eur. Child Adolesc. Psychiatry (2017). https://doi.org/10.1007/s00787-017-0968-0

    Article  Google Scholar 

  16. Rahman, Md. M., Ferdous, S.M., Ahmed, S.I.: Increasing intelligibility in the speech of the autistic children by an interactive computer game. In: Conference Record IEEE International Symposium on Multimedia, pp. 383–387 (2010)

  17. Arellano, D., Rauh, R., Krautheim, B., Spicker, M., Schaller, U.M., Helzle, V., Deussen, O.: Interactive testbed for research in autism—the SARA project. Univ Access Inf Soc, pp 1–16 (2017). https://doi.org/10.1007/s10209-016-0521-9

  18. Cerezo, E., Marco, J., Baldassarri, S.: Hybrid games: designing tangible interfaces for very young children and children with special needs. In: More Playful User Interfaces, pp. 17–48. Springer, Singapore (2015)

  19. Marshall, P., Price, S., and Rogers Y.: Conceptualising tangibles to support learning. Conference on interaction Design and Children. IDC ‘03 (2003)

  20. Schneider, B., Jermann, P., Zufferey, G., Dillenbourg, P.: Benefits of a tangible interface for collaborative learning and interaction. IEEE Trans. Learn. Technol. 4(3), 222–232 (2011)

    Article  Google Scholar 

  21. Hamidi, F.: Digital tangible games for speech intervention. Technical Report, York University (Technical Report CSE-2012-02) (2012)

  22. Tartaro A., Cassell, J.: Playing with virtual peers: bootstrapping contingent discourse in children with autism. In: 8th International conference for the learning sciences. Vol. 2 (2008)

  23. Gal, E., Lamash, L., Bauminger-Zviely, N., Zancanaro, M., Weiss, P.L.: Using multitouch collaboration technology to enhance social interaction of children with high-functioning autism. Phys. Occup. Ther. Pediatr. 36(1), 46–58 (2016)

    Article  Google Scholar 

  24. Silva, G.F.M., Raposo, A., Suplino, M.: Exploring collaboration patterns in a multitouch game to encourage social interaction and collaboration among users with autism spectrum disorder. Comput. Support. Coop. Work 24(2–3), 149–175 (2015)

    Article  Google Scholar 

  25. Zancanaro, M., Giusti, L., Bauminger-Zviely, N., Eden, S., Gal, E., Weiss P.L.: NoProblem! a collaborative interface for teaching conversation skills to children with high functioning autism spectrum disorder. In: Nijholt, A. (Ed.) Playful User Interfaces: Interfaces that Invite Social and Physical Interaction, Singapore, pp. 209–224 (2014)

  26. Wing, L., Gould, J.: J Autism Dev Disord. Kluwer Academic Publishers-Plenum Publishers, Print ISSN: 0162-3257, (1979) 9:11. https://doi.org/10.1007/BF01531288

  27. Ramis, S., Perales, F.J., Campins, M., Riquelme, I.: Un videojuego serio para el estudio de expresiones faciales en personas con Autismo. Cognitive Area Networks, 9º Simposio CEA de Bioingeniería, Interfaces Cerebro-Máquina. Neurotecnologías para la Asistencia y la Rehabilitación. 6 y 7 de Julio 2017. Institut Guttmann, Barcelona. ISSN 2341-4243. (España): 2017. (in Spanish)

  28. Achenbach, T.M., Edelbrock, C.S.: Manual for the child behavior checklist and revised child behavior profile. University of Vermont, Department of Psychiatry, Burlington (1983)

    Google Scholar 

  29. Izard, C.E., Haskins, F.W., Schultz, D., Trentacosta, C.J., King, K.A.: Emotion Matching Task. University of Delaware, Newark (2003)

    Google Scholar 

  30. Shield, A., Cicchetti, D.: Emotion regulation among school-age children: the development and validation of a new criterion Q-sort scale. Mt. Hope Family Center, University of Rochester, New York (1997)

    Google Scholar 

  31. Achenbach, T.M., Edelbrock, C.S.: Child Behavior Checklist (CBC), Burlington (Vt). Accessed May 2019

  32. Das, J.P., Naglieri, J.A., Kirby, J.R.: Assessment of Cognitive Processes: the P.A.S.S. Theory of Intelligence. Allyn and Bacon, Toronto (1994)

    Google Scholar 

  33. Marco, J., Baldassarri, S., Cerezo, E.: NIK Vision: Developing a Tangible Application for and with Children. J. Univ. Comput. Sci. 19(15), 2266–2291 (2013)

    Google Scholar 

  34. Kaltenbrunner, M., Bencina, R.: ReacTIVision: a computer-vision framework for table-based tangible interaction. In: Proceedings of the first international conference on Tangible and embedded interaction (TEI’07), Baton Rouge, Louisiana, pp. 69–74. (2007)

  35. Communication Matrix.: https://communicationmatrix.org.

  36. Marco, J., Baldassarri, S., Cerezo, E., Xu, Y., Read, J.C.: Let the experts talk: an experience of tangible game design with children. ACM Inter. 17(1), 58–61 (2010)

    Article  Google Scholar 

  37. Arroyo, I., Cooper, D.G., Burleson, W., Woolf, B.P., Muldner, K., Christopherson, R.: Emotion sensors go to school. AIED 200, 17–24 (2009)

    Google Scholar 

  38. Cabibihan, J.J., Javed, H., Aldosari, M., Frazier, T.W., Elbashir, H.: Sensing technologies for autism spectrum disorder screening and intervention. Sensors 17(1), 46 (2016)

    Article  Google Scholar 

  39. Chen, J., Wang, G., Zhang, K., Wang, G., Liu, L.: A pilot study on evaluating children with autism spectrum disorder using computer games. Comput. Hum. Behav. 90, 204–214 (2019)

    Article  Google Scholar 

  40. Ekman, P.: Facial expressions of emotion: an old controversy and new findings. In Bruce, V., Cowey, A., Ellis, A. W., Perrett, D. I. (Eds.) (1992)

  41. Ramis, S., Perales, F.J., Campins, M., Riquelme, I.: Un videojuego serio para el estudio de expresiones faciales en personas con Autismo. CEA Bioingenieria’17 Conference 4(1), 85–92 (2017)

    Google Scholar 

  42. Richards, J.E., Casey, B.J.: Heart rate variability during attention phases in young infants. Psychophysiology 28(1), 43–53 (1991)

    Article  Google Scholar 

  43. Jerritta, S., Murugappan, M., Nagarajan, R., Wan, K.: Physiological signals based human emotion recognition: a review. In: Signal Processing and its Applications (CSPA), pp. 410–415 (2011)

  44. Hamada, M., Zaidan, B.B., Zaidan, A.A.: A systematic review for human EEG brain signals based emotion classification, feature extraction, brain condition, group comparison. J. Med. Syst. 42(9), 162 (2018)

    Article  Google Scholar 

  45. Van Dooren, M., Janssen, J.H.: Emotional sweating across the body: comparing 16 different skin conductance measurement locations. Physiol. Behav. 106(2), 298–304 (2012)

    Article  Google Scholar 

  46. Mazefsky, C.A., Herrington, J., Siegel, M., Scarpa, A., Maddox, B.B., Scahill, L., White, S.W.: The role of emotion regulation in autism spectrum disorder. J. Am. Acad. Child Adolesc. Psychiatry 52(7), 679–688 (2013)

    Article  Google Scholar 

  47. Mercado, J., Espinosa-Curiel, I., Escobedo, L., Tentori, M.: Developing and evaluating a BCI video game for neurofeedback training: the case of autism. Multimed. Tools Appl. 1, 1–38 (2018)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the management and pedagogical team of the “Jean Piaget Special Education School” for supporting and helping us in this research. This work was partially supported by the Government of Spain through contracts TIN2015-67149-C3-1R and TIN2015-67149-C3-2R, the Aragonese Government and the UE through the FEDER 2014-2020 “Construyendo Europa desde Aragón” action (Group T25_17D) and the support of Capes/Cnpq from Brazil. Also, the third author is benefited by fellowship FPI/1828/2015 of the Conselleria d’Educació, Cultura i Universitats of Govern de les Illes Balears under an operational program co-financed by the European Social Fund.

Author information

Authors and Affiliations

  1. Advanced Computer Graphics Group (GIGA), Computer Science Department, Engineering Research Institute of Aragon (I3A), Universidad de Zaragoza, Ed. Ada Byron, C/María de Luna n. 1, 50015, Saragossa, Spain

    Sandra Baldassarri

  2. Centro de Estudos Interdisciplinares em Novas Tecnologias da Educação - Campus Centro. Av. Paulo Gama, 110 Anexo III - 3º andar, Universidade Federal do Rio Grande do Sul (UFRGS), Porto Alegre, RS, Brazil

    Liliana Passerino

  3. Computer Graphics, Vision and Artificial Intelligence Group A. Turmeda, DMI-UIB, EPS. Crta. Valldemossa, KM 7.5 S/N, 07122, Palma de Mallorca, Spain

    Silvia Ramis & Francisco J. Perales

  4. Facultat d’Infermeria i Fisioteràpia, Institut Universitari d’Investigació en Ciències de la Salut (IUNICS), UIB. Crta. Valldemossa, KM 7.5 S/N, 07122, Palma de Mallorca, Spain

    Inma Riquelme

Authors
  1. Sandra Baldassarri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liliana Passerino
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Silvia Ramis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Inma Riquelme
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Francisco J. Perales
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Francisco J. Perales.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Baldassarri, S., Passerino, L., Ramis, S. et al. Toward emotional interactive videogames for children with autism spectrum disorder. Univ Access Inf Soc 20, 239–254 (2021). https://doi.org/10.1007/s10209-020-00725-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10209-020-00725-8

Keywords

Search

Navigation