Skip to main content

Advertisement

SpringerLink
Log in

Implementation of escape room system based on augmented reality involving deep convolutional neural network

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Escape room is a live-action adventure game, where the players search clues, solve puzzles and achieve the assigned tasks. This paper proposed a novel escape room system combining augmented reality and deep learning technology. The system adopts a client–server architecture and can be divided into the server module, the smart glasses module and the player–hardware interaction module. The player–hardware interaction module consists of subsystems each of which includes a Raspberry Pi 3. HoloLens is used as the smart glasses in the experiment of the paper. The server communicates with all the Raspberry Pis and HoloLens through TCP/IP protocol and manages all the devices to achieve the game flow by following the process timeline. The smart glasses module provides two display modes, i.e., the AR 3D models display and the 2D text clues display. In the first mode, the SDK Vuforia is used for detection and tracking of markers. In the second mode, the scene images captured by HoloLens camera are sent to the pre-trained image classifier based on deep convolutional neural network. Considering both the image category and the game status value, the server decides the text clue image to be displayed on HoloLens. The accuracy of the image classification model reaches 94.9%, which can be correctly classified for a certain rotation angle and partial occlusion. The integration of AR, deep learning, electronics and escape room games opens up exciting new directions for the development of escape room. Finally, a built mini-escape room is analyzed to prove that the proposed system can support more complicated narratives showing the potential of achieving immersion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig.3
Fig.4
Fig.5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig.13
Fig.14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  • Billinghurst M, Clark A, Lee G (2015) A survey of augmented reality. Found Trends Hum-Comput Interact 8(2–3):73–272

    Article  Google Scholar 

  • Cruz E, Orts-Escolano S, Gomez-Donoso F et al (2019) An augmented reality application for improving shopping experience in large retail stores. Virtual Reality 23:281–291

    Article  Google Scholar 

  • Goh KS, Chang E, Cheng KT (2001) SVM binary classifier ensembles for image classification. In: Proceedings of the tenth international conference on information and knowledge management. ACM, pp 395–402

  • Griffiths MD, Nuyens F (2017) An overview of structural characteristics in problematic video game playing. Curr Addict Rep 4:272–283

    Article  Google Scholar 

  • He K, Zhang X, Ren S et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  • Huang F, Zhou Y, Yu Y et al (2011) Piano ar: a markerless augmented reality based piano teaching system. In: 2011 third international conference on intelligent human-machine systems and cybernetics. IEEE, vol 2, pp 47–52

  • Kersten-Oertel M, Gerard I, Drouin S et al (2015) Augmented reality in neurovascular surgery: feasibility and first uses in the operating room. Int J Comput Assist Radiol Surg 10(11):1823–1836

    Article  Google Scholar 

  • Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105

  • LeCun Y, Boser B, Denker JS et al (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551

    Article  Google Scholar 

  • LeCun Y, Bottou L, Bengio Y et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  • Lee H (2019) Real-time manufacturing modeling and simulation framework using augmented reality and stochastic network analysis. Virtual Real 23:85–99

    Article  Google Scholar 

  • Lima JP, Roberto R, Simões F et al (2017) Markerless tracking system for augmented reality in the automotive industry. Expert Syst Appl 82:100–114

    Article  Google Scholar 

  • Lu AS, Baranowski T, Thompson D, Buday R (2012) Story immersion of videogames for youth health promotion: a review of literature. Games Health J 1(3):199–204. https://doi.org/10.1089/g4h.2011.0012

    Article  Google Scholar 

  • Miller CH (2019) Digital Storytelling 4e: a creator's guide to interactive entertainment. CRC Press, Boca Raton

    Book  Google Scholar 

  • Plowman L (1996) Narrative, linearity and interactivity: making sense of interactive multimedia. Br J Educ Technol 27:92–105

    Article  Google Scholar 

  • Qin H, Patrick Rau PL, Salvendy G (2009) Measuring player immersion in the computer game narrative. Int J Hum-Comput Interact 25(2):107–133

    Article  Google Scholar 

  • Rehman U, Cao S (2017) Augmented-reality-based indoor navigation: a comparative analysis of handheld devices versus google glass. IEEE Trans Human-Mach Syst 47(1):140–151

    Google Scholar 

  • Shi X, Manduchi R (2003) A study on Bayes feature fusion for image classification. In: 2003 conference on computer vision and pattern recognition workshop. IEEE, vol 8, pp 95–95

  • Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  • Sun G, Qiu C, Qi Q et al (2015) The implementation of a live interactive augmented reality game creative system. In: International conference on cyberspace technology. IET

  • Szegedy C, Liu W, Jia Y et al (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9

  • Thomas BH (2012) A survey of visual, mixed, and augmented reality gaming. Comput Entertain 10(1):1–33

    Article  Google Scholar 

  • Warmelink H, Mayer I, Weber J, Heijligers B, Haggis M, Peters E, Louwerse M (2017) AMELIO: evaluating the team-building potential of a mixed reality escape room game. Extended abstracts publication of the annual symposium on computer-human interaction in play. ACM, Amsterdam, Netherlands, pp 111–123

    Chapter  Google Scholar 

  • Wolf MJP (2001) The medium of the video game. University of Texas Press, Austin

    Google Scholar 

  • Zhang H, Berg AC, Maire M et al (2006) SVM-KNN: discriminative nearest neighbor classification for visual category recognition. In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR'06). IEEE, vol 2, pp 2126–2136

Download references

Author information

Authors and Affiliations

  1. Department of Mechatronics, College of Mechanical Engineering, Chongqing University, Chongqing, 400044, China

    Hong Zeng, Xingxi He & Honghu Pan

Authors
  1. Hong Zeng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xingxi He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Honghu Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xingxi He.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zeng, H., He, X. & Pan, H. Implementation of escape room system based on augmented reality involving deep convolutional neural network. Virtual Reality 25, 585–596 (2021). https://doi.org/10.1007/s10055-020-00476-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-020-00476-0

Keywords

Search

Navigation