Distant augmented reality: Bringing a new dimension to user experience using drones

https://doi.org/10.1016/j.daach.2020.e00140Get rights and content

Abstract

Augmented Reality (AR) is a live view that blends real world images with virtual information such as 3D models or sounds. AR is used to virtually reconstruct the ancient buildings of cultural heritage sites. Such sites can be presented to tourist groups in the form they were built in the past. AR can be implemented using mobile phones, tablets or smart glasses. Here, we present a new dimension to AR for cultural heritage using drones. We name it “Distant Augmented Reality”. The goal of this study is to augment video imagery acquired using drones with 3D models of the Roman Baths which is an important cultural heritage site in Turkey. For this purpose, two different tracking techniques were implemented: GPS based drone tracking and monocular camera based scene tracking. With these two techniques drone position and orientation was detected and tracked for augmenting the 3D model of the baths.

Introduction

Along with the ever-evolving technology, human needs have begun to change and increase everyday. With this rapid change, it became more difficult to satisfy people’s demands and expectations. These expectations are pushing the technology further in every aspect of life and multimedia technologies are no exception (Ge et al., 2017a; Yuan et al., 2013).

Augmented Reality (AR) presents more than what people see every day by placing virtual content such as text, image, sound or 3D models on a real-world images. AR helps people to improve their perceptions, and makes it easier to understand complex structures by enhancing the environment (Ribo et al., 2002). Azuma (1997) defines AR system with three characteristics: it integrates real and virtual objects in a real environment, runs interactively in real time and aligns real objects with virtual ones. In this definition, there is are constraints about displays in which virtual objects are blended with real ones. Hence, the display device can be a mobile phone, tablet or Head Mounted Display (HMD). Additionally, there is no restriction to how the real world images (Ge et al., 2017b) are captured. So, it can be using a see-through display, mobile device camera or even a drone camera which transmits images to a mobile device.

In this study, an AR application for cultural heritage was designed. The aim here is to present a novel method for AR which we coin: Distant Augmented Reality. To demonstrate this notion an application was developed. For a proof of concept, Roman Baths which is the one of the famous historical site of Ankara (Turkey), was chosen. The first step to realizing this was designing and optimizing the 3D model for the baths. Then two methods were developed for tracking the drone and the scene. First method is a GPS-based approach which includes receiving Global Positioning System (GPS) data from drone and converting this data to a proper format to be used. Second method is monocular scene tracking which implies finding the camera position and orientation according to the objects available in the environment. With these ideas presented and the proof of concept shown here, it will be possible to develop such systems in order to improve the user experience while visiting heritage sites.

The major contributions of this study can be listed as follows:

  • The main novelty presented here is the use of drones to provide an additional “eye” to the heritage site, which will enhance the user experience in the learning of history.

  • Two different tracking methods were presented, namely vision-based and GPS-based. These methods are used to combine the images acquired by the drone with the 3D models generated by the computer.

  • Detailed high-fidelity model of Roman Baths was designed and created according to the reconstruction images by professional archaeologists.

The rest of the paper is structured as follows: Section 2 presents the user tracking methods in AR and gives examples from literature. Section 3 describes distant augmented reality system. Section 4 explains two methods which developed for tracking the drones position and orientation followed by Section 5 describing designing and blending the model with three subsections. Section 6 demonstrates developed applications final state and finally, Section 7 summarizes the study.

Section snippets

Literature review and background

The literature presents that AR can be applied to different application areas from engineering to medical (Billinghurst et al., 2015; Min et al., 2017). Tourism and Cultural Heritage was presented as an important and promising area for AR applications (Papagiannakis et al., 2008; Sekhavat, 2017). AR for cultural heritage is very interesting topic for preserving and transferring the ancient heritage to the next generations. AR applications for cultural heritage attracts interest to museums or

System architecture

We designed a novel system to augment the cultural heritage site using drones. In this system, visitors can see the heritage site in a form which they have never seen before. With the help of the unmanned aerial vehicle visitors will feel a whole new experience. According to this system, the video stream of the drone flying over the ruined bath area will be received by mobile devices which visitors hold, will be superimposed with the bath model on the mobile device. Thus, a very interesting

Tracking system

The main problem in AR system is tracking the user or the display device continuously and with minimum delay. In cultural heritage sites, visitors move actively in the area. So the tracking system must be very efficient to superimpose the models on the video accurately. In this study, two tracking systems were developed; GPS-based drone tracking and monocular camera-based scene tracking.

Augmentation system

In this section, methods used for augmenting the models on the acquired images will be explained. First, the modelling of the Roman Baths will be described. Then, optimization results of the model will be demonstrated. Finally, rendering method used will be explained.

Results

After the two methods were implemented, application was tested for both. Fig. 18, Fig. 19 show result of the GPS-based drone tracking and monocular scene tracking respectively. Even if some registering problems occurred, results were acceptable for GPS based method. Monocular scene tracking method resulted in more accurate and satisfactory results.

The measured frame per second of application can be seen in Fig. 20. The red point shows that the time in which the model was placed to the scene. As

Conclusion

AR is a promising and appealing research area for cultural heritage. It does not just make the ancient area more interesting but also allows to preserve original structures for future generations. In this paper, we presented a novel concept for AR in cultural heritage sites: Distant Augmented Reality. Firstly, a model of Roman Baths of Ankara was designed and optimized. Than, to demonstrate the method an application was designed using two different tracking methods. In the GPS-based method;

Acknowledgements

This study is funded by TÜBITAK Grant no: (3001:215E156) for the project titled “Development of a Mobile and Distant Augmented Reality System for Cultural Heritage Sites”.

References (40)

  • G.E. Bostanci

    User Tracking Methods for Augmented Reality Applications in Cultural Heritage

    (2014)
  • E. Bostanci et al.

    Making visits to museums more fun with augmented reality using kinect, drones and games

    Int. J. Comput. Commun.

    (2016)
  • E. Bostanci et al.

    Vision-based user tracking for outdoor augmented reality

  • E. Bostanci et al.

    Augmented reality applications for cultural heritage using kinect

    Human-centric.Comput. Inf. Sci.

    (2015)
  • J.-Y. Bouguet

    Camera calibration toolbox for matlab

  • D. Chekhlov et al.

    Ninja on a plane: automatic discovery of physical planes for augmented reality using visual slam

  • P. Dahne et al.

    Archeoguide: system architecture of a mobile outdoor augmented reality system

  • R. De Chiara et al.

    Real positioning in virtual environments using game engines

  • Dji company Dji
  • C. Ge et al.

    Toward qoe-assured 4k video-on-demand delivery through mobile edge virtualization with adaptive prefetching

    IEEE Trans. Multimed.

    (2017)
  • Cited by (21)

    • Virtual reconstruction of the historic city of Al-Zubarah in Qatar

      2021, Digital Applications in Archaeology and Cultural Heritage
      Citation Excerpt :

      Now we see buildings in virtual reality in their original forms in their proposed presentation, rehabilitation, or new additions. Some studies use a single technique while others use a hybrid approach where, for example, combining photogrammetry and laser scanning to produce 3D documentation (Alshawabkeh et al., 2020; Granshaw, 2014; Kersten, 2007), conducting a comparative study between two different systems such as Terrestrial Laser Scanning vs Dense Image Matching (Kersten et al., 2015), applying monocular camera and GPS based drone to produce Augmented Reality (AR) (Unal et al., 2020), or producing architectural-historical 4D documentation by the application of photogrammetry, terrestrial laser scanning, and historical Analysis (Kersten et al., 2014). This section looks at Cairo, Petra, and Rome, as three cases to give an insight into the study of al-Zubarah.

    View all citing articles on Scopus
    View full text