Distant augmented reality: Bringing a new dimension to user experience using drones
Introduction
Along with the ever-evolving technology, human needs have begun to change and increase everyday. With this rapid change, it became more difficult to satisfy people’s demands and expectations. These expectations are pushing the technology further in every aspect of life and multimedia technologies are no exception (Ge et al., 2017a; Yuan et al., 2013).
Augmented Reality (AR) presents more than what people see every day by placing virtual content such as text, image, sound or 3D models on a real-world images. AR helps people to improve their perceptions, and makes it easier to understand complex structures by enhancing the environment (Ribo et al., 2002). Azuma (1997) defines AR system with three characteristics: it integrates real and virtual objects in a real environment, runs interactively in real time and aligns real objects with virtual ones. In this definition, there is are constraints about displays in which virtual objects are blended with real ones. Hence, the display device can be a mobile phone, tablet or Head Mounted Display (HMD). Additionally, there is no restriction to how the real world images (Ge et al., 2017b) are captured. So, it can be using a see-through display, mobile device camera or even a drone camera which transmits images to a mobile device.
In this study, an AR application for cultural heritage was designed. The aim here is to present a novel method for AR which we coin: Distant Augmented Reality. To demonstrate this notion an application was developed. For a proof of concept, Roman Baths which is the one of the famous historical site of Ankara (Turkey), was chosen. The first step to realizing this was designing and optimizing the 3D model for the baths. Then two methods were developed for tracking the drone and the scene. First method is a GPS-based approach which includes receiving Global Positioning System (GPS) data from drone and converting this data to a proper format to be used. Second method is monocular scene tracking which implies finding the camera position and orientation according to the objects available in the environment. With these ideas presented and the proof of concept shown here, it will be possible to develop such systems in order to improve the user experience while visiting heritage sites.
The major contributions of this study can be listed as follows:
- •
The main novelty presented here is the use of drones to provide an additional “eye” to the heritage site, which will enhance the user experience in the learning of history.
- •
Two different tracking methods were presented, namely vision-based and GPS-based. These methods are used to combine the images acquired by the drone with the 3D models generated by the computer.
- •
Detailed high-fidelity model of Roman Baths was designed and created according to the reconstruction images by professional archaeologists.
The rest of the paper is structured as follows: Section 2 presents the user tracking methods in AR and gives examples from literature. Section 3 describes distant augmented reality system. Section 4 explains two methods which developed for tracking the drones position and orientation followed by Section 5 describing designing and blending the model with three subsections. Section 6 demonstrates developed applications final state and finally, Section 7 summarizes the study.
Section snippets
Literature review and background
The literature presents that AR can be applied to different application areas from engineering to medical (Billinghurst et al., 2015; Min et al., 2017). Tourism and Cultural Heritage was presented as an important and promising area for AR applications (Papagiannakis et al., 2008; Sekhavat, 2017). AR for cultural heritage is very interesting topic for preserving and transferring the ancient heritage to the next generations. AR applications for cultural heritage attracts interest to museums or
System architecture
We designed a novel system to augment the cultural heritage site using drones. In this system, visitors can see the heritage site in a form which they have never seen before. With the help of the unmanned aerial vehicle visitors will feel a whole new experience. According to this system, the video stream of the drone flying over the ruined bath area will be received by mobile devices which visitors hold, will be superimposed with the bath model on the mobile device. Thus, a very interesting
Tracking system
The main problem in AR system is tracking the user or the display device continuously and with minimum delay. In cultural heritage sites, visitors move actively in the area. So the tracking system must be very efficient to superimpose the models on the video accurately. In this study, two tracking systems were developed; GPS-based drone tracking and monocular camera-based scene tracking.
Augmentation system
In this section, methods used for augmenting the models on the acquired images will be explained. First, the modelling of the Roman Baths will be described. Then, optimization results of the model will be demonstrated. Finally, rendering method used will be explained.
Results
After the two methods were implemented, application was tested for both. Fig. 18, Fig. 19 show result of the GPS-based drone tracking and monocular scene tracking respectively. Even if some registering problems occurred, results were acceptable for GPS based method. Monocular scene tracking method resulted in more accurate and satisfactory results.
The measured frame per second of application can be seen in Fig. 20. The red point shows that the time in which the model was placed to the scene. As
Conclusion
AR is a promising and appealing research area for cultural heritage. It does not just make the ancient area more interesting but also allows to preserve original structures for future generations. In this paper, we presented a novel concept for AR in cultural heritage sites: Distant Augmented Reality. Firstly, a model of Roman Baths of Ankara was designed and optimized. Than, to demonstrate the method an application was designed using two different tracking methods. In the GPS-based method;
Acknowledgements
This study is funded by TÜBITAK Grant no: (3001:215E156) for the project titled “Development of a Mobile and Distant Augmented Reality System for Cultural Heritage Sites”.
References (40)
- et al.
General-purpose modular hardware and software framework for mobile outdoor augmented reality applications in engineering
Adv. Eng. Inf.
(2008) - et al.
Tourists’ intention to visit a destination: the role of augmented reality (ar) application for a heritage site
Comput. Hum. Behav.
(2015) 3d visualization and reflexive archaeology: a virtual reconstruction of çatalhöyük history houses
Digit.Appl.Archaeol.Cult. Herit.
(2017)- et al.
Computer simulation and vr model of the tesla’s wardenclyffe laboratory
Digit.Appl.Archaeol.Cult. Herit.
(2017) - et al.
Virtual and augmented reality for rich interaction with cultural heritage sites: a case study from the roman theater at byblos
Digit.Appl.Archaeol. Cult. Herit.
(2017) Roman baths of ankara, Ω
(2017)Ankara roma hamamı
A survey of augmented reality, Presence
Teleoperators & Virtual Environments
(1997)- et al.
A motion-stabilized outdoor augmented reality system
Virtual Reality, 1999. Proceedings
(1999) - et al.
A survey of augmented reality
Foundations and Trends® in Human–Computer Interaction
(2015)
User Tracking Methods for Augmented Reality Applications in Cultural Heritage
Making visits to museums more fun with augmented reality using kinect, drones and games
Int. J. Comput. Commun.
Vision-based user tracking for outdoor augmented reality
Augmented reality applications for cultural heritage using kinect
Human-centric.Comput. Inf. Sci.
Camera calibration toolbox for matlab
Ninja on a plane: automatic discovery of physical planes for augmented reality using visual slam
Archeoguide: system architecture of a mobile outdoor augmented reality system
Real positioning in virtual environments using game engines
Toward qoe-assured 4k video-on-demand delivery through mobile edge virtualization with adaptive prefetching
IEEE Trans. Multimed.
Cited by (21)
Smart counties: technologies, considerations, characteristics, challenges, policies, and theoretical concerns
2022, Smart Cities Policies and Financing: Approaches and SolutionsVirtual reconstruction of the historic city of Al-Zubarah in Qatar
2021, Digital Applications in Archaeology and Cultural HeritageCitation Excerpt :Now we see buildings in virtual reality in their original forms in their proposed presentation, rehabilitation, or new additions. Some studies use a single technique while others use a hybrid approach where, for example, combining photogrammetry and laser scanning to produce 3D documentation (Alshawabkeh et al., 2020; Granshaw, 2014; Kersten, 2007), conducting a comparative study between two different systems such as Terrestrial Laser Scanning vs Dense Image Matching (Kersten et al., 2015), applying monocular camera and GPS based drone to produce Augmented Reality (AR) (Unal et al., 2020), or producing architectural-historical 4D documentation by the application of photogrammetry, terrestrial laser scanning, and historical Analysis (Kersten et al., 2014). This section looks at Cairo, Petra, and Rome, as three cases to give an insight into the study of al-Zubarah.
Virtual and Augmented Experience in Virtual Learning Tours †
2023, Information (Switzerland)FlyVR in the Sky: Virtual Reality Framework for Unmanned Aerial Vehicles in B5G
2023, Lecture Notes in Networks and SystemsTechnology application and development trend of intelligent unmanned system
2022, Chinese Journal of Ship Research