Abstract
With the fast development of consumer-level RGB-D cameras, real-world indoor three-dimensional (3D) scene modeling and robotic applications are gaining more attention. However, indoor 3D scene modeling is still challenging because the structure of interior objects may be complex and the RGB-D data acquired by consumer-level sensors may have poor quality. There is a lot of research in this area. In this survey, we provide an overview of recent advances in indoor scene modeling methods, public indoor datasets and libraries which can facilitate experiments and evaluations, and some typical applications using RGB-D devices including indoor localization and emergency evacuation.
摘要
随着消费级RGB-D摄像机的快速发展, 真实世界的室内三维场景建模和机器人应用越来越受到重视. 然而, 室内三维场景建模仍具有挑战性, 因室内物体结构可能具有较高复杂性, 在此情况下, 消费者级传感器采集的RGB-D数据质量需进一步提升. 近年来, 在提高消费者级传感器采集的RGB-D数据质量方面, 有很多值得关注的研究. 本文介绍了室内场景建模方法的最新进展、 室内公共数据集和库以及RGB-D设备的典型应用, 包括室内定位和紧急疏散.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Biswas J, Veloso M, 2010. WiFi localization and navigation for autonomous indoor mobile robots. IEEE Int Conf on robotics and automation, p.4379–4384. https://doi.org/10.1109/ROBOT.2010.5509842
Chen AY, Chu JC, 2016. TDVRP and BIM integrated approach for in-building emergency rescue routing. J Comput Civil Eng, 30(5):C4015003. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000522
Chen AY, Huang T, 2015. Toward BIM-enabled decision making for in-building response missions. IEEE Trans Intell Transp Syst, 16(5):2765–2773. https://doi.org/10.1109/TITS.2015.2422138
Chen C, Yang BS, Song S, et al., 2018. Calibrate multiple consumer RGB-D cameras for low-cost and efficient 3D indoor mapping. Remot Sens, 10(2):328. https://doi.org/10.3390/rs10020328
Chen JW, Bautembach D, Izadi S, 2013. Scalable real-time volumetric surface reconstruction. ACM Trans Graph, 32(4):113. https://doi.org/10.1145/2461912.2461940
Chen K, Lai YK, Wu YX, et al., 2014. Automatic semantic modeling of indoor scenes from low-quality RGB-D data using contextual information. ACM Trans Graph, 33(6): 208. https://doi.org/10.1145/2661229.2661239
Chen SN, Zheng YL, Luo PP, et al., 2017. Visual search based indoor localization in low light via RGB-D camera. Int Comput Inform Eng, 11(3):403–406. https://doi.org/10.5281/zenodo.1129950
Cheng JCP, Tan Y, Song YZ, et al., 2018. Developing an evacuation evaluation model for offshore oil and gas platforms using BIM and agent-based model. Autom Constr, 89:214–224. https://doi.org/10.1016/j.autcon.2018.02.011
Chintalapudi K, Padmanabha IA, Padmanabhan VN, 2010. Indoor localization without the pain. Proc 16th Annual Int Conf on Mobile Computing and Networking, p.173–184. https://doi.org/10.1145/1859995.1860016
Choi J, Choi J, Kim I, 2014. Development of BIM-based evacuation regulation checking system for high-rise and complex buildings. Autom Constr, 46:38–49. https://doi.org/10.1016/j.autcon.2013.12.005
Dai A, Chang AX, Savva M, et al., 2017. ScanNet: richly-annotated 3D reconstructions of indoor scenes. IEEE Conf on Computer Vision and Pattern Recognition, p.2432–2443. https://doi.org/10.1109/CVPR.2017.261
dos Santos DR, Basso MA, Khoshelham K, et al., 2016. Mapping indoor spaces by adaptive coarse-to-fine registration of RGB-D data. IEEE Geosci Remot Sens Lett, 13(2):262–266. https://doi.org/10.1109/LGRS.2015.2508880
Endres F, Hess J, Sturm J, et al., 2014. 3-D mapping with an RGB-D camera. IEEE Trans Robot, 30(1):177–187. https://doi.org/10.1109/TRO.2013.2279412.
Engelhard N, Endres F, Hess J, et al., 2011. Real-time 3D visual SLAM with a hand-held RGB-D camera. Proc RGB-D Workshop on 3D Perception in Robotics at the European Robotics Forum, 180:1–15.
Fehr M, Furrer F, Dryanovski I, et al., 2017. TSDF-based change detection for consistent long-term dense reconstruction and dynamic object discovery. IEEE Int Conf on Robotics and Automation, p.5237–5244. https://doi.org/10.1109/ICRA.2017.7989614
Fox D, Burgard W, Thrun S, 1999. Markov localization for mobile robots in dynamic environments. J Artif Intell Res, 11:391–427. https://doi.org/10.1613/jair.616
Handa A, Whelan T, McDonald J, et al., 2014. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. IEEE Int Conf on Robotics and Automation, p.1524–1531. https://doi.org/10.1109/ICRA.2014.6907054
He FN, Habib A, 2018. Three-point-based solution for automated motion parameter estimation of a multi-camera indoor mapping system with planar motion constraint. ISPRS J Photogr Remot Sens, 142:278–291. https://doi.org/10.1016/j.isprsjprs.2018.06.011
Helbing D, Farkas I, Vicsek T, 2000. Simulating dynamical features of escape panic. Nature, 407(6803):487–490. https://doi.org/10.1038/35035023
Henry P, Krainin M, Herbst E, et al., 2012. RGB-D mapping: using Kinect-style depth cameras for dense 3D modeling of indoor environments. Int J Robot Res, 31(5):647–663. https://doi.org/10.1177/0278364911434148
Huang AS, Bachrach A, Henry P, et al., 2017. Visual odometry and mapping for autonomous flight using an RGB-D camera. In: Christensen HI, Khatib O (Eds.), Robotics Research. Springer, Cham, p.235–252. https://doi.org/10.1007/978-3-319-29363-9_14
Ikehata S, Yang H, Furukawa Y, 2015. Structured indoor modeling. IEEE Int Conf on Computer Vision, p.1323–1331. https://doi.org/10.1109/ICCV.2015.156
Kerl C, Sturm J, Cremers D, 2013. Robust odometry estimation for RGB-D cameras. IEEE Int Conf on Robotics and Automation, p.3748–3754. https://doi.org/10.1109/ICRA.2013.6631104
Konolige K, Agrawal M, 2008. FrameSLAM: from bundle adjustment to real-time visual mapping. IEEE Trans Robot, 24(5):1066–1077. https://doi.org/10.1109/tro.2008.2004832
Lai K, Bo L, Fox D, 2014. Unsupervised feature learning for 3D scene labeling. IEEE Int Conf on Robotics and Automation, p.3050–3057. https://doi.org/10.1109/ICRA.2014.6907298
Li M, Chen RZ, Liao X, et al., 2018. A real-time indoor visual localization and navigation method based on Tango smartphone. Ubiquitous Positioning Indoor Navigation and Location Based Service, p.1–6. https://doi.org/10.1109/UPINLBS.2018.8559720
Liu HM, Zhang GF, Bao HJ, 2016. Robust keyframe-based monocular SLAM for augmented reality. IEEE Int Symp on Mixed and Augmented Reality, p.1–10. https://doi.org/10.1109/ISMAR-Adjunct.2016.0111
Liu R, Du J, Issa RRA, 2014. Human library for emergency evacuation in BIM-based serious game environment. Int Conf on Computing in Civil and Building Engineering, p.544–551. https://doi.org/10.1061/9780784413616.068
Ma J, Jia W, Zhang J, 2017. Research of building evacuation path to guide based on BIM. 29th Chinese Control and Decision Conf, p.1814–1818. https://doi.org/10.1109/CCDC.2017.7978811
Mur-Artal R, Tardós JD, 2017. ORB-SLAM2: an open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Trans Robot, 33(5):1255–1262. https://doi.org/10.1109/TRO.2017.2705103
Newcombe RA, Izadi S, Hilliges O, et al., 2011. KinectFusion: real-time dense surface mapping and tracking. 10th IEEE Int Symp on Mixed and Augmented Reality, p.127–136. https://doi.org/10.1109/ISMAR.2011.6092378
Nießner M, Zollhöfer M, Izadi S, et al., 2013. Real-time 3D reconstruction at scale using voxel hashing. ACM Trans Graph, 32(6):169. https://doi.org/10.1145/2508363.2508374
Ortiz LE, Cabrera VE, Gonçalves LMG, 2018. Depth data error modeling of the ZED 3D vision sensor from stereolabs. ELCVIA Electron Lett Comput Vis Imag Anal, 17(1):1–15. https://doi.org/10.5565/rev/elcvia.1084
Rüppel U, Schatz K, 2011. Designing a BIM-based serious game for fire safety evacuation simulations. Adv Eng Inform, 25(4):600–611. https://doi.org/10.1016/j.aei.2011.08.001
Santos R, Costa AA, Grilo A, 2017. Bibliometric analysis and review of building information modelling literature published between 2005 and 2015. Autom Constr, 80:118–136. https://doi.org/10.1016/j.autcon.2017.03.005
Segal A, Haehnel D, Thrun S, 2009. Generalized-ICP. Proc Robotics: Science and Systems, 2:4. https://doi.org/10.15607/RSS.2009.V.021
Shi Y, Xu K, Nießner M, et al., 2018. PlaneMatch: patch coplanarity prediction for robust RGB-D reconstruction. LNCS, 11212:767–784. https://doi.org/10.1007/978-3-030-01237-3_46
Song S, Lichtenberg SP, Xiao J, 2015. SUN RGB-D: a RGB-D scene understanding benchmark suite. IEEE Conf on Computer Vision and Pattern Recognition, p.567–576. https://doi.org/10.1109/CVPR.2015.7298655
Steinbrucker F, Sturm J, Cremers D, 2011. Real-time visual odometry from dense RGB-D images. IEEE Int Conf on Computer Vision Workshops, p.719–722. https://doi.org/10.1109/ICCVW.2011.6130321
Sturm J, Engelhard N, Endres F, et al., 2012. A benchmark for the evaluation of RGB-D SLAM systems. IEEE/RSJ Int Conf on Intelligent Robots and Systems, p.573–580. https://doi.org/10.1109/IROS.2012.6385773
Tang SJ, Li Y, Yuan ZL, et al., 2019a. A vertex-to-edge weighted closed-form method for dense RGB-D indoor SLAM. IEEE Access, 7:32019–32029. https://doi.org/10.1109/ACCESS.2019.2900990
Tang SJ, Zhang YJ, Li Y, et al., 2019b. Fast and automatic reconstruction of semantically rich 3D indoor maps from low-quality RGB-D sequences. Sensors, 19(3):533. https://doi.org/10.3390/s19030533
Thomas D, Sugimoto A, 2017. Modeling large-scale indoor scenes with rigid fragments using RGB-D cameras. Comput Vis Imag Underst, 157:103–116. https://doi.org/10.1016/j.cviu.2016.11.008
Vestena KM, dos Santos DR, Oilveira EM Jr, et al., 2016. A weighted closed-form solution for RGB-D data registration. Int Arch Photogr Remote Sens Spat Inform Sci, XLI-B3:403–409. https://doi.org/10.5194/isprsarchives-XLI-B3-403-2016
Wang KK, Zhang GF, Bao HJ, 2014. Robust 3D reconstruction with an RGB-D camera. IEEE Trans Imag Process, 23(11):4893–4906. https://doi.org/10.1109/TIP.2014.2352851
Westoby MJ, Brasington J, Glasser NF, et al., 2012. ‘Structure-from-motion’ photogrammetry: a low-cost, effective tool for geoscience applications. Geomorphology, 179:300–314. https://doi.org/10.1016/j.geomorph.2012.08.021
Whelan T, Michael K, Maurice F, et al., 2013. Kintinuous: Spatially Extended Kinectfusion. Technical Report No. MIT-CSAIL-TR-2012-020. https://doi.org/10.1721.1/71756
Winterhalter W, Fleckenstein F, Steder B, et al., 2015. Accurate indoor localization for RGB-D smartphones and tablets given 2D floor plans. IEEE/RSJ Int Conf on Intelligent Robots and Systems, p.3138–3143. https://doi.org/10.1109/IROS.2015.7353811
Yuan ZL, Jia HF, Liao MJ, et al., 2017. Simulation model of self-organizing pedestrian movement considering following behavior. Front Inform Technol Electron Eng, 18(8):1142–1150. https://doi.org/10.1631/FITEE.1601592
Yuan ZL, Jia HF, Zhang LF, et al., 2018. A social force evacuation model considering the effect of emergency signs. Simulation, 94(8):723–737. https://doi.org/10.1177/0037549717741350
Yuan ZL, Guo RZ, Tang SJ, et al., 2019. Simulation of the separating crowd behavior in a T-shaped channel based on the social force model. IEEE Access, 7:13668–13682. https://doi.org/10.1109/ACCESS.2019.2894345
Zeng M, Zhao FK, Zheng JX, et al., 2012. A memory-efficient KinectFusion using octree. In: Hu SM, Martin RR (Eds.), Computational Visual Media. Springer, Berlin, p.234–241. https://doi.org/10.1007/978-3-642-34263-9_30
Zou H, Li N, Cao L, 2016. Immersive virtual environments for investigating building emergency evacuation behaviors: a feasibility study. 33th Int Symp on Automation and Robotics in Construction, p.1–8. https://doi.org/10.22260/ISARC2016/0040
Author information
Authors and Affiliations
Contributions
Zhilu YUAN designed the research. Shengjun TANG collected the information of RGB-D devices related products, technologies, and databases. You LI and Weixi WANG drafted the manuscript. Ming LI helped organize the manuscript. Renzhong GUO and Shengjun TANG revised and finalized the paper.
Corresponding author
Ethics declarations
Zhilu YUAN, You LI, Shengjun TANG, Ming LI, Renzhong GUO, and Weixi WANG declare that they have no conflict of interest.
Additional information
Project supported by the National Natural Science Foundation of China (Nos. 71901147, 41801392, 41901329, 41971354, and 41971341), the Research Program of Shenzhen S&T Innovation Committee, China (No. JCYJ20180305125131482), the Open Fund of Key Laboratory of Urban Land Resources Monitoring and Simulation, MNR, China (Nos. KF-2019-04-010, KF-2019-04-014, and KF-2018-03-066), the Natural Science Foundation of Guangdong Province, China (Nos. 2019A1515010748 and 2019A1515011872), the Foundation of High-Level University Phase II, China (No. 000002110335), the Foundation of Shenzhen University for New Researchers, China (No. 2019056), the Innovation Team Program of Department Education of Guangdong Province, China (No. 2017KCXTD028), and the Guangdong Science and Technology Strategic Innovation Fund (the Guangdong-Hong Kong-Macau Joint Laboratory Program) (No. 2020B1212030009)
Rights and permissions
About this article
Cite this article
Yuan, Z., Li, Y., Tang, S. et al. A survey on indoor 3D modeling and applications via RGB-D devices. Front Inform Technol Electron Eng 22, 815–826 (2021). https://doi.org/10.1631/FITEE.2000097
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1631/FITEE.2000097