Abstract
In spatial augmented reality applications, incorrect projection mapping may occur when projecting images onto moving non-rigid surfaces. This may detract from the user experience, as the image may not be perceived as originally intended. This is especially apparent when using low-cost projectors and cameras or when surfaces are moving quickly. In this paper, an algorithm is developed which predicts the motion of a non-rigid surface, so that when an image is being projected onto the surface, the projection “matches” the surface shape, while using low-cost equipment. Using an interconnected mass–spring–damper system to model the surface, the surface position is predicted using a Kalman filter-based algorithm, which also compensates for the processing delays and fast-moving surfaces. To accurately model real-world materials, the mass–spring system parameters are found using a system identification approach. When the prediction algorithm is implemented experimentally, in real-time, the results show convergent results in the sense that the surface predictions converge to the measured position of a non-rigid surface. The error results show that the algorithm is both accurate and robust and can currently be applied in spatial augmented reality applications.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Anderson BDO, Moore JB (2012) Optimal filtering. Courier Corporation, North Chelmsford
Arasaratnam I (2009) Cubature Kalman filtering theory & applications. PhD thesis, McMaster University
Bermano AH, Billeter M, Iwai D, Grundhöfer A (2017) Makeup lamps: live augmentation of human faces via projection. In: Computer graphics forum, volume 36. Wiley Online Library, pp 311–323
Bradley D, Popa T, Sheffer A, Heidrich W, Boubekeur T (2008) Markerless garment capture. In: ACM transactions on graphics (TOG), volume 27. ACM, p 99
Bridson R, Marino S, Fedkiw R (2003) Simulation of clothing with folds and wrinkles. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation. Eurographics Association, pp 28–36
Chen Y, Rui Y, Huang TS et al (2006) Multicue hmm-ukf for real-time contour tracking. IEEE Trans Pattern Anal Mach Intell 28(9):1525
Choi K-J, Ko H-S (2005) Stable but responsive cloth. In: ACM SIGGRAPH 2005 courses, p 1. ACM
Eberhardt B, Weber A, Strasser W (1996) A fast, flexible, particle-system model for cloth draping. IEEE Comput Graph Appl 16(5):52–59
Gibson SFF, Mirtich B (1997) A survey of deformable modeling in computer graphics. Technical report, Citeseer
Gillette R, Peters C, Vining N, Edwards E, Sheffer A (2015) Real-time dynamic wrinkling of coarse animated cloth. In: Proceedings of the 14th ACM SIGGRAPH/eurographics symposium on computer animation, pp 17–26
Gomes Adam Daniel (2016) Prediction for projection on time-varying surfaces. Master’s thesis, University of Waterloo
Granville V, Krivánek M, Rasson J-P (1994) Simulated annealing: a proof of convergence. IEEE Trans Pattern Anal Mach Intell 16(6):652–656
Grinspun E, Hirani AN, Desbrun M, Schröder P (2003) Discrete shells. In: Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on computer animation. Eurographics Association, pp 62–67
Hilsmann A, Schneider DC, Eisert P (2010) Realistic cloth augmentation in single view video under occlusions. Comput Graph 34(5):567–574
Kavan L, Gerszewski D, Bargteil AW, Sloan P-P (2011) Physics-inspired upsampling for cloth simulation in games. In: ACM SIGGRAPH 2011 papers, pp 1–10
Kocev B, Ritter F, Linsen L (2014) Projector-based surgeon-computer interaction on deformable surfaces. Int J Comput Assist Radiol Surg 9(2):301–312
Livolsi B (2015) Input lag and projectors for gaming: what it is and why you should care. Projector Central. [Online]. http://www.projectorcentral.com/projector-input-lag.htm
Lloyd B, Székely G, Harders M (2007) Identification of spring parameters for deformable object simulation. IEEE Trans Visual Comput Graph 13(5):1081–1094
Narita G, Watanabe Y, Ishikawa M (2015) Dynamic projection mapping onto a deformable object with occlusion based on high-speed tracking of dot marker array. In: Proceedings of the 21st ACM symposium on virtual reality software and technology. ACM, pp 149–152
Nealen A, Müller M, Keiser R, Boxerman E, Carlson M (2006) Physically based deformable models in computer graphics. In: Computer graphics forum, volume 25. Wiley Online Library, pp 809–836
Piper B, Ratti C, Ishii H (2002) Illuminating clay: a 3-d tangible interface for landscape analysis. In: Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, pp 355–362
Natural Point (2011) Optitrack. Natural Point, Inc.,[Online]. urlhttp://www.naturalpoint. com/optitrack/
Provot X (1995) Deformation constraints in a mass-spring model to describe rigid cloth behaviour. In: Graphics interface. Canadian Information Processing Society, pp 147–147
Punpongsanon P, Iwai D, Sato K (2015) Projection-based visualization of tangential deformation of nonrigid surface by deformation estimation using infrared texture. Virtual Real 19(1):45–56
Sakamaki S, Hashimoto N (2013) Time-delay compensation for dynamic projection mapping. In: SIGGRAPH Asia 2013 Posters, pp 1–1
Steimle J, Jordt A, Maes P (2013) Flexpad: highly flexible bending interactions for projected handheld displays. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 237–246
Teschner M, Heidelberger B, Muller M, Gross M (2004) A versatile and robust model for geometrically complex deformable solids. In: Computer graphics international, 2004. Proceedings. IEEE, pp 312–319
Varol A, Salzmann M, Tola E, Fua P (2009) Template-free monocular reconstruction of deformable surfaces. In 2009 IEEE 12th international conference on computer vision. IEEE, pp 1811–1818
Acknowledgements
We would like to acknowledge Madeleine Wang for her help in running experiments, Cong Yue for her work with the CKF, and Yousef Sawires for moral support.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Supplementary material 1 (MP4 42055 kb)
Supplementary material 2 (MP4 43117 kb)
Rights and permissions
About this article
Cite this article
Gomes, A., Fernandes, K. & Wang, D. Surface Prediction for Spatial Augmented Reality Applications. Virtual Reality 25, 761–771 (2021). https://doi.org/10.1007/s10055-020-00490-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-020-00490-2