Lei Chu
Skip Abstract Section
Abstract
We present a novel approach for completing and reconstructing 3D shapes from incomplete scanned data by using deep neural networks. Rather than being trained on supervised completion tasks and applied on a testing shape, the network is optimized from scratch on the single testing shape to fully adapt to the shape and complete the missing data using contextual guidance from the known regions. The ability to complete missing data by an untrained neural network is usually referred to as the deep prior. In this article, we interpret the deep prior from a neural tangent kernel (NTK) perspective and show that the completed shape patches by the trained CNN are naturally similar to existing patches, as they are proximate in the kernel feature space induced by NTK. The interpretation allows us to design more efficient network structures and learning mechanisms for the shape completion and reconstruction task. Being more aware of structural regularities than both traditional and other unsupervised learning-based reconstruction methods, our approach completes large missing regions with plausible shapes and complements supervised learning-based methods that use database priors by requiring no extra training dataset and showing flexible adaptation to a particular shape instance.
- Nachman Aronszajn. 1950. Theory of reproducing kernels. Trans. Amer. Math. Soc. 68, 3 (1950), 337–404. Google Scholar
Cross Ref
- Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, Russ R. Salakhutdinov, and Ruosong Wang. 2019. On exact computation with an infinitely wide neural net. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 8141–8150. Google ScholarDigital Library
- Matan Atzmon and Yaron Lipman. 2020. SAL: Sign agnostic learning of shapes from raw data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’20). 2565–2574.Google ScholarCross Ref
- Gerhard H. Bendels, Ruwen Schnabel, and Reinhard Klein. 2005. Detail-preserving surface inpainting. In Proceedings of the Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST’05), Mark Mudge, Nick Ryan, and Roberto Scopigno (Eds.). The Eurographics Association. Google ScholarDigital Library
- Matthew Berger, Joshua A. Levine, Luis Gustavo Nonato, Gabriel Taubin, and Claudio T. Silva. 2013. A benchmark for surface reconstruction. ACM Trans. Graph. 32, 2 (Apr. 2013). Google ScholarDigital Library
- Matthew Berger, Andrea Tagliasacchi, Lee M. Seversky, Pierre Alliez, Joshua A. Levine, Andrei Sharf, and Claudio Silva. 2014. State of the art in surface reconstruction from point clouds. In Proceedings of the Eurographics STAR (Proc. of EG’14), Sylvain Lefebvre and Michela Spagnuolo (Eds.). The Eurographics Association.Google Scholar
- Rohan Chabra, Jan Eric Lenssen, Eddy Ilg, Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. 2020. Deep local shapes: Learning local SDF priors for detailed 3D reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV’20).Google Scholar
- Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. 2015. ShapeNet: An information-rich 3D model repository. CoRR abs/1512.03012 (2015).Google Scholar
- Zezhou Cheng, Matheus Gadelha, Subhransu Maji, and Daniel Sheldon. 2019. A Bayesian perspective on the deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google ScholarCross Ref
- Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. 2020. Implicit functions in feature space for 3D shape reconstruction and completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’20).Google ScholarCross Ref
- Ulrich Clarenz, Udo Diewald, Gerhard Dziuk, Martin Rumpf, and Raluca E. Rusu. 2004. A finite element method for surface restoration with smooth boundary conditions. Comput.-aided Geom. Des. 21, 5 (2004), 427–445. Google ScholarDigital Library
- Brian Curless and Marc Levoy. 1996. A volumetric method for building complex models from range images. In Proceedings of the 23rd Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’96). Association for Computing Machinery, New York, NY, 303–312. Google ScholarDigital Library
- Angela Dai, Christian Diller, and Matthias Nießner. 2020. SG-NN: Sparse generative neural networks for self-supervised scene completion of RGB-D scans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’20).Google ScholarCross Ref
- Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner. 2017. Shape completion using 3D-encoder-predictor CNNs and shape synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google ScholarCross Ref
- James Davis, Stephen R. Marschner, Matt Garr, and Marc Levoy. 2002. Filling holes in complex surfaces using volumetric diffusion. In Proceedings of the 1st International Symposium on 3D Data Processing Visualization and Transmission. 428–441.Google ScholarCross Ref
- Simon S. Du, Kangcheng Hou, Russ R. Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu Xu. 2019. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 5723–5733. Google ScholarDigital Library
- Haoqiang Fan, Hao Su, and Leonidas J. Guibas. 2017. A point set generation network for 3D object reconstruction from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17).Google Scholar
- Michael Firman, Oisin Mac Aodha, Simon Julier, and Gabriel J. Brostow. 2016. Structured prediction of unobserved voxels from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’16).Google Scholar
- Free3D. 2020. Free3D. Retrieved from: https://free3d.com/.Google Scholar
- Matheus Gadelha, Rui Wang, and Subhransu Maji. 2019. Shape reconstruction using differentiable projections and deep priors. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’19).Google ScholarCross Ref
- Yossi Gandelsman, Assaf Shocher, and Michal Irani. 2019. “Double-DIP”: Unsupervised image decomposition via coupled deep-image-priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google ScholarCross Ref
- Pallabi Ghosh, Vibhav Vineet, Larry S. Davis, Abhinav Shrivastava, Sudipta Sinha, and Neel Joshi. 2020. Deep Depth Prior for Multi-View Stereo. arxiv:cs.CV/2001.07791Google Scholar
- Rafael C. Gonzalez and Richard E. Woods. 2002. Digital Image Processing. Prentice Hall. 2002276271Google Scholar
- Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 2018. 3D semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google Scholar
- Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. 2020. Implicit geometric regularization for learning shapes. In Proceedings of the International Conference on Machine Learning (ICML’20).Google Scholar
- Xiaoguang Han, Zhen Li, Haibin Huang, Evangelos Kalogerakis, and Yizhou Yu. 2017. High-resolution shape completion using deep neural networks for global structure and local geometry inference. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’17). Google ScholarCross Ref
- Boris Hanin and Mihai Nica. 2020. Finite depth and width corrections to the neural tangent kernel. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or. 2019. MeshCNN: A network with an edge. ACM Trans, Graph, (SIGGRAPH) 38, 4 (2019). Google ScholarDigital Library
- Rana Hanocka, Gal Metzer, Raja Giryes, and Daniel Cohen-Or. 2020. Point2Mesh: A self-prior for deformable meshes. ACM Trans. Graph. (SIGGRAPH) 39, 4 (2020). Google ScholarDigital Library
- Gur Harary, Ayellet Tal, and Eitan Grinspun. 2014. Context-based coherent surface completion. ACM Trans. Graph. 33, 1 (2014). Google ScholarDigital Library
- Reinhard Heckel and Paul Hand. 2019. Deep decoder: Concise image representations from untrained non-convolutional networks. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Pedro Hermosilla, Tobias Ritschel, and Timo Ropinski. 2019. Total denoising: Unsupervised learning of 3D point cloud cleaning. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’19).Google Scholar
- Hui Huang, Shihao Wu, Daniel Cohen-Or, Minglun Gong, Hao Zhang, Guiqing Li, and Baoquan Chen. 2013. L1-medial skeleton of point cloud. ACM Trans. Graph. 32, 4 (2013). Google ScholarDigital Library
- Zhiyang Huang, Nathan Carr, and Tao Ju. 2019. Variational implicit point set surfaces. ACM Trans. Graph. (SIGGRAPH) 38, 4 (2019). Google ScholarDigital Library
- Arthur Jacot, Franck Gabriel, and Clement Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 8571–8580. Google ScholarDigital Library
- Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. 2020. Local implicit grid representations for 3D scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’20).Google ScholarCross Ref
- Tao Ju. 2009. Fixing geometric errors on polygonal models: A survey. J. Comput. Sci. Technol. 24, 1 (2009), 19–29. Google ScholarDigital Library
- Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson surface reconstruction. In Proceedings of the 4th Eurographics Symposium on Geometry Processing. Google ScholarDigital Library
- Michael Kazhdan and Hugues Hoppe. 2013. Screened Poisson surface reconstruction. ACM Trans. Graph. 32, 3 (2013). Google ScholarDigital Library
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).Google Scholar
- Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Trans. Graph. 36, 4 (2017). Google ScholarDigital Library
- Vladislav Kraevoy and Alla Sheffer. 2005. Template-based mesh completion. In Proceedings of the Eurographics Symposium on Geometry Processing, Mathieu Desbrun and Helmut Pottmann (Eds.). The Eurographics Association. Google ScholarDigital Library
- Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998).Google ScholarCross Ref
- Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. 2019. Wide neural networks of any depth evolve as linear models under gradient descent. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 8572–8583. Google ScholarDigital Library
- Guo Li, Ligang Liu, Hanlin Zheng, and Niloy J. Mitra. 2010. Analysis, reconstruction and manipulation using arterial snakes. ACM Trans. Graph. 29, 6 (2010). Google ScholarDigital Library
- Yangyan Li, Qian Zheng, Andrei Sharf, Daniel Cohen-Or, Baoquan Chen, and Niloy J. Mitra. 2011. 2D-3D fusion for layer decomposition of urban facades. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’11). Google ScholarDigital Library
- Peter Liepa. 2003. Filling holes in meshes. In Proceedings of the Eurographics/ACM SIGGRAPH Symposium on Geometry Processing (SGP’03). Eurographics Association, Goslar, DEU, 200–205. Google ScholarDigital Library
- William E. Lorensen and Harvey E. Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. ACM Siggraph Comput. Graph. 21, 4 (1987), 163–169. Google ScholarDigital Library
- Niloy J. Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan. 2013. Symmetry in 3D geometry: Extraction and applications. Comput. Graph. Forum 32, 6 (2013). Google ScholarDigital Library
- Fangzhou Mu, Yingyu Liang, and Yin Li. 2020. Gradients as features for deep representation learning. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. 2020. Neural tangents: Fast and easy infinite neural networks in Python. In Proceedings of the International Conference on Learning Representations.Google Scholar
- Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. 2019. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google ScholarCross Ref
- Mark Pauly, Niloy J. Mitra, Joachim Giesen, Markus H. Gross, and Leonidas J. Guibas. 2005. Example-based 3D scan completion. In Proceedings of the Symposium on Geometry Processing. 23–32. Google ScholarDigital Library
- Mark Pauly, Niloy J. Mitra, Johannes Wallner, Helmut Pottmann, and Leonidas J. Guibas. 2008. Discovering structural regularity in 3D geometry. ACM Trans. Graph. (SIGGRAPH) 27, 3 (2008). Google ScholarDigital Library
- Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. 2020. Convolutional occupancy networks. In Proceedings of the European Conference on Computer Vision (ECCV’20).Google Scholar
- Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess. 2016. Unsupervised learning of 3D structure from images. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 4996–5004. Google ScholarDigital Library
- Gernot Riegler, Ali Osman Ulusoy, Horst Bischof, and Andreas Geiger. 2017. Octnetfusion: Learning depth fusion from data. In Proceedings of the International Conference on 3D Vision (3DV’17). IEEE, 57–66.Google ScholarCross Ref
- Jason Rock, Tanmay Gupta, Justin Thorsen, JunYoung Gwak, Daeyun Shin, and Derek Hoiem. 2015. Completing 3D object shape from one depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’15).Google ScholarCross Ref
- Ruwen Schnabel, Patrick Degener, and Reinhard Klein. 2009. Completion and reconstruction with primitive shapes. In Computer Graphics Forum, Vol. 28. Wiley Online Library, 503–512.Google Scholar
- Bernhard Schölkopf, Ralf Herbrich, and Alex J. Smola. 2001. A generalized representer theorem. In Proceedings of the International Conference on Computational Learning Theory. Springer, 416–426. Google ScholarDigital Library
- Bernhard Schölkopf, Alexander J. Smola, and Klaus-Robert Müller. 1999. Kernel principal component analysis. In Advances in Kernel Methods: Support Vector Learning. The MIT Press, Cambridge, MA, 327–352. Google ScholarDigital Library
- Andrei Sharf, Marc Alexa, and Daniel Cohen-Or. 2004. Context-based surface completion. ACM Trans. Graph. 23, 3 (2004). Google ScholarDigital Library
- Abhishek Sharma, Oliver Grau, and Mario Fritz. 2016. VConv-DAE: Deep volumetric shape learning without object labels. In Proceedings of the European Conference on Computer Vision. Springer, 236–250.Google ScholarCross Ref
- John Shawe-Taylor and Nello Cristianini. 2004. Kernel Methods for Pattern Analysis. Cambridge University Press. Google ScholarDigital Library
- Chao-Hui Shen, Hongbo Fu, Kang Chen, and Shi-Min Hu. 2012. Structure recovery by part assembly. ACM Trans. Graph. (SIGGRAPH Asia) 31, 6 (2012). Google ScholarDigital Library
- Ivan Sipiran, Robert Gregor, and Tobias Schreck. 2014. Approximate symmetry detection in partial 3D meshes. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 131–140. Google ScholarDigital Library
- Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. 2020. Implicit neural representations with periodic activation functions. In Proceedings of the International Conference on Advances in Neural Information Processing Systems.Google Scholar
- Edward Smith and David Meger. 2017. Improved adversarial systems for 3D object generation and reconstruction. arXiv preprint arXiv:1707.09557 (2017).Google Scholar
- Pablo Speciale, Martin R. Oswald, Andrea Cohen, and Marc Pollefeys. 2016. A symmetry prior for convex variational 3D reconstruction. In Proceedings of the European Conference on Computer Vision. Springer, 313–328.Google ScholarCross Ref
- David Stutz and Andreas Geiger. 2018. Learning 3D shape completion from laser scan data with weak supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18).Google ScholarCross Ref
- Minhyuk Sung, Vladimir G. Kim, Roland Angst, and Leonidas Guibas. 2015. Data-driven structural priors for shape completion. ACM Trans. Graph. (SIGGRAPH Asia) 34, 6 (2015). Google ScholarDigital Library
- Julián Tachella, Junqi Tang, and Mike Davies. 2020. CNN denoisers as non-local filters: The neural tangent denoiser. arXiv preprint arXiv:2006.02379 (2020).Google Scholar
- Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. 2020. Fourier features let networks learn high frequency functions in low dimensional domains. In Proceedings of the International Conference on Advances in Neural Information Processing Systems.Google Scholar
- Maxim Tatarchenko, Stephan R. Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox. 2019. What do single-view 3D reconstruction networks learn? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google ScholarCross Ref
- Sebastian Thrun and Ben Wegbreit. 2005. Shape from symmetry. In Proceedings of the IEEE International Conference on Computer Vision (ICCV’05). Google ScholarDigital Library
- Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’18). Google Scholar
- Max Welling and Yee Whye Teh. 2011. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the International Conference on Machine Learning (ICML’11). Google ScholarDigital Library
- Francis Williams, Teseo Schneider, Claudio Silva, Denis Zorin, Joan Bruna, and Daniele Panozzo. 2019. Deep geometric prior for surface reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’19).Google ScholarCross Ref
- Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google Scholar
- Guoliang Xu. 2009. Mixed finite element methods for geometric modeling using general fourth order geometric flows. Comput.-aided Geom. Des. 26, 4 (May 2009). Google ScholarDigital Library
- Wei Zhao, Shuming Gao, and Hongwei Lin. 2007. A robust hole-filling algorithm for triangular mesh. Vis. Comput. 23, 12 (2007), 987–997. Google ScholarDigital Library
- Qingnan Zhou and Alec Jacobson. 2016. Thingi10K: A dataset of 10,000 3D-printing models. arXiv:1605.04797 (2016).Google Scholar
- Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. 2018. Open3D: A modern library for 3D data processing. arXiv:1801.09847 (2018).Google Scholar
Index Terms
- Unsupervised Shape Completion via Deep Prior in the Neural Tangent Kernel Perspective
Recommendations
Deep learning of curvature features for shape completion
AbstractThe paper presents a novel solution to the issue of incomplete regions in 3D meshes obtained through digitization. Traditional methods for estimating the surface of missing geometry and topology often yield unrealistic outcomes for intricate ...
Graphical abstractDisplay Omitted
Highlights- We perform 3D surface reconstructions using generative inpainting techniques.
- 2D representation of a 3D surface geometry based on its curvature.
- Application of a general purpose neural network for inpainting.
- Our approach does ...
Learning Shape Priors for Single-View 3D Completion And Reconstruction
Computer Vision – ECCV 2018AbstractThe problem of single-view 3D shape completion or reconstruction is challenging, because among the many possible shapes that explain an observation, most are implausible and do not correspond to natural objects. Recent research in the field has ...
Neural tangent kernel: convergence and generalization in neural networks (invited paper)
STOC 2021: Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of ComputingThe Neural Tangent Kernel is a new way to understand the gradient descent in deep neural networks, connecting them with kernel methods. In this talk, I'll introduce this formalism and give a number of results on the Neural Tangent Kernel and explain how ...
Comments