Abstract
We discuss modeling, algorithmic, and software aspects that allow a simulation tool called Chrono::Granular to run billion-degree-of-freedom dynamics problems on commodity hardware, i.e., a workstation with one GPU. The ability to scale the solution to large problem sizes is traced back to an adimensionalization process combined with the use of mixed-precision data types that reduce memory pressure and improve arithmetic intensity, judicious use of the memory ecosystem on GPU cards as exposed by CUDA on Nvidia architectures, and a software implementation that prioritizes execution speed over modeling generality. The simulation approach is demonstrated for 3D scenarios with up to 710 million bodies for the frictionless case (of relevance in emulsions), and up to 210 million bodies for scenarios with friction (of relevance in terradynamics, additive manufacturing, soft-matter physics). The frictional contact model used draws on the Discrete Element Method (DEM). A performance benchmark shows linear scaling with problem size up to GPU memory capacity. The implementation has an application programming interface that enables it to interact in a cosimulation framework with third-party dynamics engines. This interaction is anchored by a force–displacement data exchange protocol that brings in external bodies as geometries defined by triangle meshes. We demonstrate the cosimulation mechanism by interfacing to an open source, multiphysics simulation engine called Chrono. Therein, triangular meshes define moving boundary conditions for Chrono::Granular, which in turn provides forces and torques acting on the triangular meshes. Several tests are considered for validation and scaling analysis purposes. The limiting aspects of the current implementation are its exclusive support of monodisperse granular systems, and its lack of handling geometries beyond spheres. These limitations are addressed by ongoing work.
Similar content being viewed by others
References
Richard, P., Nicodemi, M., Delannay, R., Ribiere, P., Bideau, D.: Slow relaxation and compaction of granular systems. Nat. Mater. 4(2), 121–128 (2005)
Moreau, J.J.: Unilateral contact and dry friction in finite freedom dynamics. In: Moreau, J.J., Panagiotopoulos, P.D. (eds.) Nonsmooth Mechanics and Applications, pp. 1–82. Springer, Berlin (1988)
Cundall, P., Strack, O.: A discrete element model for granular assemblies. Geotechnique 29, 47–65 (1979)
Pazouki, A., Kwarta, M., Williams, K., Likos, W., Serban, R., Jayakumar, P., Negrut, D.: Compliant versus rigid contact: a comparison in the context of granular dynamics. Phys. Rev. E 96, 042905 (2017)
Goyon, J., Colin, A., Ovarlez, G., Ajdari, A., Bocquet, L.: Spatial cooperativity in soft glassy flows. Nature 454(7200), 84–87 (2008)
Jop, P., Forterre, Y., Pouliquen, O.: A constitutive law for dense granular flows. Nature 441(7094), 727–730 (2006)
Kamrin, K., Koval, G.: Nonlocal constitutive relation for steady granular flow. Phys. Rev. Lett. 108(17), 178301 (2012)
Iwashita, K., Oda, M.: Rolling resistance at contacts in simulation of shear band development by DEM. J. Eng. Mech. 124(3), 285–292 (1998)
Silbert, L., Ertaş, D., Grest, G., Halsey, T., Levine, D., Plimpton, S.: Granular flow down an inclined plane: bagnold scaling and rheology. Phys. Rev. E 64(5), 051302 (2001)
da Cruz, F., Emam, S., Prochnow, M., Roux, J.N., Chevoir, F.: Rheophysics of dense granular materials: discrete simulation of plane shear flows. Phys. Rev. E 72, 021309 (2005)
Parteli, E., Poschel, T.: Particle-based simulation of powder application in additive manufacturing. Powder Technol. 288, 96–102 (2016)
Furuichi, M., Nishiura, D., Asai, M., Hori, T.: Poster: the first real-scale DEM simulation of a sandbox experiment using 2.4 billion particles. In: Supercomputing Conference (2017)
Furuichi, M., Nishiura, D., Kuwano, O., Bauville, A., Hori, T., Sakaguchi, H.: Arcuate stress state in accretionary prisms from real-scale numerical sandbox experiments. Sci. Rep. 8, 12 (2018)
Nishiura, D., Sakaguchi, H., Yamamoto, S.: Multibillion particle DEM to simulate centrifuge model tests of geomaterials. In: Physical Modelling in Geotechnics, Volume 1: Proceedings of the 9th International Conference on Physical Modelling in Geotechnics (ICPMG 2018), London, United Kingdom, July 17–20, 2018, p. 227. CRC Press, Boca Raton (2018)
Strohmaier, E., Dongarra, J., Simon, H., Meuer, M.: TOP500 Supercomputer Site. http://www.top500.org
Zhou, Y.C., Xu, B.H., Yu, A.B., Zulli, P.: An experimental and numerical study of the angle of repose of coarse spheres. Powder Technol. 125(1), 45–54 (2002)
Oda, M., Iwashita, K.: Study on couple stress and shear band development in granular media based on numerical simulation analyses. Int. J. Eng. Sci. 38(15), 1713–1740 (2000)
Ai, J., Chen, J.-F., Rotter, M., Ooi, J.: Assessment of rolling resistance models in discrete element simulations. Powder Technol. 206(3), 269–282 (2011)
Geer, S., Bernhardt-Barry, M., Garboczi, E., Whiting, J., Donmez, A.: A more efficient method for calibrating discrete element method parameters for simulations of metallic powder used in additive manufacturing. Granul. Matter 20(4), 77 (2018)
NVIDIA Corporation: Compute unified device architecture toolkit documentation (2019). https://docs.nvidia.com/cuda
Negrut, D., Serban, R., Li, A., Seidl, A.: Unified Memory in CUDA 6.0: a brief overview of related data access and transfer issues. Technical Report TR-2014-09, Simulation-Based Engineering Laboratory, University of Wisconsin-Madison, (2014). https://sbel.wisc.edu/wp-content/uploads/sites/569/2018/05/TR-2014-09.pdf
Fleischmann, J., Serban, R., Negrut, D., Jayakumar, P.: On the importance of displacement history in soft-body contact models. J. Comput. Nonlinear Dyn. 11(4), 044502 (2016)
Musin, O.R.: The kissing problem in three dimensions. arXiv Mathematics e-prints (2004). math/0410324
Green, O.: Hashgraph – scalable hash tables using a sparse graph data structure (2019)
Hockney, R., Eastwood, J.: Computer Simulation Using Particles. CRC Press, Boca Raton (1988)
Mazhar, H., Heyn, T., Negrut, D.: A scalable parallel method for large collision detection problems. Multibody Syst. Dyn. 26, 37–55 (2011). https://doi.org/10.1007/s11044-011-9246-y
Hairer, E., Norsett, S., Wanner, G.: Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, Berlin (2009)
Cundall, P.: Formulation of a three-dimensional distinct element model–Part I. A scheme to detect and represent contacts in a system composed of many polyhedral blocks. Int. J. Rock Mech. Min. Sci. Geomech. Abstr. 25(3), 107–116 (1988)
Chung, J., Lee, J.M.: A new family of explicit time integration methods for linear and non-linear structural dynamics. Int. J. Numer. Methods Eng. 37(23), 3961–3976 (1994)
Schweizer, B., Li, P., Lu, D.: Explicit and implicit cosimulation methods: stability and convergence analysis for different solver coupling approaches. J. Comput. Nonlinear Dyn. 10(5), 051007 (2015)
Ericson, C.: Real Time Collision Detection. Morgan Kaufmann, San Francisco (2005)
Zhou, Z., Pinson, D., Zou, R., Yu, A.: Discrete particle simulation of gas fluidization of ellipsoidal particles. Chem. Eng. Sci. 66(23), 6128–6145 (2011)
Hou, Q., Zhou, Z., Yu, A.: Micromechanical modeling and analysis of different flow regimes in gas fluidization. Chem. Eng. Sci. 84, 449–468 (2012)
Gan, J., Zhou, Z., Yu, A.: A GPU-based DEM approach for modeling of particulate systems. Powder Technol. 301, 1172–1182 (2016)
University of Tennessee: High Performance Computing Challenge Benchmark (2019). http://icl.cs.utk.edu/hpcc/hpcc_results_lat_band.cgi
Mankoc, C., Janda, A., Arevalo, R., Pastor, J., Zuriguel, I., Garcimartín, A., Maza, D.: The flow rate of granular materials through an orifice. Granul. Matter 9(6), 407–414 (2007)
Rakhsha, M., Kelly, C., Olsen, N., Serban, R., Negrut, D.: Multibody dynamics vs. fluid dynamics: two perspectives on the dynamics of granular flows. J. Comput. Nonlinear Dyn. (2020). https://doi.org/10.1115/1.4047237
Cleary, P., Sawley, M.: DEM modelling of industrial granular flows: 3D case studies and the effect of particle shape on hopper discharge. Appl. Math. Model. 26(2), 89–111 (2002)
Bertrand, F., Leclaire, L., Levecque, G.: DEM-based models for the mixing of granular materials. Chem. Eng. Sci. 60(8–9), 2517–2531 (2005)
Fleissner, F., Eberhard, P.: Load balanced parallel simulation of particle-fluid DEM-SPH systems with moving boundaries. In: Parallel Computing: Architectures, Algorithms and Applications, vol. 48, pp. 37–44 (2007)
Kloss, C., Goniva, C., Hager, A., Amberger, S., Pirker, S.: Models, algorithms and validation for opensource DEM and CFD–DEM. Prog. Comput. Fluid Dyn. 12(2–3), 140–152 (2012)
Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programming with the Message-Passing Interface, 2nd edn. MIT Press, Cambridge (1999)
Bouffard, J., Bertrand, F., Chaouki, J., Dumont, H.: Discrete element investigation of flow patterns and segregation in a spheronizer. Comput. Chem. Eng. 49, 170–182 (2013)
Alizadeh, E., Bertrand, F., Chaouki, J.: Comparison of DEM results and Lagrangian experimental data for the flow and mixing of granules in a rotating drum. AIChE J. 60(1), 60–75 (2014)
Hou, Q., Dong, K., Yu, A.: DEM study of the flow of cohesive particles in a screw feeder. Powder Technol. 256, 529–539 (2014)
He, Y., Evans, T., Yu, A., Yang, R.: A GPU-based DEM for modeling large scale powder compaction with wide size distributions. Powder Technol. 333, 219–228 (2018)
Toson, P., Siegmann, E., Trogrlic, M., Kureck, H., Khinast, J., Jajcevic, D., Doshi, P., Blackwood, D., Bonnassieux, A., Daugherity, P.D., et al.: Detailed modeling and process design of an advanced continuous powder mixer. Int. J. Pharm. 552(1–2), 288–300 (2018)
Govender, N., Wilke, D., Kok, S.: Blaze-DEMGPU: modular high performance DEM framework for the GPU architecture. SoftwareX 5, 62–66 (2016)
Longmore, J.-P., Marais, P., Kuttel, M.M.: Towards realistic and interactive sand simulation: a GPU-based framework. Powder Technol. 235, 983–1000 (2013)
Recuero, A.M., Serban, R., Peterson, B., Sugiyama, H., Jayakumar, P., Negrut, D.: A high-fidelity approach for vehicle mobility simulation: nonlinear finite element tires operating on granular material. J. Terramech. 72, 39–54 (2017)
Zhao, C.-L., Zang, M.-Y.: Application of the FEM/DEM and alternately moving road method to the simulation of tire-sand interactions. J. Terramech. 72, 27–38 (2017)
Johnson, J.B., Kulchitsky, A.V., Duvoy, P., Iagnemma, K., Senatore, C., Arvidson, R.E., Moore, J.: Discrete element method simulations of Mars exploration rover wheel performance. J. Terramech. 62, 31–40 (2015)
Dunatunga, S., Kamrin, K.: Continuum modelling and simulation of granular flows through their many phases. J. Fluid Mech. 779, 483 (2015)
Tasora, A., Serban, R., Mazhar, H., Pazouki, A., Melanz, D., Fleischmann, J., Taylor, M., Sugiyama, H., Negrut, D.: Chrono: an open source multi-physics dynamics engine. In: Kozubek, T. (ed.) High Performance Computing in Science and Engineering. Lecture Notes in Computer Science, pp. 19–49. Springer, Berlin (2016)
Project Chrono: Chrono: an Open Source Framework for the Physics-Based Simulation of Dynamic Systems (2020). http://projectchrono.org. Accessed: 2020-03-03
Acknowledgements
The modeling/numerical method development associated with this project was funded through Army Research Office grant W911NF1910431. The hardware assets used herein have been available through Army Research Office grant W911NF1810476. The software development effort associated with this project was funded through National Science Foundation grant CISE—1835674.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Kelly, C., Olsen, N. & Negrut, D. Billion degree of freedom granular dynamics simulation on commodity hardware via heterogeneous data-type representation. Multibody Syst Dyn 50, 355–379 (2020). https://doi.org/10.1007/s11044-020-09749-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11044-020-09749-7