Skip to main content
Log in

Billion degree of freedom granular dynamics simulation on commodity hardware via heterogeneous data-type representation

  • Published:
Multibody System Dynamics Aims and scope Submit manuscript

Abstract

We discuss modeling, algorithmic, and software aspects that allow a simulation tool called Chrono::Granular to run billion-degree-of-freedom dynamics problems on commodity hardware, i.e., a workstation with one GPU. The ability to scale the solution to large problem sizes is traced back to an adimensionalization process combined with the use of mixed-precision data types that reduce memory pressure and improve arithmetic intensity, judicious use of the memory ecosystem on GPU cards as exposed by CUDA on Nvidia architectures, and a software implementation that prioritizes execution speed over modeling generality. The simulation approach is demonstrated for 3D scenarios with up to 710 million bodies for the frictionless case (of relevance in emulsions), and up to 210 million bodies for scenarios with friction (of relevance in terradynamics, additive manufacturing, soft-matter physics). The frictional contact model used draws on the Discrete Element Method (DEM). A performance benchmark shows linear scaling with problem size up to GPU memory capacity. The implementation has an application programming interface that enables it to interact in a cosimulation framework with third-party dynamics engines. This interaction is anchored by a force–displacement data exchange protocol that brings in external bodies as geometries defined by triangle meshes. We demonstrate the cosimulation mechanism by interfacing to an open source, multiphysics simulation engine called Chrono. Therein, triangular meshes define moving boundary conditions for Chrono::Granular, which in turn provides forces and torques acting on the triangular meshes. Several tests are considered for validation and scaling analysis purposes. The limiting aspects of the current implementation are its exclusive support of monodisperse granular systems, and its lack of handling geometries beyond spheres. These limitations are addressed by ongoing work.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Richard, P., Nicodemi, M., Delannay, R., Ribiere, P., Bideau, D.: Slow relaxation and compaction of granular systems. Nat. Mater. 4(2), 121–128 (2005)

    Google Scholar 

  2. Moreau, J.J.: Unilateral contact and dry friction in finite freedom dynamics. In: Moreau, J.J., Panagiotopoulos, P.D. (eds.) Nonsmooth Mechanics and Applications, pp. 1–82. Springer, Berlin (1988)

    Google Scholar 

  3. Cundall, P., Strack, O.: A discrete element model for granular assemblies. Geotechnique 29, 47–65 (1979)

    Google Scholar 

  4. Pazouki, A., Kwarta, M., Williams, K., Likos, W., Serban, R., Jayakumar, P., Negrut, D.: Compliant versus rigid contact: a comparison in the context of granular dynamics. Phys. Rev. E 96, 042905 (2017)

    Google Scholar 

  5. Goyon, J., Colin, A., Ovarlez, G., Ajdari, A., Bocquet, L.: Spatial cooperativity in soft glassy flows. Nature 454(7200), 84–87 (2008)

    Google Scholar 

  6. Jop, P., Forterre, Y., Pouliquen, O.: A constitutive law for dense granular flows. Nature 441(7094), 727–730 (2006)

    MATH  Google Scholar 

  7. Kamrin, K., Koval, G.: Nonlocal constitutive relation for steady granular flow. Phys. Rev. Lett. 108(17), 178301 (2012)

    Google Scholar 

  8. Iwashita, K., Oda, M.: Rolling resistance at contacts in simulation of shear band development by DEM. J. Eng. Mech. 124(3), 285–292 (1998)

    Google Scholar 

  9. Silbert, L., Ertaş, D., Grest, G., Halsey, T., Levine, D., Plimpton, S.: Granular flow down an inclined plane: bagnold scaling and rheology. Phys. Rev. E 64(5), 051302 (2001)

    Google Scholar 

  10. da Cruz, F., Emam, S., Prochnow, M., Roux, J.N., Chevoir, F.: Rheophysics of dense granular materials: discrete simulation of plane shear flows. Phys. Rev. E 72, 021309 (2005)

    Google Scholar 

  11. Parteli, E., Poschel, T.: Particle-based simulation of powder application in additive manufacturing. Powder Technol. 288, 96–102 (2016)

    Google Scholar 

  12. Furuichi, M., Nishiura, D., Asai, M., Hori, T.: Poster: the first real-scale DEM simulation of a sandbox experiment using 2.4 billion particles. In: Supercomputing Conference (2017)

    Google Scholar 

  13. Furuichi, M., Nishiura, D., Kuwano, O., Bauville, A., Hori, T., Sakaguchi, H.: Arcuate stress state in accretionary prisms from real-scale numerical sandbox experiments. Sci. Rep. 8, 12 (2018)

    Google Scholar 

  14. Nishiura, D., Sakaguchi, H., Yamamoto, S.: Multibillion particle DEM to simulate centrifuge model tests of geomaterials. In: Physical Modelling in Geotechnics, Volume 1: Proceedings of the 9th International Conference on Physical Modelling in Geotechnics (ICPMG 2018), London, United Kingdom, July 17–20, 2018, p. 227. CRC Press, Boca Raton (2018)

    Google Scholar 

  15. Strohmaier, E., Dongarra, J., Simon, H., Meuer, M.: TOP500 Supercomputer Site. http://www.top500.org

  16. Zhou, Y.C., Xu, B.H., Yu, A.B., Zulli, P.: An experimental and numerical study of the angle of repose of coarse spheres. Powder Technol. 125(1), 45–54 (2002)

    Google Scholar 

  17. Oda, M., Iwashita, K.: Study on couple stress and shear band development in granular media based on numerical simulation analyses. Int. J. Eng. Sci. 38(15), 1713–1740 (2000)

    Google Scholar 

  18. Ai, J., Chen, J.-F., Rotter, M., Ooi, J.: Assessment of rolling resistance models in discrete element simulations. Powder Technol. 206(3), 269–282 (2011)

    Google Scholar 

  19. Geer, S., Bernhardt-Barry, M., Garboczi, E., Whiting, J., Donmez, A.: A more efficient method for calibrating discrete element method parameters for simulations of metallic powder used in additive manufacturing. Granul. Matter 20(4), 77 (2018)

    Google Scholar 

  20. NVIDIA Corporation: Compute unified device architecture toolkit documentation (2019). https://docs.nvidia.com/cuda

  21. Negrut, D., Serban, R., Li, A., Seidl, A.: Unified Memory in CUDA 6.0: a brief overview of related data access and transfer issues. Technical Report TR-2014-09, Simulation-Based Engineering Laboratory, University of Wisconsin-Madison, (2014). https://sbel.wisc.edu/wp-content/uploads/sites/569/2018/05/TR-2014-09.pdf

  22. Fleischmann, J., Serban, R., Negrut, D., Jayakumar, P.: On the importance of displacement history in soft-body contact models. J. Comput. Nonlinear Dyn. 11(4), 044502 (2016)

    Google Scholar 

  23. Musin, O.R.: The kissing problem in three dimensions. arXiv Mathematics e-prints (2004). math/0410324

  24. Green, O.: Hashgraph – scalable hash tables using a sparse graph data structure (2019)

  25. Hockney, R., Eastwood, J.: Computer Simulation Using Particles. CRC Press, Boca Raton (1988)

    MATH  Google Scholar 

  26. Mazhar, H., Heyn, T., Negrut, D.: A scalable parallel method for large collision detection problems. Multibody Syst. Dyn. 26, 37–55 (2011). https://doi.org/10.1007/s11044-011-9246-y

    Article  MATH  Google Scholar 

  27. Hairer, E., Norsett, S., Wanner, G.: Solving Ordinary Differential Equations I: Nonstiff Problems. Springer, Berlin (2009)

    MATH  Google Scholar 

  28. Cundall, P.: Formulation of a three-dimensional distinct element model–Part I. A scheme to detect and represent contacts in a system composed of many polyhedral blocks. Int. J. Rock Mech. Min. Sci. Geomech. Abstr. 25(3), 107–116 (1988)

    Google Scholar 

  29. Chung, J., Lee, J.M.: A new family of explicit time integration methods for linear and non-linear structural dynamics. Int. J. Numer. Methods Eng. 37(23), 3961–3976 (1994)

    MathSciNet  MATH  Google Scholar 

  30. Schweizer, B., Li, P., Lu, D.: Explicit and implicit cosimulation methods: stability and convergence analysis for different solver coupling approaches. J. Comput. Nonlinear Dyn. 10(5), 051007 (2015)

    Google Scholar 

  31. Ericson, C.: Real Time Collision Detection. Morgan Kaufmann, San Francisco (2005)

    Google Scholar 

  32. Zhou, Z., Pinson, D., Zou, R., Yu, A.: Discrete particle simulation of gas fluidization of ellipsoidal particles. Chem. Eng. Sci. 66(23), 6128–6145 (2011)

    Google Scholar 

  33. Hou, Q., Zhou, Z., Yu, A.: Micromechanical modeling and analysis of different flow regimes in gas fluidization. Chem. Eng. Sci. 84, 449–468 (2012)

    Google Scholar 

  34. Gan, J., Zhou, Z., Yu, A.: A GPU-based DEM approach for modeling of particulate systems. Powder Technol. 301, 1172–1182 (2016)

    Google Scholar 

  35. University of Tennessee: High Performance Computing Challenge Benchmark (2019). http://icl.cs.utk.edu/hpcc/hpcc_results_lat_band.cgi

  36. Mankoc, C., Janda, A., Arevalo, R., Pastor, J., Zuriguel, I., Garcimartín, A., Maza, D.: The flow rate of granular materials through an orifice. Granul. Matter 9(6), 407–414 (2007)

    MATH  Google Scholar 

  37. Rakhsha, M., Kelly, C., Olsen, N., Serban, R., Negrut, D.: Multibody dynamics vs. fluid dynamics: two perspectives on the dynamics of granular flows. J. Comput. Nonlinear Dyn. (2020). https://doi.org/10.1115/1.4047237

    Article  Google Scholar 

  38. Cleary, P., Sawley, M.: DEM modelling of industrial granular flows: 3D case studies and the effect of particle shape on hopper discharge. Appl. Math. Model. 26(2), 89–111 (2002)

    MATH  Google Scholar 

  39. Bertrand, F., Leclaire, L., Levecque, G.: DEM-based models for the mixing of granular materials. Chem. Eng. Sci. 60(8–9), 2517–2531 (2005)

    Google Scholar 

  40. Fleissner, F., Eberhard, P.: Load balanced parallel simulation of particle-fluid DEM-SPH systems with moving boundaries. In: Parallel Computing: Architectures, Algorithms and Applications, vol. 48, pp. 37–44 (2007)

    Google Scholar 

  41. Kloss, C., Goniva, C., Hager, A., Amberger, S., Pirker, S.: Models, algorithms and validation for opensource DEM and CFD–DEM. Prog. Comput. Fluid Dyn. 12(2–3), 140–152 (2012)

    MathSciNet  Google Scholar 

  42. Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programming with the Message-Passing Interface, 2nd edn. MIT Press, Cambridge (1999)

    MATH  Google Scholar 

  43. Bouffard, J., Bertrand, F., Chaouki, J., Dumont, H.: Discrete element investigation of flow patterns and segregation in a spheronizer. Comput. Chem. Eng. 49, 170–182 (2013)

    Google Scholar 

  44. Alizadeh, E., Bertrand, F., Chaouki, J.: Comparison of DEM results and Lagrangian experimental data for the flow and mixing of granules in a rotating drum. AIChE J. 60(1), 60–75 (2014)

    Google Scholar 

  45. Hou, Q., Dong, K., Yu, A.: DEM study of the flow of cohesive particles in a screw feeder. Powder Technol. 256, 529–539 (2014)

    Google Scholar 

  46. He, Y., Evans, T., Yu, A., Yang, R.: A GPU-based DEM for modeling large scale powder compaction with wide size distributions. Powder Technol. 333, 219–228 (2018)

    Google Scholar 

  47. Toson, P., Siegmann, E., Trogrlic, M., Kureck, H., Khinast, J., Jajcevic, D., Doshi, P., Blackwood, D., Bonnassieux, A., Daugherity, P.D., et al.: Detailed modeling and process design of an advanced continuous powder mixer. Int. J. Pharm. 552(1–2), 288–300 (2018)

    Google Scholar 

  48. Govender, N., Wilke, D., Kok, S.: Blaze-DEMGPU: modular high performance DEM framework for the GPU architecture. SoftwareX 5, 62–66 (2016)

    Google Scholar 

  49. Longmore, J.-P., Marais, P., Kuttel, M.M.: Towards realistic and interactive sand simulation: a GPU-based framework. Powder Technol. 235, 983–1000 (2013)

    Google Scholar 

  50. Recuero, A.M., Serban, R., Peterson, B., Sugiyama, H., Jayakumar, P., Negrut, D.: A high-fidelity approach for vehicle mobility simulation: nonlinear finite element tires operating on granular material. J. Terramech. 72, 39–54 (2017)

    Google Scholar 

  51. Zhao, C.-L., Zang, M.-Y.: Application of the FEM/DEM and alternately moving road method to the simulation of tire-sand interactions. J. Terramech. 72, 27–38 (2017)

    Google Scholar 

  52. Johnson, J.B., Kulchitsky, A.V., Duvoy, P., Iagnemma, K., Senatore, C., Arvidson, R.E., Moore, J.: Discrete element method simulations of Mars exploration rover wheel performance. J. Terramech. 62, 31–40 (2015)

    Google Scholar 

  53. Dunatunga, S., Kamrin, K.: Continuum modelling and simulation of granular flows through their many phases. J. Fluid Mech. 779, 483 (2015)

    MathSciNet  MATH  Google Scholar 

  54. Tasora, A., Serban, R., Mazhar, H., Pazouki, A., Melanz, D., Fleischmann, J., Taylor, M., Sugiyama, H., Negrut, D.: Chrono: an open source multi-physics dynamics engine. In: Kozubek, T. (ed.) High Performance Computing in Science and Engineering. Lecture Notes in Computer Science, pp. 19–49. Springer, Berlin (2016)

    Google Scholar 

  55. Project Chrono: Chrono: an Open Source Framework for the Physics-Based Simulation of Dynamic Systems (2020). http://projectchrono.org. Accessed: 2020-03-03

Download references

Acknowledgements

The modeling/numerical method development associated with this project was funded through Army Research Office grant W911NF1910431. The hardware assets used herein have been available through Army Research Office grant W911NF1810476. The software development effort associated with this project was funded through National Science Foundation grant CISE—1835674.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dan Negrut.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kelly, C., Olsen, N. & Negrut, D. Billion degree of freedom granular dynamics simulation on commodity hardware via heterogeneous data-type representation. Multibody Syst Dyn 50, 355–379 (2020). https://doi.org/10.1007/s11044-020-09749-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11044-020-09749-7

Keywords

Navigation