Skip to main content
SpringerLink
Log in

A scalable parallel algorithm for direct-forcing immersed boundary method for multiphase flow simulation on spectral elements

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In this work, we propose a highly scalable parallel double binned ghost particle (DBGP) algorithm for direct-forcing immersed boundary spectral element method for multiphase flow simulations. In particular, the DBGP algorithm is designed to obtain fully distributed data storage and scalable data transfer across hundreds of thousands of processors. The proposed algorithm uses a queen and worker data structure for fully resolved particles to demarcate particle-level and marker-level quantities and communication. In the DBGP algorithm, each particle’s centroid is represented by a queen marker and the particle surface is covered with a uniform distribution of surface worker markers. The queen marker contains information on the translational and rotational motion of a particle and integrates the force and torque computed at all the worker markers, while the worker marker implements the fluid–particle interaction. Ghost queen and ghost worker markers are generated for each real queen and real worker marker during computation for particle-level and marker-level communications, respectively. A double Cartesian binning process is introduced that divides the physical domain into a coarse queen-bin and a fine worker-bin structure in three dimensions. The queen-bin and worker-bin sizes are determined by their zone of influence at the particle-level and marker-level communication, respectively. Bin-to-rank maps that relate each queen-bin and worker-bin to all the MPI ranks that they interact with are created. By using the queen/worker marker representation and two-layer bin-to-rank maps, data communication across very large number of MPI ranks is efficiently carried out. A scaling analysis has been conducted, showing excellent performance of the DBGP algorithm for up to 16,384 MPI ranks in both weak and strong scaling studies. The proposed method has been demonstrated to accurately predict sedimentation of particle clouds. The simulated correlation between the mean settling velocity and volume fraction is in good agreement with empirical correlations from previous studies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Akiki G, Balachandar S (2016) Immersed boundary method with non-uniform distribution of Lagrangian markers for a non-uniform Eulerian mesh. J Comput Phys 307:34–59

    Article  MathSciNet  Google Scholar 

  2. Akiki G, Moore W, Balachandar S (2017) Pairwise-interaction extended point-particle model for particle-laden flows. J Comput Phys 351:329–357

    Article  MathSciNet  Google Scholar 

  3. Bagchi P, Balachandar S (2002) Shear versus vortex-induced lift force on a rigid sphere at moderate Re. J Fluid Mech 473:379–388

    Article  MathSciNet  Google Scholar 

  4. Balachandar S, Eaton JK (2010) Turbulent dispersed multiphase flow. Annu Rev Fluid Mech 42:111–133

    Article  Google Scholar 

  5. Berrut JP, Trefethen LN (2004) Barycentric Lagrange interpolation. SIAM Rev 46(3):501–517

    Article  MathSciNet  Google Scholar 

  6. Breugem WP (2012) A second-order accurate immersed boundary method for fully resolved simulations of particle-laden flows. J Comput Phys 231(13):4469–4498

    Article  MathSciNet  Google Scholar 

  7. Capecelatro J, Desjardins O (2013) An Euler–Lagrange strategy for simulating particle-laden flows. J Comput Phys 238:1–31

    Article  MathSciNet  Google Scholar 

  8. CEED team (2020) GSLIB Sparse communication library. https://ceed.exascaleproject.org/gslib/. Accessed 25 May 2020

  9. Cundall PA, Strack OD (1979) A discrete numerical model for granular assemblies. Geotechnique 29(1):47–65

    Article  Google Scholar 

  10. Deville MO, Fischer PF, Mund EH (2002) High-order methods for incompressible fluid flow, vol 9. Cambridge University Press, Cambridge

    Book  Google Scholar 

  11. Fischer P, Kruse J, Mullen J, Tufo H, Lottes J, Kerkemeier S (2008) Nek5000: Open source spectral element CFD solver. Argonne National Laboratory, Mathematics and Computer Science Division, Argonne, see https://nek5000.mcs.anl.gov/. Accessed 25 May 2020

  12. Fornberg B (1980) A numerical study of steady viscous flow past a circular cylinder. J Fluid Mech 98(4):819–855

    Article  Google Scholar 

  13. Glowinski R, Pan TW, Hesla TI, Joseph DD (1999) A distributed Lagrange multiplier/fictitious domain method for particulate flows. Int J Multiph Flow 25(5):755–794

    Article  MathSciNet  Google Scholar 

  14. Gottlieb S, Shu CW (1998) Total variation diminishing Runge–Kutta schemes. Math Comput Am Math Soc 67(221):73–85

    Article  MathSciNet  Google Scholar 

  15. He L, Tafti DK, Nagendra K (2017) Evaluation of drag correlations using particle resolved simulations of spheres and ellipsoids in assembly. Powder Technol 313:332–343

    Article  Google Scholar 

  16. Johnson T, Patel V (1999) Flow past a sphere up to a Reynolds number of 300. J Fluid Mech 378:19–70

    Article  Google Scholar 

  17. Kempe T, Fröhlich J (2012) An improved immersed boundary method with direct forcing for the simulation of particle laden flows. J Comput Phys 231(9):3663–3684

    Article  MathSciNet  Google Scholar 

  18. Mazzuoli M, Blondeaux P, Vittori G, Uhlmann M, Simeonov J, Calantoni J (2020) Interface-resolved direct numerical simulations of sediment transport in a turbulent oscillatory boundary layer. J Fluid Mech 885:A28

    Article  Google Scholar 

  19. Mohd-Yusof J (1997) Combined immersed-boundary/b-spline methods for simulations of flow in complex geometries. Annual Research Briefs. NASA Ames Research Center-Stanford University Center of Turbulence Research: Stanford pp 317–327

  20. Mordant N, Pinton JF (2000) Velocity measurement of a settling sphere. Eur Phys J B Condens Matter Complex Syst 18(2):343–352

    Article  Google Scholar 

  21. Nagendra K, Tafti DK, Viswanath K (2014) A new approach for conjugate heat transfer problems using immersed boundary method for curvilinear grid based solvers. J Comput Phys 267:225–246

    Article  MathSciNet  Google Scholar 

  22. Navarro HA, de Souza Braun MP (2013) Determination of the normal spring stiffness coefficient in the linear spring-dashpot contact model of discrete element method. Powder Technol 246:707–722

    Article  Google Scholar 

  23. Offermans N, Marin O, Schanen M, Gong J, Fischer P, Schlatter P, Obabko A, Peplinski A, Hutchinson M, Merzari E (2016) On the strong scaling of the spectral element solver nek5000 on petascale systems. In: Proceedings of the Exascale Applications and Software Conference 2016. ACM, p 5

  24. Patera AT (1984) A spectral element method for fluid dynamics: laminar flow in a channel expansion. J Comput Phys 54(3):468–488

    Article  Google Scholar 

  25. Peskin CS (2002) The immersed boundary method. Acta Numer 11:479–517

    Article  MathSciNet  Google Scholar 

  26. Pinelli A, Naqavi I, Piomelli U, Favier J (2010) Immersed-boundary methods for general finite-difference and finite-volume Navier–Stokes solvers. J Comput Phys 229(24):9073–9091

    Article  MathSciNet  Google Scholar 

  27. Richardson J, Zaki W (1954) This week’s citation classic. Trans Inst Chem Eng 32:35–53

    Google Scholar 

  28. Roma AM, Peskin CS, Berger MJ (1999) An adaptive version of the immersed boundary method. J Comput Phys 153(2):509–534

    Article  MathSciNet  Google Scholar 

  29. Saff EB, Kuijlaars AB (1997) Distributing many points on a sphere. Math Intell 19(1):5–11

    Article  MathSciNet  Google Scholar 

  30. Schiller L, Naumann A (1933) Über die grundlegenden berechnungen bei der schwerkraftaufbereitung. Z Ver Dtsch Ing 77:318–320

    Google Scholar 

  31. Sharma N, Patankar NA (2005) A fast computation technique for the direct numerical simulation of rigid particulate flows. J Comput Phys 205(2):439–457

    Article  Google Scholar 

  32. Taneda S (1956) Experimental investigation of the wake behind a sphere at low Reynolds numbers. J Phys Soc Jpn 11(10):1104–1108

    Article  Google Scholar 

  33. Tenneti S, Garg R, Subramaniam S (2011) Drag law for monodisperse gas–solid systems using particle-resolved direct numerical simulation of flow past fixed assemblies of spheres. Int J Multiph Flow 37(9):1072–1092

    Article  Google Scholar 

  34. Tenneti S, Subramaniam S (2014) Particle-resolved direct numerical simulation for gas-solid flow model development. Annu Rev Fluid Mech 46:199–230

    Article  MathSciNet  Google Scholar 

  35. Tufo HM, Fischer PF (2001) Fast parallel direct solvers for coarse grid problems. J Parallel Distrib Comput 61(2):151–177

    Article  Google Scholar 

  36. Uhlmann M (2004) Simulation of particulate flows on multi-processor machines with distributed memory, Technical report. Centro de Investigaciones Energeticas Medioambientales y Tecnologicas (CIEMAT)

  37. Uhlmann M (2005) An immersed boundary method with direct forcing for the simulation of particulate flows. J Comput Phys 209(2):448–476

    Article  MathSciNet  Google Scholar 

  38. van der Hoef MA, Ye M, van Sint Annaland M, Andrews A, Sundaresan S, Kuipers J (2006) Multiscale modeling of gas-fluidized beds. Adv Chem Eng 31:65–149

    Article  Google Scholar 

  39. Valero-Lara P (2014) Accelerating solid–fluid interaction based on the immersed boundary method on multicore and GPU architectures. J Supercomput 70(2):799–815

    Article  Google Scholar 

  40. Wang S, He G, Zhang X (2013) Parallel computing strategy for a flow solver based on immersed boundary method and discrete stream-function formulation. Comput Fluids 88:210–224

    Article  MathSciNet  Google Scholar 

  41. Wang Z, Fan J, Luo K (2008) Parallel computing strategy for the simulation of particulate flows with immersed boundary method. Sci China Ser E Technol Sci 51(8):1169–1176

    Article  MathSciNet  Google Scholar 

  42. Yin X, Koch DL (2007) Hindered settling velocity and microstructure in suspensions of solid spheres with moderate reynolds numbers. Phys Fluids 19(9):093302

    Article  Google Scholar 

  43. Yu Z, Lin Z, Shao X, Wang LP (2016) A parallel fictitious domain method for the interface-resolved simulation of particle-laden flows and its application to the turbulent channel flow. Eng Appl Comput Fluid Mech 10(1):160–170

    Google Scholar 

  44. Zwick D, Balachandar S (2019) A scalable Euler–Lagrange approach for multiphase flow simulation on spectral elements. Int J High Perform Comput Appl. https://doi.org/10.1177/1094342019867756

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by the US Department of Energy, National Nuclear Security Administration, Advanced Simulation and Computing Program, as a Cooperative Agreement under the Predictive Science Academic Alliance Program, under Contract No. DENA0002378. YY would like to thank David Zwick and Jorge Salinas for their insightful discussion and help on the code development.

Author information

Authors and Affiliations

  1. Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL, USA

    Yunchao Yang & S. Balachandar

Authors
  1. Yunchao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S. Balachandar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yunchao Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, Y., Balachandar, S. A scalable parallel algorithm for direct-forcing immersed boundary method for multiphase flow simulation on spectral elements. J Supercomput 77, 2897–2927 (2021). https://doi.org/10.1007/s11227-020-03371-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-020-03371-2

Keywords

Search

Navigation