Dynamic load balancing for predictions of storm surge and coastal flooding

https://doi.org/10.1016/j.envsoft.2021.105045Get rights and content

Highlights

  • Load balancing reduces prediction time for coastal flooding by up to 45%.

  • The floodplain portion of the mesh is dynamically removed from calculations.

  • Performance depends significantly on selected rebalancing heuristics.

Abstract

As coastal circulation models have evolved to predict storm-induced flooding, they must include progressively more overland regions that are normally dry, to where now it is possible for more than half of the domain to be needed in none or only some of the computations. While this evolution has improved real-time forecasting and long-term mitigation of coastal flooding, it poses a problem for parallelization in an HPC environment, especially for static paradigms in which the workload is balanced only at the start of the simulation. In this study, a dynamic rebalancing of computational work is developed for a finite-element-based, shallow-water, ocean circulation model of extensive overland flooding. The implementation has a low overhead cost, and we demonstrate a realistic hurricane-forced coastal flooding simulation can achieve peak speed-ups near 45% over the static case, thus operating now at 80−90% efficiency.

Section snippets

Software availability

The code, instructions, and a test to run the software detailed in this work can be found by this private link: https://figshare.com/s/41827afadc318047e2ea. The ADvanced Hydrodynamic CIRCulation (ADCIRC) code was originally developed in FORTRAN77 by Joannes Westerink, Rick Luettich, and Clint Dawson. Since then it has been continuously updated using FORTRAN90 and FORTRAN77 and maintained by a community of developers. ADCIRC is a commercial code (www.adcirc.org) that is otherwise free for

ADCIRC hydrodynamic model

The dynamic load balancing application is built around the ADvanced hydrodynamic CIRculation model (Luettich and Westerink, 2004) to improve the computational performance of both existing and future modeling systems that rely on ADCIRC. ADCIRC (http://adcirc.org) has become one of the most widely used community modeling platforms for storm surge/coastal flooding predictions across academia, United States governmental agencies and the private sector. This is due to its inclusion of critical

Results

Simulations were executed on the Computational Hydraulics Laboratory's computer cluster called Aegaeon (https://coast.nd.edu/). Aegaeon contains 83 compute nodes of dual 12 core E5-2680, 2.50 GHz Haswell processors (1, 992 cores in total). Each node contains 64 GB of random access memory that is shared among each node's 24 cores. The nodes are connected via a high-speed 56 GB Infiniband network. File input/output was disabled (i.e., no logging and no output file writing) however, for the

Discussion and conclusion

The aim of this work is to reduce the wall-clock times spent modeling wind-driven coastal flooding on unstructured triangular meshes using the ADCIRC solver. Regional coastal ADCIRC meshes often contain relatively large amounts of dry-state vertices to represent the finely-detailed nature of the coastal floodplains. Considering variable resolution unstructured mesh/model development is both challenging and time-consuming, modelers cannot design a stable and robust modeling/mesh system that is

Funding

This material is based upon work supported by the U.S. Department of Homeland Security under Grant Award Number 2015-ST-061-ND0001-01 and National Science Foundation Grant Award Number NSF ACI-1339738. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security or the National Science Foundation.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

We thank the two anonymous reviewers who helped improve the quality of the manuscript. We thank Ocean Weather Inc. for allowing us to use their meteorological forcing inputs for the Hurricane Irene test problem. We thank Dr. Brian Blanton at Renaissance Computing Institute at the University of North Carolina at Chapel Hill for providing the mesh and input files used in the Hurricane Irene test problem. KR prepared the manuscript, designed and implemented the coding upgrades into ADCIRC,

References (60)

  • G. Karypis et al.

    A parallel algorithm for multilevel graph partitioning and sparse matrix ordering

    J. Parallel Distr. Comput.

    (Jan. 1998)
  • A.B. Kennedy et al.

    Subgrid theory for storm surge modeling

    Ocean Model.

    (Dec. 2019)
  • A. Khalid et al.

    Advancing real-time flood prediction in large estuaries: iflood a fully coupled surge-wave automated web-based guidance system

    Environ. Model. Software

    (Sep. 2020)
  • T. Kärnä et al.

    A fully implicit wetting–drying method for dg-fem shallow water models, with an application to the scheldt estuary

    Comput. Methods Appl. Mech. Eng.

    (2011)
  • D.R. Lynch et al.

    A wave equation model for finite element tidal computations

    Comput. Fluids

    (Sep. 1979)
  • D. Marks et al.

    A variable resolution right TIN approach for gridded oceanographic data

    Comput. Geosci-uk.

    (Dec. 2017)
  • O. Quetzalcóatl et al.

    SMC, a coastal modeling system for assessing beach processes and coastal interventions: application to the Brazilian coast

    Environ. Model. Software

    (Jun. 2019)
  • K.J. Roberts et al.

    On the automatic and a priori design of unstructured mesh resolution for coastal ocean circulation models

    Ocean Model.

    (2019)
  • B.F. Sanders et al.

    Parbrezo: a parallel, unstructured grid, godunov-type, shallow-water code for high-resolution flood inundation modeling at the regional scale

    Adv. Water Resour.

    (2010)
  • A. Sebastian et al.

    Characterizing hurricane storm surge behavior in Galveston bay using the SWAN+ADCIRC model

    Coast. Eng.

    (Jun. 2014)
  • P. Taeb et al.

    An operational coastal forecasting tool for performing ensemble modeling

    Estuar. Coast Shelf Sci.

    (Feb. 2019)
  • J. Teng et al.

    Flood inundation modelling: a review of methods, recent advances and uncertainty analysis

    Environ. Model. Software

    (Apr. 2017)
  • J.C. Warner et al.

    A wetting and drying scheme for ROMS

    Comput. Geosci-uk

    (Aug. 2013)
  • R. Wittmann et al.

    High performance shallow water kernels for parallel overland flow simulations based on fullswof2d

    Comput. Math. Appl.

    (2017)
  • A. Androsov et al.

    FESOM-c v.2: coastal dynamics on hybrid unstructured meshes

    Geosci. Model Dev. (GMD)

    (Mar. 2019)
  • P.D. Bates et al.

    A new method for moving–boundary hydrodynamic problems in shallow water

    Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences

    (1999)
  • J. Behrens et al.

    Efficiency considerations in triangular adaptive mesh refinement

    Phil. Trans. R. Soc. A.

    (Nov. 2009)
  • B. Blanton et al.

    An integrated scenario Ensemble-Based framework for hurricane evacuation modeling: Part 2—hazard modeling

    Risk Anal.

    (Apr. 2018)
  • B. Blanton et al.

    Urgent computing of storm surge for North Carolina's coast

  • E.G. Boman et al.

    The zoltan and isorropia parallel toolkits for combinatorial scientific computing: partitioning, ordering and coloring

    Sci. Programming-neth.

    (2012)
  • Cited by (5)

    • Improving coastal flooding predictions by switching meshes during a simulation

      2021, Ocean Modelling
      Citation Excerpt :

      These meshes can be large in their spatial coverage, with millions of finite elements to represent one or multiple state coastlines (Thomas et al., 2019), and in their cost, requiring several hours even on thousands of computational cores (Dietrich et al., 2012). A significant portion of the cost is due to the inclusion of floodplains and other sub-aerial regions, which remain dry for most of the simulation and become wet only as the storm makes landfall, but which are represented typically by 50 to 90 percent of the total number of finite elements (Roberts et al., 2021). Although ADCIRC meshes can be designed to reduce the number of elements in these floodplains (Bilskie et al., 2020), there is a need to adapt during the simulation to include the floodplains only as they are wetted.

    • Runtime Techniques for Automatic Process Virtualization

      2022, ACM International Conference Proceeding Series
    • Effects of Model Resolution and Coverage on Storm-Driven Coastal Flooding Predictions

      2022, Journal of Waterway, Port, Coastal and Ocean Engineering
    1

    Present address: Escola Politécnica, Dept. Mechanical Engineering, Av. Professor Mello Moraes, 2231, Cidade Universitária, São Paulo - SP, Brazil.

    View full text