Skip to main content
SpringerLink
Log in

User-level failure detection and auto-recovery of parallel programs in HPC systems

  • Research Article
  • Published:
Frontiers of Computer Science Aims and scope Submit manuscript

Abstract

As the mean-time-between-failures (MTBF) continues to decline with the increasing number of components on large-scale high performance computing (HPC) systems, program failures might occur during the execution period with high probability. Ensuring successful execution of the HPC programs has become an issue that the unprivileged users should be concerned. From the user perspective, if the program failure cannot be detected and handled in time, it would waste resources and delay the progress of program execution. Unfortunately, the unprivileged users are unable to perform program state checking due to execution control by the job management system as well as the limited privilege. Currently, automated tools for supporting user-level failure detection and auto-recovery of parallel programs in HPC systems are missing. This paper proposes an innovative method for the unprivileged user to achieve failure detection of job execution and automatic resubmission of failed jobs. The state checker in our method is encapsulated as an independent job to reduce interference with the user jobs. In addition, we propose a dual-checker mechanism to improve the robustness of our approach. We implement the proposed method as a tool named automatic re-launcher (ARL) and evaluate it on the Tianhe-2 system. Experiment results show that ARL can detect the execution failures effectively on Tianhe-2 system. In addition, the communication and performance overhead caused by ARL is negligible. The good scalability of ARL makes it applicable for large-scale HPC systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Egwutuoha I P, Levy D, Selic B, Chen S. A survey of fault tolerance mechanisms and checkpoint/restart implementations for high performance computing systems. The Journal of Supercomputing, 2013, 65: 1302–1326

    Article  Google Scholar 

  2. Lu C D. Failure data analysis of hpc systems. 2013, arXiv preprint arXiv:1302.4779

  3. Cappello F, Geist A, Gropp W D, Kale L V, Kramer W T, Snir M. Toward exascale resilience. International Journal of High Performance Computing Applications, 2009, 23: 374–385

    Article  Google Scholar 

  4. Bertier M, Marin O, Sens P. Performance analysis of a hierarchical failure detector. In: Proceedings of the 2003 International Conference on Dependable Systems and Networks. 2003, 635–644

  5. Luecke G R, Zou Y, Coyle J, Hoekstra J, Kraeva M. Deadlock detection in MPI programs. Concurrency and Computation: Practice and Experience, 2002, 14: 911–932

    Article  MATH  Google Scholar 

  6. Gao Q, Qin F, Panda D K. DMTracker: finding bugs in large-scale parallel programs by detecting anomaly in data movements. In: Proceedings of the 2007 ACM/IEEE Conference on Supercomputing. 2007, 1–12

  7. Arnold D C, Ahn D H, De Supinski B R, Lee G L, Miller B P, Schulz M. Stack trace analysis for large-scale debugging. In: Proceddings of the 2007 IEEE International Parallel and Distributed Processing Symposium. 2007, 1–10

  8. Laguna I, Gamblin T, De Supinski B R, Bagchi S, Bronevetsky G, Anh D H, Schulz M, Rountree B. Large scale debugging of parallel tasks with AutomaDeD. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. 2011, 1–10

  9. Wu X, Mueller F. Elastic and scalable tracing and accurate replay of non-deterministic events. In: Proceedings of the 27th International ACM Conference on International Conference on Supercomputing. 2013, 59–68

  10. Gupta R, Beckman P, Park B H, Lusk E, Hargrove P, Geist A, Panda D, Lumsdaine A, Dongarra J. CIFTS: a coordinated infrastructure for fault-tolerant systems. In: Proceedings of the 2009 International Conference on Parallel Processing. 2009, 237–245

  11. Zhang G Z, Liu Y, Yang H L, Qian D P. A lightweight and flexible tool for distinguishing between hardware malfunctions and program bugs in debugging large-scale programs. IEEE Access, 2018, 6: 71892–71905

    Article  Google Scholar 

  12. Bosilca G, Bouteiller A, Guermouche A, Herault T, Robert Y, Sens P, Dongarra J. A failure detector for HPC platforms. The International Journal of High Performance Computing Applications, 2018, 32: 139–158

    Article  Google Scholar 

  13. Berrocal E, Bautista-Gomez L, Di S, Lan Z L, Cappello F. Lightweight silent data corruption detection based on runtime data analysis for HPC applications. In: Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing. 2015, 275–278

  14. Berrocal E, Bautista-Gomez L, Di S, Cappello F. Exploring partial replication to improve lightweight silent data corruption detection for HPC applications. In: Proceedings of Europe Conference on Parallel Processing. 2016, 419–430

  15. Berrocal E, Bautista-Gomez L, Di S, Lan Z L, Cappello F. Toward general software level silent data corruption detection for parallel applications. IEEE Transactions on Parallel and Distributed Systtems, 2017, 28(12): 3642–3655

    Article  Google Scholar 

  16. Guo L Z, Li D, Laguna I, Schulz M. FlipTracker: understanding natural error resilience in HPC applications. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis. 2018, 94–107

  17. Gomez L B, Cappello F. Detecting silent data corruption through data dynamic monitoring for scientific applications. In: Proceedings of the 19th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming. 2014, 381–382

  18. Subasi O, Di S, Gomez L B, Balaprakash P, Unsal O, Labarta J, Cristal A, Krishnamoorthy S, Cappello F. Exploring the capabilities of support vector machines in detecting silent data corruptions. Sustainable Computing: Informatics and Systems, 2018, 19: 277–290

    Google Scholar 

  19. Liu J, Agrawal G. Soft error detection for iterative applications using offline training. In: Proceedings of the 23rd IEEE International Conference on High Performance Computing. 2016, 2–11

  20. Hassani A, Skjellum A, Brightwell R. Design and evaluation of FA-MPI, a transactional resilience scheme for non-blocking MPI. In: Proceedings of the 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. 2014, 750–755

  21. Li Y, Lan Z, Gujrati P, Sun X H. Fault-aware runtime strategies for highperformance computing. IEEE Transactions on Parallel and Distributed Systems, 2009, 4(20): 460–473

    Google Scholar 

  22. Hargrove P H, Duell J C. Berkeley lab checkpoint/restart (BLCR) for linux clusters. Journal of Physics: Conference Series, 2006, 46(1): 494–499

    Google Scholar 

  23. Gomez L B, Tsuboi S, Komatitsch D, Cappello F, Maruyama N, Matsuoka S. FTI: high performance fault tolerance interface for hybrid systems. In: Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis. 2011, 1–12

  24. Buntinas D, Coti C, Herault T, Lemarinier P, Pilard L. Blocking vs. non-blocking coordinated checkpointing for large-scale fault tolerant MPI protocols. Future Generation Computer Systems, 2008, 24(1): 73–84

    Article  Google Scholar 

  25. Di S, Bouguerra M S, Gomez L B, Cappello F. Optimization of multilevel checkpoint model for large scale HPC applications. In: Proceedings of the 28th IEEE International Parallel and Distributed Processing Symposium. 2014, 1181–1190

  26. Ho J C Y, Wang C L, Lau F C M. Scalable group-based checkpoint/restart for large-scale message-passing systems. In: Proceedings of 2008 IEEE International Symposium on Parallel and Distributed Processing. 2008, 1–12

  27. Agarwal S, Garg R, Gupta M S, Moreira J E. Adaptive incremental checkpointing for massively parallel systems. In: Proceedings of the 18th Annual International Conference on Supercomputing. 2004, 277–286

  28. Nicolae B, Cappello F. AI-Ckpt: leveraging memory access patterns for adaptive asynchronous incremental checkpointing. In: Proceedings of the 22nd International Symposium on High-Performance Parallel and Distributed Computing. 2013, 155–166

  29. Ferreira K B, Riesen R, Bridges P, Arnold D, Brightwell R. Accelerating incremental checkpointing for extreme-scale computing. Future Generation Computer Systems, 2014, 30: 66–77

    Article  Google Scholar 

  30. Nicolae B, Moody A, Gonsiorowski E, Mohror K, Cappello F. VeloC: towards high performance adaptive asynchronous checkpointing at large scale. In: Proceedings of IEEE International Parallel and Distributed Processing Symposium. 2019, 911–920

  31. Losada N, Bosilca G, Bouteiller A, Gonzalez P, Martin M J. Local rollback for resilient MPI applications with application-level checkpointing and message logging. Future Generation Computer Systems, 2019, 91: 450–464

    Article  Google Scholar 

  32. Lan Z L, Li Y W. Adaptive fault management of parallel applications for high performance computing. IEEE Transactions on Computers, 2008, 57: 1647–1660

    Article  MATH  Google Scholar 

  33. Ibtesham D, Arnold D, Bridges P G, Ferreira K B, Brightwell R. On the viability of compression for reducing the overheads of checkpoint/restart-based fault tolerance. In: Proceedings of the 41st International Conference on Parallel Processing. 2012, 148–157

  34. Zhou P, Liu W, Fei L, Fei L, Lu S, Qin F, Zhou Y Y, Midkiff S, Torrellas G. Accmon: automatically detecting memory-related bugs via program counter-based invariants. In: Proceedings of the 37th Annual IEEE/ACM International Symposium on Microarchitecture. 2004, 269–280

  35. Zheng Z, Li Y, Lan Z L. Anomaly localization in large-scale clusters. In: Proceedings of 2007 IEEE International Conference on Cluster Computing. 2007, 322–330

  36. Yu L, Lan Z L. A scalable, non-parametric method for detecting performance anomaly in large scale computing. IEEE Transactions on Parallel and Distributed Systems, 2016, 27(1): 1902–1914

    Article  Google Scholar 

  37. Zheng Z, Yu L, Tang W, Lan Z L, Gupta R, Desai N, Coghlan S, Buettner D. Co-analysis of RAS log and job log on Blue Gene/P. In: Proceddings of 2011 IEEE International Parallel & Distributed Processing Symposium. 2011, 840–851

  38. Berrocal E, Yu L, Wallace S, Papka M E, Lan Z L. Exploring void search for fault detection on extreme scale systems. In: Proceedings of IEEE International Conference on Cluster Computing. 2014, 1–9

  39. Gujrati P, Li Y, Lan Z L, Thakur R, White J. A meta-learning failure predictor for Blue Gene/L systems. In: Proceedings of 2007 International Conference on Parallel Processing. 2007

  40. Zheng Z, Lan Z L, Park B H, Geist A. System log pre-processing to improve failure prediction. In: Proceedings of 2009 IEEE/IFIP International Conference on Dependable Systems & Networks. 2009, 572–577

Download references

Acknowledgements

This work was supported by National Key R&D Program of China (2020YFB150001) and the National Natural Science Foundation of China (Grant No. 62072018).

Author information

Authors and Affiliations

  1. State Key Laboratory of Software Development Environment, Beijing, 100191, China

    Guozhen Zhang & Hailong Yang

  2. Sino-German Joint Software Institute, Beihang University, Beijing, 100191, China

    Guozhen Zhang, Yi Liu, Hailong Yang & Depei Qian

  3. School of Computer Science and Engineering, Beihang University, Beijing, 100191, China

    Guozhen Zhang, Yi Liu, Hailong Yang & Depei Qian

  4. Science and Technology on Space System Simulation Laboratory Beijing Simulation Center, Beijing, 100854, China

    Jun Xu

Authors
  1. Guozhen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hailong Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Depei Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hailong Yang.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, G., Liu, Y., Yang, H. et al. User-level failure detection and auto-recovery of parallel programs in HPC systems. Front. Comput. Sci. 15, 156107 (2021). https://doi.org/10.1007/s11704-020-0190-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11704-020-0190-y

Keywords

Search

Navigation