Skip to main content
SpringerLink
Log in

Exascale Data Processing in Heterogeneous Distributed Computing Infrastructure for Applications in High Energy Physics

  • Published:
Physics of Particles and Nuclei Aims and scope Submit manuscript

Abstract

Research in High Energy and Nuclear Physics is impossible without the use of significant computing power and software for processing, simulation, and analysis of data. The construction and commissioning of facilities such as the Large Hadron Collider and NICA required new approaches and algorithms for developing the data processing and management systems. At the same time, heterogeneous computing resources are being integrated into a single cyber infrastructure and the concept of a “scientific data lake” is being developed. This paper discusses the evolution of data processing systems and computer solutions for experiments in the field of particle physics.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.
Fig. 8.
Fig. 9.
Fig. 10.
Fig. 11.
Fig. 12.
Fig. 13.
Fig. 14.
Fig. 15.
Fig. 16.
Fig. 17.
Fig. 18.
Fig. 19.
Fig. 20.
Fig. 21.
Fig. 22.
Fig. 23.
Fig. 24.
Fig. 25.
Fig. 26.
Fig. 27.
Fig. 28.
Fig. 29.
Fig. 30.
Fig. 31.
Fig. 32.
Fig. 33.
Fig. 34.
Fig. 35.
Fig. 36.
Fig. 37.
Fig. 38.
Fig. 39.
Fig. 40.
Fig. 41.
Fig. 42.
Fig. 43.
Fig. 44.
Fig. 45.
Fig. 46.
Fig. 47.
Fig. 48.
Fig. 49.
Fig. 50.
Fig. 51.
Fig. 52.
Fig. 53.
Fig. 54.
Fig. 55.
Fig. 56.
Fig. 57.
Fig. 58.
Fig. 59.
Fig. 60.
Fig. 61.
Fig. 62.
Fig. 63.
Fig. 64.
Fig. 65.
Fig. 66.
Fig. 67.
Fig. 68.
Fig. 69.

Similar content being viewed by others

Notes

  1. Already at an early stage, the concept of a scientific data lake in Russia was supported by Academician of the Russian Academy of Sciences G.V. Trubnikov. At the moment, leading universities (St. Petersburg State University, NRNU MEPhI, Plekhanov Russian University of Economics) and scientific organizations (JINR and the Petersburg Nuclear Physics Institute of the NRC KI) are participating in this project. This project is supported by a grant from the Russian Science Foundation. Scientists and IT specialists of LIT JINR play one of the leading roles in the development of the data lake concept in Russia.

REFERENCES

  1. LHC—The Large Hadron Collider. http://lhc.web.cern. ch/lhc.

  2. The ATLAS Collaboration, G. Aad et al., “The ATLAS Experiment at the CERN Large Hadron Collider,” J. Instrum. 3, 08003 (2008).

    Article  Google Scholar 

  3. The CMS Collaboration, S. Chatrchyan et al., “The CMS experiment at the CERN LHC,” J. Instrum. 3, 08004 (2008).

    Article  Google Scholar 

  4. ALICE Collaboration, K. Aamond et al., “The ALICE experiment at the CERN LHC,” J. Instrum. 3, 08002 (2008).

    Article  Google Scholar 

  5. FAIR Baseline Technical Report, Ed. by H.H. Gutbrod et al., ISBN 3-9811298-0-6 Nov. 2006

  6. XFEL: The European X-ray Free-Electron Laser Technical Design Report, Ed. by M. Altarelli et al., DESY 2006-097 (DESY, 2007).

  7. G.V. Trubnikov et al. “Project of the Nuclotron-based Ion Collider Facility (NICA) at JINR”, in Proceedings EPAC 08 (Genoa, 2008), pp. 2581–2583.

  8. The ATLAS Collaboration, “Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC,” Phys. Lett. B 716, 1–29 (2012).

    Article  ADS  Google Scholar 

  9. J. Ratchford and U. Colombo, “Megascience,” in UNESCO World Science Report, 1996.

  10. J. Markoff, “A Deluge of Data Shapes a New Era in Computing”. http://www.nytimes.com/2009/12/15/ science/15books.html.

  11. http://www.fourthparadigm.org.

  12. J. Gray, “eScience—A Transformed Scientific Method”, Talk given to the NRC-CSTB, Mountain View, CA, United States, January 11, 2007. http://research.microsoft.com/en-us/um/people/gray/talks/NRC-CSTB_eScience.ppt.

  13. A. S. Tanenbaum and M. Van Steen, Distributed Systems. Principles and Paradigms (net, 2017).

  14. V. V. Voevodin and Vl. V. Voevodin, Parallel Computing (BVKh, St. Petersburg, 2002), p. 608.

  15. N. N. Govorun, “Some issues of application of electronic computers in physics research,” Extended abstract of Doctoral dissertation in Physics and Mathematics (JINR, Dubna, 1969).

  16. WLCG: Worldwide LHC Computing Grid. http://wlcg.web.cern.ch.

  17. I. Foster and K. Kesselman, GRID: A Blueprint for the New Computing Infrastructure (Morgan Kaufman Publishers, 1999).

    Google Scholar 

  18. LHCOPN: LHC Optical Private Network. http://wlcg.web.cern.ch.

  19. LHCONE: LHC Open Network Environment. http://lhcone.cern.ch.

  20. A. Klimentov, “On federated organization of distributed CERN,” Superkomp’yutery 20, 26–28 (2015).

    Google Scholar 

  21. Globus Project. http://toolkit.globus.org/toolkit/about.html.

  22. V. Ilyin, V. Korenkov, A. Kryukov, Yu. Ryabov, and A. Soldatov, “Russian Data Intensive Grid (RDIG): Current status and perspectives towards national Grid initiative,” in Proceedings of the International Conference on Distributed computing and Grid-Technologies in Science and Education, GRID-2008, ISBN 978-5-9530-0198-4 (Dubna, 2008), pp. 100–108.

  23. Vl. V. Voevodin and S. A. Zhumatii, Computations and Cluster Systems (Izd. MGU, Moscow, 2007).

  24. M. Aderholz et al., Models of Networked Analysis at Regional Centers for LHC Experiments (MONARC), Phase 2 Report CERN/LCB, 2000-001; http://monarc. web.cern.ch/MONARC.

  25. S. Campana, A. DiGirolamo, J. Elmsheuser, S. Jezequel, A. Klimentov, J. Schovancova, C. Serfon, G. Stewart, D. van der Ster, I. Ueda, and A. Vaniachine, “ATLAS Distributed Computing Operations: Experience and improvements after 2 full years of data-taking,” in Proceedings of the 19th International Conference on Computing in High Energy and Nuclear Physics, May 2012 (CHEP12).

  26. P. Nilsson, K. De, A. Filipcic, A. Klimentov et al., “Extending ATLAS Computing to Commercial Clouds and Supercomputers,” PoS ISGC2014, 034 (2014).

  27. A. Zarochentsev, A. Kiryanov, A. Klimentov, D. Krasnopevtsev, and P. Hristov, “Federated data storage and management infrastructure,” J. Phys. Conf. Ser. 762, 012016 (2016).

    Article  Google Scholar 

  28. A. Klimentov, A. Kir’yanov, and A. Zarochintsev, “Russian scientific data lake,” Otkrytye sistemy, No. 4, (2018).

  29. Load Sharing Facility. https://www-03.ibm.com/systems/spectrum-computing/products/lsf/index.html.

  30. Portable Batch System. http://www.pbspro.org/.

  31. HTCondor. Official site: https://research.cs. wisc.edu/htcondor/

  32. S. Bagnaso, L. Betev, P. Buncic et al., “The ALICE Workload Management System: Status before the real data taking,” J. Phys. Conf. Ser. 219, 062004 (2010).

    Article  Google Scholar 

  33. S. K. Paterson and A. Tsaregorodtsev, “DIRAC optimized workload management,” J. Phys. Conf. Ser. 119, 062040 (2008).

    Article  Google Scholar 

  34. K. De, A. Klimentov, T. Maeno, T. Wenaus et al., “Next generation workload management system for Big Data on heterogeneous distributed computing,” J. Phys. Conf. Ser. 608 (1), 012040 (2015).

    Article  Google Scholar 

  35. S. Cittolin et al., “A Remus based crate controller for the autonomous processing of multichannel data streams,” Preprint CERN 81-07, Geneva.

  36. S. Burov, A. Klimentov, V. Koutsenko et al., “The distributed DAQ system of hadron calorimeter prototype,” Preprint ITEP-18, Moscow (1989).

  37. A. A. Klimentov, “Development of a complex of automated stands for testing in production, assembling, and commissioning of a hadron calorimeter of the L3 detector at the LEP accelerator,” Extended abstract of Candidate’s dissertation in Physics and Mathematics (ITEP, Moscow, 1991). RGB OD, 9 91-2/3642-7.

  38. V. Choutko and A. Klimentov, “Computing strategy of Alpha-Magnetic spectrometer experiment,” Nucl. Instrum. Methods Phys. Res., Sect. A 502, 461–466 (2003).

    Google Scholar 

  39. A. Klimentov et al., “AMS-02 Computing and Ground Data Handling”, in Proceedings of the Conference on Computing in High Energy Physics, Sep 2004, Interlaken, Switzerland.

  40. J.-P. Baud et al., “CASTOR status and evolution,” in Proceedings of the Conference on Computing in High Energy and Nuclear Physics Conference, La Jolla, California, United States (CHEP 2003).

  41. Mihai Dobre and C. Stratan, “Monarc simulation framework,” in Proceedings of the RoEduNet International Conference, Buletinul Stiintific al Universitatii “Politehnica” din Timisoara, Romania, Seria Automatica si Calculatoare Periodica Politechnica. IEEE T. Automat. Cont. Comput. Sc. 49 (63), 35–42 (2004).

  42. Enabling Grids for E-science in Europe. http://www.eu-egee.org

  43. Project on Development of Fundamental Grid Technologies, Globus Alliance. http://www.globus.org/.

  44. ROOT: Data analysis framework. https://root.cern.ch.

  45. D. Costanzo, A. Klimentov et al., “Metadata for ATLAS,” Preprint ATL-GEN-PUB-2007-001, ATL-COM-GEN-2007-001.

  46. M. Lassnig, “Using machine learning algorithms to forecast network and system load metrics for ATLAS Distributed Computing,” in Proceedings of the Conference on Computing in High Energy and Nuclear Physics, San Francisco, United States, October 2016.

  47. M. Titov, G. Zaruba, A. Klimentov, and K. De, “A probabilistic analysis of data popularity in ATLAS data caching,” J. Phys. Conf. Ser. 396, 032106 (2012).

    Article  Google Scholar 

  48. perfSONAR. http://www.perfsonar.net/

  49. A. Klimentov and M. Titov “ATLAS Data Transfer Request Package (DaTRI),” J. Phys. Conf. Ser., Proceedings of the 18th Int. Conf. on Computing in High Energy and Nuclear Physics (CHEP2010).

  50. J. Schovancova et al. “Evolution of HammerCloud to commission CERN Compute resources,” in Proceedings of the 23rd International Conference on Computing in High Energy and Nuclear Physics (Sofia, 2018).

  51. A. Anisenkov, A. Klimentov, R. Kuskov and T. Wenaus, “ATLAS Grid information system,” J. Phys. Conf. Ser. 331, 072002 (2011).

    Article  Google Scholar 

  52. M. Pradillo et al., “Consolidating WLCG topology and configuration in the Computing Resource Information Catalogue,” in Proceedings of the Conference on Computing in High Energy and Nuclear Physics, San Francisco, United States, October 2016.

  53. D. Oleynik, A. Petrosyan, V. Garonne, and S. Campana, “On behalf of the ATLAS Collaboration: DDM DQ2 Deletion Service, Implementation of Central Deletion Service for ATLAS Experiment,” in Proceedings of the 5th Intern. Conf. “Distributed Computing and Grid-Technologies in Science and Education, GRID-2012”, ISBN 978-5-9530-0345-2 (Dubna, 2012), pp.189–194.

  54. Les Robertson, “LHC Data Analysis will start on the Grid. What’s next?” in Proceedings of the Conference on Computing in High Energy and Nuclear Physics, Prague, Czech Republic, March 2009 (CHEP 2009).

  55. Aristotle, Metaphysics (2015). ISBN 978-5-699-83195-1.

  56. H. S. C. Martin, S. Jha, S. Howorka, and P. V. Coveney. “Determination of free energy profiles for the translocation of polynucleotides through α-Hemolysin nanopores using non-equilibrium molecular dynamics simulations,” J. Chem. Theor. Comput. 5, 1955–2192 (2009).

    Article  Google Scholar 

  57. K. De, S. Jha, A. Klimentov, D. Oleynik, J. Wells et al. “High-throughput computing on high-performance platforms: A Case Study,” Oct 2017, arxiv.org 1704.00978 v2.

  58. Top500, November 2019. https://top500.org/lists/ top500/2019/11.

  59. G. Stewart. “Evolution of computing and software at LHC: from Run2 to HL-LHC,” in Proceedings of the Conference on Computing in High Energy and Nuclear Physics, April 2015, Okinawa, Japan.

  60. M. Borodin, K. De, J. Garcia Navarro, D. Golubkov, A. Klimentov, T. Maeno, and A. Vaniachine, “Scaling up ATLAS production system for the LHC Run 2 and beyond: project ProdSys2,” J. Phys. Conf. Ser. 664 062005 (2015).

    Article  Google Scholar 

  61. P. J. Laycock, N. Ozturk, M. Beckingham, R. Henderson, and L. Zhou, “Derived physics data production in ATLAS: Experience with Run 1 and looking ahead,” J. Phys. Conf. Ser. 513, 03252 (2014).

    Article  Google Scholar 

  62. http://www.wired.com/magazine/2013/04/bigdata.

  63. Bernd Panzer-Steindel, “Introduction to CERN Computing,” CERN Summer Lecture 2015.

    Google Scholar 

  64. Proceedings of Workshop “BigData Processing and Analysis Challenges”, January 29–31, 2015 (NRC KI, Moscow, 2015); https://indico.cern.ch/event/364112/.

  65. M. A. Grigorieva, M. V. Golosova, M. Y. Gubin, A. A. Klimentov, V. V. Osipova, and E. A. Ryabinkin, “Evaluating non-relational storage technology for HEP metadata and meta-data catalog,” J. Phys. Conf. Ser. 762, 012017 (2016).

    Article  Google Scholar 

  66. Django software foundation; https://www.djangoproject.com/foundation/.

  67. ds.j3; https://d3js.or.

  68. K. De, A. Klimentov, J. Schovancova, and T. Wenaus, “The new generation of the ATLAS PanDA monitoring system,” PoS ISGC2014, 035 (2014).

  69. T. Korchuganova, S. Padolski, T. Wenaus. “ATLAS BigPanDA monitoring and its evolution,” in Proceedings of the 7th International Conf. “Distributed Computing and Grid-technologies in Science and Education” (Dubna, Russia, 2016).

  70. I. Foster, Y. Zhao, I. Raicu, and S. Lu, “Cloud Computing and Grid Computing 360-Degree Compared”. https://arxiv.org/pdf/0901.0131.pdf.

  71. M. Sevior, “Belle Monte-Carlo production on the Amazon EC2 cloud,” In Proceedings of the International Conf. ISGC, Taipei, Taiwan, April 2009.

  72. Google Compute Engine Portal. https://cloud.google.com/products/compute-engine.

  73. HTCondor Project. http://research.cs.wisc.edu/htcondor.

  74. CVMFS Portal. http://cernvm.cern.ch/portal/filesystem.

  75. CernVM-FS Docuentation. https://cernvm.cern.ch

  76. K. De, A. Klimentov, M. Lassnig, et al., “ATLAS & Google — "Data Ocean” R&D Project,” Preprint CERN ATL-SOFT-PUB-2017-002, Geneva.

  77. Amazon EC2. http://aws.amazon.com/ec2/pricing

  78. TOP500 list. https://en.wikipedia.org/wiki/TOP500

  79. International Conference on Supercomputers 2016, Salt Lake City, United States. http://sc16.supercomputing.org.

  80. Geant4. http://geant4.cern.ch.

  81. SAGA-Python (Simple API for Grid Applications). http://saga-project.github.io/saga-python/.

  82. Gaudi Project. http://gaudi.web.cern.ch/gaudi/.

  83. AliROOT. ALICE Offline project. https://alice-offline.web.cern.ch.

  84. M. Al-Turani et al., “ALFA: The new ALICE-FAIR software framework,” J. Phys. Conf. Ser. 664, 072001 (2015).

    Article  Google Scholar 

  85. V. A. Aulov, A. A. Klimentov, R. Yu. Mashinistov, A. V. Nedoluzhko, A. M. Novikov, A. A. Poida, I. S. Tertychnyi, A. B. Teslyuk, and F. S. Sharko, “Integration of heterogeneous computing infrastructures for genomic sequencing data analysis,” Matematicheskaya biologiya i bioinformatika 11, 205–213 (2016).

  86. K. G. Skryabin, E. B. Prokhortchouk, A. M. Mazur, E. S. Boulygina, S. V. Tsygankova, A. V. Nedoluzhko, S. M. Rastorguev, V. B. Matveev, N. N. Chekanov, D. A. Goranskaya, A. B. Teslyuk, N. M. Gruzdeva, V. E. Velikhov, D. G. Zaridze, and M. V. Kovalchuk. “Combining two technologies for full genome sequencing of human,” Acta Nat. 1 (3), 102–107 (2009).

    Article  Google Scholar 

  87. M. Schubert, L. Ermini, C. D. Sarkissian, H. Jonsson, A. Ginolhac, R. Schaefer, M. D. Martin, R. Fernandez, M. Kircher, M. McCue, E. Willerslev, and L. Orlando, “Characterization of ancient and modern genomes by SNP detection and phylogenomic and metagenomic analysis using PALEOMIX,” Nat. Protoc. 9, 1056–1082 (2014).

    Article  Google Scholar 

  88. “ATLAS Collab. Search for a charged Higgs boson produced in the vector-boson fusion mode with decay H(±)→W(±)Z using pp collisions at √s=8  TeV with the ATLAS experiment,” Phys. Rev. Lett. 114, 231801 (2015).

  89. A. Klimentov, M. Grigorieva, A. Kiryanov, and A. Zarochentsev, “BigData and computing challenges in high energy and nuclear physics,” J. Inst. 12, C06044 (2017).

  90. “HEP Software Foundation. A Roadmap for HEP Software and Computing R&D for the 2020s,” arXiv: 1712.06982 [physics.comp-ph], Dec 2017.

  91. M. Lassnig, A. Klimentov, K. De, et al. “ATLAS & Google - The Data Ocean Project,” in Proceedings of the 23rd International Conference on Computing in High Energy and Nuclear Physics, Sofia, Bulgaria, July 9–13, 2018 (CHEP, 2018).

  92. A. Zarochentsev, A. Kiryanov, A. Klimentov, D. Krasnopevtsev, and P. Hristov. “Federated data storage and management infrastructure,” J. Phys. Conf. Ser. 762, 012016 (2016).

    Article  Google Scholar 

Download references

ACKNOWLEDGMENTS

I am grateful to my teachers and mentors, Professors A.V. Arefiev, Yu.A. Kamyshkov, S.C.C. Ting, and Yu.V. Galaktionov and my colleagues in the ATLAS, AMS, and L3 experiments for fruitful joint work, discussions, and materials provided.

Many of the results presented in this review are joint works with M.S. Borodin, D.V. Golubkov, M.A. Grigorieva, D.V. Krasnopevtsev, T.A. Korchuganova, A.A. Alekseev, A.V. Vanyashin, R.Yu. Mashinistov, M.A. Titov, S. Jha; ATLAS members: T. Wenaus, T. Maeno, K. De, F. Barreiro, S. Yu. Panitkin, S.V. Podolskii, and S.Yu. Smirnov; colleagues from the ORNL (J. Wells) and CERN (I. Bird, S. Campana, A. DiGirolamo, M. Lassnig, and G. Stewart). Especially noteworthy is the decisive contribution of A.K. Kir’yanov and A.K. Zarochentsev to the development of a prototype of a federated computer infrastructure and a data lake, starting from the earliest developments, as well as the important role of LIT JINR staff in a number of pioneering research on supercomputers (D. A. Oleinik), on the development and adaptation of data processing systems for fixed target experiments (A.Sh. Petrosyan), and the development of architecture and methods for testing data lakes (V.V. Mitsyn). I am grateful to Professors A.I. Avetisyan, V. V. Voevodin, V.P. Gerdt, P.V. Zrelov, V.A. Il’in, V.V. Ivanov, V.V. Koren’kov, and G.A. Ososkov for discussion of the work.

Funding

Since 2019, the work on the development of the Russian scientific data lake has been carried out at the Plekhanov Russian University of Economics and are supported by the Russian Science Foundation (grant no. 19-71-30008).

Author information

Authors and Affiliations

  1. Brookhaven National Laboratory, Upton, NY, United States

    A. A. Klimentov

Authors
  1. A. A. Klimentov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. A. Klimentov.

Additional information

Translated by E. Chernokozhin

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Klimentov, A.A. Exascale Data Processing in Heterogeneous Distributed Computing Infrastructure for Applications in High Energy Physics. Phys. Part. Nuclei 51, 995–1068 (2020). https://doi.org/10.1134/S1063779620060052

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S1063779620060052

Search

Navigation