Abstract
Scientific computing has evolved considerably in recent years. Scientific applications have become more complex and require an increasing number of computing resources to perform on a large scale. Grid computing has become widely used and is the chosen infrastructure for many scientific calculations and projects, even though it demands a steep learning curve. The computing and storage resources in the Grid are limited, heterogeneous and often overloaded. This heterogeneity is not only present in the hardware setups, but also in the software composition, where configuration permissions are limited. It also has a negative effect on the portability of scientific applications. The use of Cloud resources could eliminate those constraints. In the Cloud, resources are provisioned on demand and can be scaled up and down, while scientists can easily customize their execution environments in the form of virtual machines. Extending the Grid with Cloud resources would improve the utilization of shared resources and would enable the use of additional resources when the Grid resources are overloaded – known as Cloud bursting. We propose an integration model of the Grid and the Cloud using the HTCondor batch system and the NorduGrid ARC middleware. This model enables batch job execution in any public or private Cloud by deploying a virtualized Grid cluster using the ARC middleware - PaaS model for running Grid applications. An evaluation of the virtual Grid cluster was made and compared with the physical one by running NAMD simulations.
Similar content being viewed by others
References
Salnikov, A.: RAINBOW - ARC in the cloud. In: RAINBOW - ARC in the Cloud. Helsinki, Finland (2014)
Apoa1 NAMD benchmark. http://www.ks.uiuc.edu/Research/namd/utilities
Arsuaga-Ríos, M., Heikkilä, S.S., Duellmann, D., Meusel, R., Blomer, J., Couturier, B.: Using S3 cloud storage with ROOT and cvmFS. J. Phys. Conf. Ser. 664(2), 022001 (2015). https://doi.org/10.1088/1742-6596/664/2/022001
Brandic, I., Buyya, R.: Special section: recent advances in utility and cloud computing. Futur. Gener. Comput. Syst. 28(1), 36–38 (2012). https://doi.org/10.1016/j.future.2011.06.001
Burrows, M.: The chubby lock service for loosely-coupled distributed systems. In: Proceedings of the 7th Symposium on Operating Systems Design And Implementation, pp. 335–350. USENIX Association (2006)
Buyya, R., Yeo, C.S., Venugopal, S., Broberg, J., Brandic, I.: Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility. Futur. Gener. Comput. Syst. 25(6), 599–616 (2009). https://doi.org/10.1016/j.future.2008.12.001
Cafaro, M., Aloisio, G. (eds.): Grids, Clouds and Virtualization. Computer Communications And Networks. Springer, London (2011)
Cameron, D., Ellert, M., Jönemo, J., Konstantinov, A., Marton, I., Mohn, B., Nilsen, J.K., Nórden, M., Qiang, W., Roczei, G.: The hosting environment of the advanced resource connector middleware NORDUGRID-TECH-19 (2013)
Chadwick, D.W., Siu, K., Lee, C., Fouillat, Y., Germonville, D.: Adding federated identity management to OpenStack. Journal of Grid Computing 12 (1), 3–27 (2014). https://doi.org/10.1007/s10723-013-9283-2
Consul. https://www.consul.io/
CoreOS/etcd. https://github.com/coreos/etcd
Cunha, J.C., Medeiros, P.D.: Special issue: parallel and distributed computing (EuroPar 2005). Concurrency and Computation: Practice and Experience 19(17), 2183–2184 (2007). https://doi.org/10.1002/cpe.1267
DIRAC interware. http://diracgrid.org/
Eerola, P., Ekelof, T., Ellert, M., Hansen, J.R., Konstantinov, A., Konya, B., Nielsen, J.L., Ould-Saada, F., Smirnova, O., Waananen, A.: The NorduGrid architecture and tools. arXiv:physics/0306002 (2003)
EUgridPMA. https://www.eugridpma.org/
European grid initiative. http://www.egi.eu
Foster, I., Zhao, Y., Raicu, I., Lu, S.: Cloud computing and grid computing 360-degree compared. In: Cloud Computing and Grid Computing 360-Degree Comparedgrid Computing Environments Workshop, 2008. GCE’08, pp. 1–10 (2008)
Furht, B., Escalante, A. (eds.): Handbook of Cloud Computing. Springer, Boston (2010)
Hariri, S., Varma, A.: High-performance distributed computing: promises and challenges. Concurrency: Practice and Experience 5(4), 233–238 (1993)
Haug, S, Sciacca, FG, Collaboration, A: ATLAS computing on swiss cloud SWITCHengines. J Phys Conf Ser 898(5), 052017 (2017). http://stacks.iop.org/1742-6596/898/i=5/a=052017
Hunter, A.A., Macgregor, A.B., Szabo, T.O., Wellington, C.A., Bellgard, M.I.: Yabi: an online research environment for grid, high performance and cloud computing. Source Code Biol. Med. 7(1), 1 (2012)
Gable, I.: HEP Cloud Production Using the Cloudscheduler/HTCondor architecture. In: CHEP, Okinawa (2015)
Li, K.: Optimal load distribution for multiple heterogeneous blade servers in a cloud computing environment. Journal of Grid Computing 11(1), 27–46 (2013). https://doi.org/10.1007/s10723-012-9239-y
Litzkow, M., Livny, M., Mutka, M.: Condor - a hunter of idle workstations. In: Proceedings of the 8th International Conference of Distributed Computing Systems, pp. 104–111 (1988)
Livny, M., Basney, J., Raman, R., Tannenbaum, T.: Mechanisms for high throughput computing. SPEEDUP Journal 11(1), 36–40 (1997)
Lopez Garcia, A., Fernandez-del-Castillo, E., Puel, M.: Identity federation with VOMS in cloud infrastructures. In: 2013 IEEE 5th International Conference on on Cloud Computing Technology and Science Cloud Computing Technology and Science (CloudCom), vol. 1, pp. 42–48 (2013)
Lopez Garcia, A., Fernandez-del-Castillo, E., Puel, M.: Voms-aware identity service for Openstack. In: EGI Community Forum (2013)
Mateescu, G., Gentzsch, W., Ribbens, C.J.: Hybrid computing—Where HPC meets grid and cloud computing. Futur. Gener. Comput. Syst. 27(5), 440–453 (2011). https://doi.org/10.1016/j.future.2010.11.003
Ellert, M., Michael Grønager, A.K., Kónya, B, Lindemann, J, Livenson, I, Nielsen, JL, Niinimäki, M., Smirnova, O., Wäänänen, A.: Advanced resource connector middleware for lightweight computational grids. Futur. Gener. Comput. Syst. 23(2), 219–240 (2007)
Montero, R.S., Huedo, E., Llorente, I.M.: Dynamic deployment of custom execution environments in grids. In: Dynamic Deployment of Custom Execution Environments in Grids, pp. 33–38, IEEE (2008). https://doi.org/10.1109/ADVCOMP.2008.8
NAMD - Scalable molecular dynamics. http://www.ks.uiuc.edu/Research/namd/
Nordugrid - Grid Research and Development collaboration. www.nordugrid.org
OpenStack - open source cloud computing software. http://www.openstack.org/
Openstack cinder. https://docs.openstack.org/cinder/latest/
OpenStack cloud sofware - openstack training guide. http://docs.openstack.org/training-guides/content/module001-ch004-openstack-architecture.html (2014)
Openstack swift. https://docs.openstack.org/swift/latest/
Osmani, L., Tarkoma, S., Eerola, P., Komu, M., Kortelainen, M.J., Kraemer, O., Lindén, T., Toor, S., White, J.: An overview of the DII-HEP OpenStack based CMS data analysis. J. Phys. Conf. Ser. 608(1), 012010 (2015). http://stacks.iop.org/1742-6596/608/i=1/a=012010
Ostermann, S., Plankensteiner, K., Prodan, R.: Using a new event-based simulation framework for investigating resource provisioning in clouds. Sci. Program. 19(2), 161–178 (2011)
Ostermann, S., Prodan, R., Fahringer, T.: Extending grids with cloud resource management for scientific computing. In: 2009 10Th IEEE/ACM International Conference On Grid Computing, pp. 42–49. IEEE (2009)
Packer. https://packer.io/
Partnership for advanced computing in europe. http://www.prace-ri.eu/
Phillips, J.C., Zheng, G., Kumar, S: NAMD: biomolecular simulation on thousands of processors. In: ACM/IEEE 2002 Conference on Supercomputing, pp. 36–36 (2002)
Phoronix test suite. http://www.phoronix-test-suite.com/
Alfieri, R., Cecchini, R., Ciaschini, V., dell’Agnello, L., Frohner, Á., Gianoli, A.: VOMS, an Authorization System for Virtual Organizations. In: VOMS, an Authorization System for Virtual Organizations, vol. 2970, pp. 33–40. Springer, Berlin (2004)
Ranganathan, A., Campbell, R.H.: What is the complexity of a distributed computing system? Complexity 12(6), 37–45 (2007). https://doi.org/10.1002/cplx.20189
Montero, R.S., Moreno-Vozmediano, R, Llorente, I.M.: An elasticity model for high throughput computing clusters. J. Parallel Distrib. Comput., Special issue on cloud computing 71(6), 750–757 (2011)
Schwiegelshohn, U., Badia, R.M., Bubak, M., Danelutto, M., Dustdar, S., Gagliardi, F., Geiger, A., Hluchy, L., Kranzlmüller, D., Laure, E., Priol, T., Reinefeld, A., Resch, M., Reuter, A., Rienhoff, O., Rüter, T., Sloot, P., Talia, D., Ullmann, K., Yahyapour, R., von Voigt, G.: Perspectives on grid computing. Futur. Gener. Comput. Syst. 26(8), 1104–1115 (2010). https://doi.org/10.1016/j.future.2010.05.010
Slovenian national supercomputing network. https://www.sling.si
Slurm workload manager. https://slurm.schedmd.com/
Sysbench benchmarking tool. https://launchpad.net/sysbench
Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurrency-Practice and Experience 17(2-4), 323–356 (2005)
The Atlas Collaboration: The ATLAS experiment at the CERN large hadron collider. J Instrum 3(08), S08,003 (2008)
Worldwide lhc computing grid. http://wlcg.web.cern.ch/
Zookeeper. https://zookeeper.apache.org/
Acknowledgements
The authors would like to thank Jan Jona Javoršek and Farid Ould-Saada for their help and advice on the subject.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Krašovec, B., Filipčič, A. Enhancing the Grid with Cloud Computing. J Grid Computing 17, 119–135 (2019). https://doi.org/10.1007/s10723-018-09472-w
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10723-018-09472-w