当前位置: X-MOL 学术Big Data Res. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Automated Deployment of a Spark Cluster with Machine Learning Algorithm Integration
Big Data Research ( IF 3.3 ) Pub Date : 2020-05-08 , DOI: 10.1016/j.bdr.2020.100135
A.M. Fernández , D. Gutiérrez-Avilés , A. Troncoso , F. Martínez–Álvarez

The vast amount of data stored nowadays has turned big data analytics into a very trendy research field. The Spark distributed computing platform has emerged as a dominant and widely used paradigm for cluster deployment and big data analytics. However, to get started up is still a task that may take much time when manually done, due to the requisites that all nodes must fulfill. This work introduces LadonSpark, an open-source and non-commercial solution to configure and deploy a Spark cluster automatically. It has been specially designed for easy and efficient management of a Spark cluster with a friendly graphical user interface to automate the deployment of a cluster and to start up the distributed file system of Hadoop quickly. Moreover, LadonSpark includes the functionality of integrating any algorithm into the system. That is, the user only needs to provide the executable file and the number of required inputs for proper parametrization. Source codes developed in Scala, R, Python, or Java can be supported on LadonSpark. Besides, clustering, regression, classification, and association rules algorithms are already integrated so that users can test its usability from its initial installation.



中文翻译:

通过机器学习算法集成自动部署Spark集群

如今存储的大量数据已将大数据分析变成一个非常流行的研究领域。Spark分布式计算平台已成为集群部署和大数据分析的主要且广泛使用的范例。但是,由于所有节点必须满足要求,因此手动启动仍然是一项需要花费大量时间的任务。这项工作介绍了LadonSpark,这是一种开源的非商业解决方案,可自动配置和部署Spark集群。它经过专门设计,可通过友好的图形用户界面轻松高效地管理Spark集群,以自动化集群的部署并快速启动Hadoop的分布式文件系统。此外,LadonSpark还具有将任何算法集成到系统中的功能。那是,用户只需要提供可执行文件和适当的参数化所需的输入数量即可。LadonSpark可支持使用Scala,R,Python或Java开发的源代码。此外,已经集成了聚类,回归,分类和关联规则算法,因此用户可以从初始安装开始测试其可用性。

更新日期:2020-05-08
down
wechat
bug