Skip to main content
SpringerLink
Log in

Workflow scheduling based on deep reinforcement learning in the cloud environment

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

As a convenient and economic computing model, cloud computing promotes the development of intelligence. Solving the workflow scheduling is a significant topic to promote the development of the cloud computing. In this work, an Actor-Critic architecture is utilized to solve this problem achieving the task executive time minimization under the task precedence constraint. It is similar to the list-based heuristic algorithm which includes the task prioritizing phase and task allocation phase. However, the results of the two phases interact with each other. In the task prioritizing phase, given a workflow represented as the data communication time matrix and task computation time matrix, a distribution over different task permutations by the improved Pointer network can be predicted. Then, the heuristic algorithm based on the HEFT achieves the task allocation to get the task executive time. Using negative task executive time as the reward signals, the model parameters by a policy gradient method in the first phase can be optimized. The simulation experiment is done from the task executive time, and the results shows that the workflow scheduling by the deep reinforcement learning is more effective comparing with other four single objective heuristic algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Arabnejad H, Barbosa JG (2014) List scheduling algorithm for heterogeneous systems by an optimistic cost table. IEEE Trans Parallel Distrib Syst 25(3):682–694

    Article  Google Scholar 

  • Asghari A, Sohrabi MK, Yaghmaee F (2020) Online scheduling of dependent tasks of cloud’s workflows to enhance resource utilization and reduce the makespan using multiple reinforcement learning-based agents. Soft Comput. https://doi.org/10.1007/s00500-020-04931-7

    Article  Google Scholar 

  • Barto AG, Sutton RD, Anderson CW (1970) Neuron like elements that can solve difficult learning control problems. IEEE Trans Syst Man Cybern 13(5):834–846

    Article  Google Scholar 

  • Bittencourt LF, Sakellariou R, Madeira ERM (2010) DAG scheduling using a lookahead variant of the heterogeneous earliest finish time algorithm. In: 2010 18th Euromicro conference on parallel, distributed and network-based processing, pp 27–34. https://doi.org/10.1109/PDP.2010.56

  • Bodrow W (2017) Impact of industry 4.0 in service oriented firm. Adv Manuf 5(4):394–400

    Article  Google Scholar 

  • Cai XJ, Hu ZM, Chen JJ (2020a) A many-objective optimization recommendation algorithm based on knowledge mining. Inf Sci 537:148–161

    Article  MathSciNet  Google Scholar 

  • Cai XJ, Hu ZM, Chen JJ (2020b) A many-objective optimization recommendation algorithm based on knowledge mining. Inf Sci 537:148–161

    Article  MathSciNet  Google Scholar 

  • Cai X, Geng S, Wu D, Cai J, Chen J (2020c) A multi-cloud model based many-objective intelligent algorithm for efficient task scheduling in internet of things. IEEE Internet Things J. https://doi.org/10.1109/JIOT.2020.3040019

    Article  Google Scholar 

  • Cui ZH, Xue F, Cai XJ, Cao Y, Wang GG, Chen JJ (2018) Detection of malicious code variants based on deep learning. IEEE Trans Ind Inform 14(7):3187–3196

    Article  Google Scholar 

  • Cui ZH, Zhang JJ, Wu D, Cai XJ, Wang H, Zhang WS, Chen JJ (2020) Hybrid many-objective particle swarm optimization algorithm for green coal production problem. Inf Sci 518:256–271

    Article  MathSciNet  Google Scholar 

  • Cui ZH, Xu XH, Xue F, Cai XJ, Cao Y, Zhang WS, Chen JJ (2020) Personalized recommendation system based on collaborative filtering for IoT scenarios. IEEE Trans Serv Comput 13(4):685–695

    Article  Google Scholar 

  • Dong TT, Xue F, Xiao CB, Li JT (2020) Task scheduling based on deep reinforcement learning in a cloud manufacturing environment. Concurr Comput Pract Exp 32(11):e5654

    Article  Google Scholar 

  • Dzmitry B, Kyunghyun C, Yoshua B (2015) Neural machine translation by jointly learning to align and translate. In: International conference on learning representations. arXiv:1409.0473

  • Faragardi HR, Sedghpour MRS, Faziahmadi S, Fahringer T, Rasouli N (2020) GRP-HEFT: a budget-constrained resource provisioning scheme for workflow scheduling in IaaS clouds. IEEE Trans Parallel Distrib Syst 31(6):1239–1254

    Article  Google Scholar 

  • Hassan M, Rehmani MH, Chen JJ (2020a) DEAL: differentially private auction for blockchain based microgrids energy trading. IEEE Trans Serv Comput 13(2):263–275

    Google Scholar 

  • Hassan M, Rehmani MH, Chen JJ (2020b) Differential privacy techniques for cyber physical systems: a survey. IEEE Commun Surv Tutor 22(1):746–789

    Article  Google Scholar 

  • Hopfield JJ, Tank DW (1985) Neural computation of decisions in optimization problems. Biol Cybern 52(3):141–152

    MATH  Google Scholar 

  • Irwan B, Hieu P, Quoc V L, Mohammad N, Samy B (2016) Neural combinatorial optimization with reinforcement learning. arXiv:1611.09940

  • Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682–692

    Article  Google Scholar 

  • Karpagam M, Geetha K, Rajan C (2020) A reactive search optimization algorithm for scientific workflow scheduling using clustering techniques. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-020-02480-3

    Article  Google Scholar 

  • Khan SQ, Ghani A, Khurram M (2020) Frequency-dependent synaptic plasticity model for neurocomputing applications. Int J Bioinspired Comput 16(1):56–66

    Article  Google Scholar 

  • Kumar H, Giri S (2020) Optimisation of makespan of a flow shop problem using multi layer neural network. Int J Comput Sci Math 11(2):107–122

    Article  MathSciNet  Google Scholar 

  • Liu QM (2020) Integrated deteriorating maintenance and patient scheduling for single medical device with heuristic algorithm. Int J Bioinspired Comput 16(2):121–131

    Article  Google Scholar 

  • Market Share: IT Services, Worldwide (2019). https://www.gartner.com/en/documents/3983385/market-share-it-services-worldwide-2019. id: g00717813

  • Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

    Article  Google Scholar 

  • Mohammadreza N, Afshin O, Martin T, Lawrence VS (2018) Reinforcement learning for solving the vehicle routing problem. arXiv:1802.04240

  • Nedjah N, Mourelle LD, Morais RG (2020) Inspiration-wise swarm intelligence meta-heuristics for continuous optimisation: a survey—part I. Int J Bioinspired Comput 15(4):207–223

    Article  Google Scholar 

  • Oriol V, Meire F, Navdeep J (2015) Pointer networks. In: Advances in neural information processing systems, pp 2692–2700. arXiv:1506.03134

  • Rajasekar P, Palanichamy Y (2020) Scheduling multiple scientific workflows using containers on IaaS cloud. J Ambient Intell Hum Comput. https://doi.org/10.1007/s12652-020-02483-0

    Article  Google Scholar 

  • Ronald W (1992) Simple statistical gradient following algorithms for connectionnist reinforcement learning. Mach Learn 8(3–4):229–256

    MATH  Google Scholar 

  • Sun D, Gao S, Liu XY, Li FY, Buyya R (2020) Performance-aware deployment of streaming applications in distributed stream computing systems. Int J Bioinspired Comput 15(1):52–62

    Article  Google Scholar 

  • Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge

    MATH  Google Scholar 

  • Tong Z, Chen HJ, Deng XM, Li KL, Li KQ (2019) A scheduling scheme in the cloud computing environment using deep Q-learning. Inf Sci 512:1170–1191

    Article  Google Scholar 

  • Tong Z, Deng XM, Chen HJ, Mei J, Liu H (2020) QL-HEFT: a novel machine learning scheduling scheme base on cloud computing environment. Neural Comput Appl 32(10):5553–5570

    Article  Google Scholar 

  • Topcuoglu H, Hariri S, Wu MY (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans Parallel Distrib Syst 13(3):260–274

    Article  Google Scholar 

  • Wang PH, Huang JR, Cui ZH, Xie LP, Chen JJ (2020) A Gaussian error correction multi-objective positioning model with NSGA-II. Concurr Comput Pract Exp 32(5):e5464

    Google Scholar 

  • Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8(3–4):279–292

    MATH  Google Scholar 

  • Yuan D, Yang Y, Liu X, Chen JJ (2010) A data placement strategy in scientific cloud workflows. Future Gener Comput Syst 26(8):1200–1214

    Article  Google Scholar 

  • Zhang X, Li XT, Yin MH (2020) An enhanced genetic algorithm for the distributed assembly permutation flowshop scheduling problem. Int J Bioinspired Comput 15(2):113–124

    Article  Google Scholar 

Download references

Acknowledgements

This paper is supported by Humanity and Social Science Research of Ministry of Education (20YJCZH200), Beijing Intelligent Logistics System Collaborative Innovation Center Open Topic (No. BILSCIC-2019KF- 05), Grass-roots Academic Team Building Project of Beijing Wuzi University (No. 2019XJJCTD04).

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, China

    Tingting Dong, Chuangbai Xiao & Jiangjiang Zhang

  2. School of Information, Beijing Wuzi University, Beijing, China

    Fei Xue

Authors
  1. Tingting Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fei Xue
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chuangbai Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiangjiang Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fei Xue.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dong, T., Xue, F., Xiao, C. et al. Workflow scheduling based on deep reinforcement learning in the cloud environment. J Ambient Intell Human Comput 12, 10823–10835 (2021). https://doi.org/10.1007/s12652-020-02884-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-020-02884-1

Keywords

Search

Navigation