Skip to main content
SpringerLink
Log in

Monte Carlo Tree Search with Last-Good-Reply Policy for Cognitive Optimization of Cloud-Ready Optical Networks

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

The rapid development of Cloud Computing and Content Delivery Networks (CDNs) brings a significant increase in data transfers that leads to new optimization challenges in inter-data center networks. In this article, we focus on the cross-stratum optimization of an inter-data center Elastic Optical Network (EON). We develop an optimization approach that employs machine learning Monte Carlo Tree Search (MCTS) algorithm for the simulation of future traffic to improve the performance of the network regarding the request blocking and the operational cost. The key novelty of our approach is using various selection strategies applied to the phase of building a search tree under different network scenarios. We evaluate the performance of these selection strategies using representative topologies and real-data provided by Amazon Web Services. The main conclusion is that the approach based on the policy of Last-Good-Reply with Forgetting enables more efficient cloud resource allocation, which results in lower request blocking, thus, reduces the operational cost of the network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Wei, Y., Blake, M.B., Dame, N.: Service-oriented computing and cloud computing challenges and opportunities. IEEE Internet Comput. 10(1), 72–75 (2010)

    Article  Google Scholar 

  2. HuiYang, H., Zhao, Y., Zhang, J., Lee, Y., Lin, Y., Zhang, F.: Cross stratum optimization architecture for optical as a Service. IEEE Commun. Mag. 15, 130–140 (2015)

    Google Scholar 

  3. Yang, H., Zhao, Y., Zhang, J., Wu, J., Han, J., Lin, Y., Lee, Y., Ji, Y.: Multi-stratum resilience with resources integration for software defined data center interconnection based on IP over elastic optical networks. Eur. Conf. Optical Commun. 18, 1735–1738 (2014)

    Google Scholar 

  4. Jinno, M., Takara, H., Kozicki, B.: Concept and enabling technologies of spectrum-sliced elastic optical path network (SLICE). In: Asia communications and photonics conference and exhibition. Shanghai, China (2009)

  5. Goscien, R., Walkowiak, K., Klinkowski, M., Rak, J.: Protection in elastic optical networks. IEEE Netw. 29(6), 88–96 (2015)

    Article  Google Scholar 

  6. Kocsis, L., Szepesvári, C.: Bandit based monte-carlo planning. In: Proceedings of ECML, pp. 282–203 (2006)

  7. Saad, W., Han, Z., Debbah, M., Hjorungnes, A., Basar, T.: Coalitional game theory for communication networks: a tutorial. IEEE Signal Process. Mag. 26, 77–97 (2009)

    Article  Google Scholar 

  8. Haeri, S., Trajkovic, L.: Deflection routing in complex networks. In: Proceedings-IEEE international symposium on circuits and systems, pp. 2217–2220 (2014)

  9. Aibin, M.: Traffic prediction based on machine learning for elastic optical networks. Optical Switch. Netw. 30, 33–39 (2018)

    Article  Google Scholar 

  10. Aibin, M., Walkowiak, K.: Monte Carlo Tree Search for cross-stratum optimization of survivable inter-data center elastic optical network. In: International workshop on reliable networks design and modeling (RNDM). Spielberg, Norway (2018)

  11. Gerstel, O., Jinno, M., Lord, A., Yoo, S.J.B.: Elastic optical networking: a new dawn for the optical layer? IEEE Commun. Mag. 50(2), 12–20 (2012)

    Article  Google Scholar 

  12. Yang, H., Cheng, L., Yuan, J., Zhang, J., Zhao, Y., Lee, Y.: Multipath protection for data center services in OpenFlow-based software defined elastic optical networks. Optical Fiber Technol. 23, 108–115 (2015)

    Article  Google Scholar 

  13. Dupas, A., Dutisseuil, E., Layec, P., Jennevé, P., Frigerio, S., Yan, Y., Hugues-Salas, E., Zervas, G., Simeonidou, D.E., Bigo, S.: Real-time demonstration of software-defined elastic interface for flexgrid networks. In: Optical fiber communication conference. Washington, D.C. (2015)

  14. Philip, V.D., Gourhant, Y.: Cross-control: a scalable multi-topology fault restoration mechanism using logically centralized controllers. In: IEEE 15th international conference on high performance switching and routing (HPSR), pp. 57–63 (2014)

  15. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  16. Bell, J.: Chapter 5—Artificial neural networks. Wiley, New York (2014)

    Google Scholar 

  17. Marsland, S.: Machine learning: an algorithmic perspective. Taylor & Francis, London (2014)

    Book  Google Scholar 

  18. Yao, Q., Yang, H., Zhu, R., Yu, A., Bai, W., Tan, Y., Zhang, J., Xiao, H.: Core, mode, and spectrum assignment based on machine learning in space division multiplexing elastic Optical Networks. IEEE Access 6, 15898–15907 (2018)

    Article  Google Scholar 

  19. Chen, X., Guo, J., Zhu, Z., Proietti, R., Castro, A., Yoo, S.J.B.: Deep-RMSA: a deep-reinforcement-learning routing, modulation and deep-RMSA: a deep-reinforcement-learning routing, modulation and spectrum assignment agent for elastic optical networks (December 2017) (2018)

  20. Liu, S., Niu, B., Li, D., Wang, M., Tang, S., Kong, J., Li, B.: DL-Assisted Cross-Layer orchestration in software-defined IP-over-EONs: from algorithm design to system prototype. pp. 1–12

  21. Mata, J., de Miguel, I., Durán, R.J., Merayo, N., Singh, S.K., Jukan, A., Chamania, M.: Artificial intelligence (AI) methods in optical networks: a comprehensive survey. Optical Switch. Netw. 28, 43–57 (2018)

    Article  Google Scholar 

  22. Proietti, R., Chen, X., Zhang, K., Liu, G., Shamsabardeh, M., Castro, A., Velasco, L., Zhu, Z., Yoo, S.J.B.: Experimental demonstration of machine-learning-aided QoT estimation in multi-domain elastic optical networks with Alien Wavelengths. J. Opt. Commun. Netw. (2018). https://doi.org/10.1364/JOCN.11.0000A1

    Article  Google Scholar 

  23. Rottondi, C., Barletta, L., Giusti, A., Tornatore, M.: Machine-learning method for quality of transmission prediction of unestablished lightpaths. J. Opt. Commun. Netw. (2018). https://doi.org/10.1364/JOCN.10.00A286

    Article  Google Scholar 

  24. Morais, R.M., Pedro, J.: Machine learning models for estimating quality of transmission in DWDM networks. J. Opt. Commun. Netw. 10(2), A286–A297 (2018)

    Article  Google Scholar 

  25. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: Evolution of optical flow estimation with deep networks. In: Proceedings—30th IEEE conference on computer vision and pattern recognition, CVPR 2017 (2017)

  26. Shahkarami, S., Musumeci, F., Cugini, F., Tornatore, M.: Machine-learning-based soft-failure detection and Identification in optical networks. Optical Fiber Commun. (2018). https://doi.org/10.1364/OFC.2018.M3A.5

    Article  Google Scholar 

  27. Panayiotou, T., Chatzis, S.P., Ellinas, G.: Leveraging statistical machine learning to address failure localization in optical networks. J. Opt. Commun. Netw. 10, 344–352 (2018)

    Article  Google Scholar 

  28. Thrane, J., Wass, J., Piels, M., Diniz, J.C., Jones, R., Zibar, D.: Machine learning techniques for optical performance monitoring from directly detected PDM-QAM signals. J. Lightwave Technol. (2017). https://doi.org/10.1109/JLT.2016.2590989

    Article  Google Scholar 

  29. Khan, F.N., Lu, C., Lau, A.P.T.: Optical performance monitoring in fiber-optic networks enabled by machine learning techniques (2018)

  30. Tomkos, I.: Toward the 6G Network Era: opportunities and challenges, pp. 34–38 (2020)

  31. Wong, D., Tseng, S., Mao, H., Aibin, M.: Regenerator Placement in Survivable Optical Networks Using Deep Tensor Neural Network. In: SIGCSE ’20: The 51st ACM Technical Symposium on Computer Science Education, pp. 3–5 (2020)

  32. Mirza, M., Sommers, J., Barford, P., Zhu, X.: A machine learning approach to TCP throughput prediction. IEEE/ACM Trans. Netw. 18(4), 1026–1039 (2010)

    Article  Google Scholar 

  33. Williams, N., Zander, S., Armitage, G.: A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification. ACM SIGCOMM Comput. Commun. Rev. 36(5), 5 (2006)

    Article  Google Scholar 

  34. Zander, S., Nguyen, T., Armitage, G.: Automated traffic classification and application identification using machine learning. In: IEEE Conference on Local Computer Networks, pp. 250–257 (2005)

  35. Huang, W., Song, G., Hong, H., Xie, K.: Deep architecture for traffic flow prediction: deep belief networks with multitask learning. IEEE Trans. Intell. Transport. Syst. 15(5), 2191–2201 (2014)

    Article  Google Scholar 

  36. Najafabadi, M.M., Khoshgoftaar, T.M., Kemp, C., Seliya, N., Zuech, R.: Machine learning for detecting brute force attacks at the network level. In: Proceedings—IEEE 14th international conference on bioinformatics and bioengineering, BIBE 2014, pp. 379–385 (2014)

  37. Sommer, C.: Shortest-path queries in static networks. ACM Comput. Surv. 46(4), 1–31 (2014)

    Article  MATH  Google Scholar 

  38. Suthaharan, S.: Big Data classification: problems and challenges in network intrusion prediction with machine learning. ACM SIGMETRICS Perform. Eval. Rev. 41(4), 70–73 (2014)

    Article  Google Scholar 

  39. Wang, L., Wang, X., Tornatore, M., Kim, K.J., Kim, S.M., Kim, D.U., Han, K.E., Mukherjee, B.: Scheduling with machine-learning-based flow detection for packet-switched optical data center networks. J. Opt. Commun. Netw. (2018). https://doi.org/10.1364/JOCN.10.000365

    Article  Google Scholar 

  40. Musumeci, F., Rottondi, C., Nag, A., Macaluso, I., Zibar, D., Ruffini, M., Tornatore, M.: An overview on application of machine learning techniques in Optical Networks. In: IEEE Communications Surveys & Tutorials, pp 1–27 (2018)

  41. Lu, G.W., Sakamoto, T., Kawanishi, T.: Flexible high-order QAM transmitters for elastic optical networks. Photon. Netw. Commun. 31, 1 (2016)

    Article  Google Scholar 

  42. Aibin, M.: Dynamic routing algorithms for cloud-ready elastic optical networks. Ph.D. thesis, Wroclaw University of Science and Technology (2017)

  43. Jinno, M., Kozicki, B., Takara, H., Watanabe, A., Sone, Y., Tanaka, T., Hirano, A.: Distance-adaptive spectrum resource allocation in spectrum-sliced elastic optical path network. IEEE Commun. Mag. 48(8), 138–145 (2010)

    Article  Google Scholar 

  44. Politi, C.T., Anagnostopoulos, V., Matrakidis, C., Stavdas, A., Park, A., Heath, M., Kingdom, U.: Dynamic operation of flexi-grid OFDM-based networks. In: Optical Fiber Communication Conference and Exposition and the National Fiber Optic Engineers Conference (OFC/NFOEC). 1 (2012)

  45. Klinkowski, M., Walkowiak, K.: On the advantages of elastic optical networks for provisioning of cloud computing traffic. IEEE Netw. 27(6), 44–51 (2013)

    Article  Google Scholar 

  46. Rothenberg, C.E., Azodolmolky, S., Uhlig, S.: Software-defined networking: a comprehensive survey. Proc. IEEE 103, 14–76 (2015)

    Article  Google Scholar 

  47. Kearns, M., Mansour, Y., Ng, A.Y.: A sparse sampling algorithm for near-optimal planning in large Markov decision processes. IJCAI Int. Joint Conf. Artif. Intell. 2, 1324–1331 (1999)

    MATH  Google Scholar 

  48. Coulom, R.: Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. Computers and games 4630, 72–83 (2007)

    Article  Google Scholar 

  49. Schadd, M.P.D., Winands, M.H.M., Tak, M.J.W., Uiterwijk, J.W.H.M.: Single-player Monte-Carlo tree search for SameGame. Knowl. Based Syst. 34, 3–11 (2012)

    Article  Google Scholar 

  50. Drake, P.: The last-good-reply policy for Monte-Carlo go. ICGA J. 32(4), 221–227 (2009)

    Article  Google Scholar 

  51. Baier, H., Drake, P.D.: The power of forgetting: improving the last-good-reply policy in Monte Carlo go. IEEE Trans. Comput. Intell. AI Games 2(4), 303–309 (2010)

    Article  Google Scholar 

  52. Chaslot, G., Fiter, C., Hoock, J.B., Rimmel, A., Teytaud, O.: Adding expert knowledge and exploration in Monte-Carlo tree search. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 6048 LNCS, pp. 1–13 (2010)

  53. Aibin, M., Blazejewski, M.: Complex Elastic Optical Network Simulator (CEONS). In: 17th International Conference on Transparent Optical Networks (ICTON), pp. 1–4. Budapest, Hungary (2015)

  54. Data Center Map: DataCenterMap.com (2017). http://www.datacentermap.com

  55. Cisco: Global Cloud Index: Forecast and Methodology, 20162021 (white paper). Tech. rep., CISCO (2018)

  56. Velasco, L., Klinkowski, M., Ruiz, M., López, V., Junyent, G.: Elastic spectrum allocation for variable traffic in flexible-grid optical networks. In: Optical Fiber Communication conference and exposition and the national fiber optic engineers conference (OFC/NFOEC). Los Angeles, USA (2012)

  57. Walkowiak, K., Kasprzak, A., Klinkowski, M.: Dynamic routing of anycast and unicast traffic in Elastic Optical Networks. In: IEEE international conference on communications (ICC), pp. 3313–3318 (2014)

  58. Wang, N., Jue, J.P.: Holding-time-aware routing, modulation, and spectrum assignment for elastic optical networks. In: IEEE global communications conference, pp. 2180–2185 (2014)

  59. Zhang, L., Lu, W., Zhou, X., Zhu, Z.: Dynamic RMSA in spectrum-sliced elastic optical networks for high-throughput service provisioning. In: International conference on computing. Networking and communications (ICNC). San Diego, USA, pp. 380–384 (2013)

  60. Zhu, Z., Lu, W., Zhang, L., Ansari, N., Member, S., Lu, W., Zhang, L., Ansari, N.: Dynamic service provisioning in elastic optical networks with hybrid single-/multi-path routing. J. Lightwave Technol. 31(1), 15–22 (2013)

    Article  Google Scholar 

Download references

Funding

The work of M. Aibin was supported by the National Science Centre, Poland under Grant No. 2016/21/N/ST7/02147. The work of K. Walkowiak was supported by the National Science Centre, Poland under Grant No. 2017/27/B/ST7/00888.

Author information

Authors and Affiliations

  1. Department of Computing, British Columbia Institute of Technology, Vancouver, BC, Canada

    Michal Aibin

  2. Department of Systems and Computer Networks, Wroclaw University of Science and Technology, Wroclaw, Poland

    Krzysztof Walkowiak

Authors
  1. Michal Aibin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Krzysztof Walkowiak
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michal Aibin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aibin, M., Walkowiak, K. Monte Carlo Tree Search with Last-Good-Reply Policy for Cognitive Optimization of Cloud-Ready Optical Networks. J Netw Syst Manage 28, 1722–1744 (2020). https://doi.org/10.1007/s10922-020-09555-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10922-020-09555-8

Keywords

Search

Navigation