Abstract
We expose the state of the art in the topic of one-way delay measurement in both traditional and software-defined networks. A representative range of standard mechanisms and recent research works, including OpenFlow and Programming Protocol-independent Packet Processors (P4)-based schemes, are covered. We classify them, discuss their advantages and drawbacks, and compare them according to their application environment, accuracy, cost, and robustness. The discussion extends to the reuse of traditional schemes in software-defined networks and the benefits and limitations of the latter with respect to reducing the overhead of network wide measurements. We conclude with a summary of learned lessons and open challenges for future work.
- Ian F. Akyildiz, Ahyoung Lee, Pu Wang, Min Luo, and Wu Chou. 2016. Research challenges for traffic engineering in software defined networks. IEEE Network 30, 3 (2016), 52–58.Google ScholarDigital Library
- Rasool Al-Saadi, Grenville Armitage, Jason But, and Philip Branch. 2019. A survey of delay-based and hybrid TCP congestion control algorithms. IEEE Communications Surveys & Tutorials 21, 4 (2019), 3609–3638.Google ScholarDigital Library
- Guy Almes, Sunil Kalidindi, Matthew Zekauskas, and Al Morton. 2016. A One-Way Delay Metric for IP Performance Metrics (IPPM). RFC 7679. IETF. https://doi.org/10.17487/RFC7679Google ScholarDigital Library
- V. Altukhov and E. Chemeritskiy. 2014. On real-time delay monitoring in software-defined networks. In Proceedings of IEEE MoNeTeC 2014. 1–6.Google Scholar
- Brice Augustin, Timur Friedman, and Renata Teixeira. 2010. Measuring multipath routing in the Internet. IEEE/ACM Transactions on Networking 19, 3 (2010), 830–840.Google ScholarDigital Library
- Francois Baccelli, Sridhar Machiraju, Darryl Veitch, and Jean C. Bolot. 2007. On optimal probing for delay and loss measurement. In Proceedings of ACM IMC 2007. 291–302.Google Scholar
- Vaibhav Bajpai and Jürgen Schönwälder. 2015. A survey on Internet performance measurement platforms and related standardization efforts. IEEE Communications Surveys & Tutorials 17, 3 (2015), 1313–1341.Google ScholarDigital Library
- Ran Ben Basat, Sivaramakrishnan Ramanathan, Yuliang Li, Gianni Antichi, Minian Yu, and Michael Mitzenmacher. 2020. PINT: Probabilistic in-band network telemetry. In Proceedings of the ACM SIGCOMM Conference. 662–680.Google ScholarDigital Library
- Pankaj Berde, Matteo Gerola, Jonathan Hart, Yuta Higuchi, Masayoshi Kobayashi, Toshio Koide, Bob Lantz, et al. 2014. ONOS: Towards an open, distributed SDN OS. In Proceedings of ACM HotSDN 2014.Google ScholarDigital Library
- Martin Bjorklund. 2010. YANG—A Data Modeling Language for the Network Configuration Protocol (NETCONF). RFC 6020. IETF. https://doi.org/10.17487/RFC6020Google ScholarDigital Library
- Burton H. Bloom. 1970. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM 13, 7 (1970), 422–426.Google ScholarDigital Library
- Jean-Chrysotome Bolot. 1993. End-to-end packet delay and loss behavior in the Internet. In Proceedings of the ACM SIGCOMM Conference. 289–298.Google ScholarDigital Library
- Pat Bosshart, Dan Daly, Glen Gibb, Martin Izzard, Nick McKeown, Jennifer Rexford, Cole Schlesinger, et al. 2014. P4: Programming protocol-independent packet processors. ACM SIGCOMM Computer Communication Review 44, 3 (2014), 87–95.Google ScholarDigital Library
- C. J. Bovy, H. T. Mertodimedjo, Gerard Hooghiemstra, Henk Uijterwaal, and Piet Van Mieghem. 2002. Analysis of end-to-end delay measurements in Internet. In Proceedings of the PAM Conference.Google Scholar
- Djalel Chefrour. 2021. “Evolution of Network Time Synchronization Towards Nanoseconds Accuracy: A Survey.” Manuscript submitted for publication.Google Scholar
- Stuart Cheshire. 1996. Latency and the quest for interactivity. In White Paper commissioned by Volpe Welty Asset Management, LLC, for the Synchronous Person-to-Person Interactive Computing Environments Meeting.Google Scholar
- Jin-Hee Choi and Chuck Yoo. 2005. One-way delay estimation and its application. Computer Communications 28, 7 (2005), 819–828.Google ScholarDigital Library
- Cisco. 2012. Introduction to Cisco IOS NetFlow—A Technical Overview. White Paper (last updated May 2012). Cisco.Google Scholar
- Cisco. 2018. IP SLAs Overview. Retrieved January 05, 2021 from https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/ipsla/configuration/15-s/sla-15-s-book/sla_overview.html.Google Scholar
- Alexander Clemm and Eric Voit. 2019. Subscription to YANG Notifications for Datastore Updates. RFC 8641. IETF. https://doi.org/10.17487/RFC8641Google ScholarDigital Library
- Corvil. 2020. Corvil Appliances. Retrieved June 30, 2020 from https://www.pico.net/corvil-analytics/corvil-classic/corvil-appliances.Google Scholar
- Huynh Tu Dang, Han Wang, Theo Jepsen, Gordon Brebner, Changhoon Kim, Jennifer Rexford, Robert Soulé, and Hakim Weatherspoon. 2017. Whippersnapper: A P4 language benchmark suite. In Proceedings of ACM SOSR 2017. 95–101.Google ScholarDigital Library
- Luca De Vito, Sergio Rapuano, and Laura Tomaciello. 2008. One-way delay measurement: State of the art. IEEE Transactions on Instrumentation and Measurement 57, 12 (2008), 2742–2750.Google ScholarCross Ref
- Stephen Donnelly, Ian Graham, and René Wilhelm. 2001. Passive calibration of an active measurement system. In Proceedings of the PAM Conference. 1–8.Google Scholar
- John Eidson and Kang Lee. 2002. IEEE 1588 standard for a precision clock synchronization protocol for networked measurement and control systems. In Proceedings of the 2nd ISA/IEEE Sensors for Industry Conference, Vol. 10. 98–105.Google ScholarCross Ref
- Rob Enns, Martin Bjorklund, Andy Bierman, and J. Schoenwaelder. 2011. Network Configuration Protocol (NETCONF). RFC 6241. IETF. https://doi.org/10.17487/RFC6241Google ScholarDigital Library
- Nick Feamster, Jennifer Rexford, and Ellen Zegura. 2013. The road to SDN. ACM Queue 11, 12 (2013), 20–40.Google ScholarDigital Library
- Yu Feng, Yue Pan, Bo Zhu, Yuchuan Deng, Jing Wu, and Hao Jiang. 2019. A new framework for network flow queuing delay prediction based on stream computing. In Proceedings of the IEEE 5th BigDataSecurity, HPSC, and IDS Conference. 212–217.Google ScholarCross Ref
- Paolo Ferrari, Alessandra Flammini, Emiliano Sisinni, Stefano Rinaldi, Dennis Brandão, and Murilo Silveira Rocha. 2018. Delay estimation of Industrial IoT applications based on messaging protocols. IEEE Transactions on Instrumentation and Measurement 67 (2018), 2188–2199.Google ScholarCross Ref
- Giuseppe Fioccola, Alessandro Capello, Mauro Cociglio, Luca Castaldelli, Mach Chen, Lianshu Zheng, Greg Mirsky, and Tal Mizrahi. 2018. Alternate-Marking Method for Passive and Hybrid Performance Monitoring. RFC 8321. IETF. https://doi.org/10.17487/RFC8321Google ScholarDigital Library
- Jim Gettys and Kathleen Nichols. 2011. Bufferbloat: Dark buffers in the Internet. ACM Queue 9, 11 (2011), 40–54.Google ScholarDigital Library
- Ian D. Graham, Stephen Donnelly, Julie Martin, and John G. Cleary. 1998. Nonintrusive and accurate measurement of unidirectional delay and delay variation on the Internet. In Proceedings of INET 1998.Google Scholar
- Omer Gurewitz, Israel Cidon, and Moshe Sidi. 2006. One-way delay estimation using network-wide measurements. IEEE Transactions on Information Theory 52, 6 (2006), 2710–2724.Google ScholarCross Ref
- Taimur Hafeez, Nadeem Ahmed, Bilal Ahmed, and Asad Waqar Malik. 2017. Detection and mitigation of congestion in SDN enabled data center networks: A survey. IEEE Access 6 (2017), 1730–1740.Google ScholarCross Ref
- Hasanin Harkous, Michael Jarschel, Mu He, Rastin Pries, and Wolfgang Kellerer. 2020. P8: P4 with predictable packet processing performance. IEEE Transactions on Network and Service Management. Early access, October 12, 2020.Google ScholarCross Ref
- Stephen Hemminger. 2005. Network emulation with NetEm. In Proceedings of LCA 2005.Google Scholar
- Christian Henke, Carsten Schmoll, and Tanja Zseby. 2008. Empirical evaluation of hash functions for multipoint measurements. ACM SIGCOMM Computer Communication Review 38, 3 (2008), 39–50.Google ScholarDigital Library
- Mukesh Hira and L. J. Wobker. 2015. Improving Network Monitoring and Management with Programmable Data Planes. Retrieved June 28, 2021 from https://opennetworking.org/news-and-events/blog/improving-network-monitoring-and-management-with-programmable-data-planes/.Google Scholar
- Internet2. 2016. One-Way Ping (OWAMP). Retrieved June 12, 2020 from http://software.internet2.edu/owamp/.Google Scholar
- Manar Jammal, Taranpreet Singh, Abdallah Shami, Rasool Asal, and Yiming Li. 2014. Software defined networking: State of the art and research challenges. Computer Networks 72 (2014), 74–98.Google ScholarCross Ref
- Sunil Kalidindi, Matthew Zekauskas, and Guy Almes. 1999. A One-Way Delay Metric for IPPM. RFC 2679. IETF. https://doi.org/10.17487/RFC2679Google ScholarDigital Library
- Sunil Kalidindi and Matthew J. Zekauskas. 1999. Surveyor: An infrastructure for internet performance measurements. In Proceedings of INET 1999.Google Scholar
- Murat Karakus and Arjan Durresi. 2017. Quality of Service (QoS) in software defined networking (SDN): A survey. Journal of Network and Computer Applications 80 (2017), 200–218.Google ScholarDigital Library
- Jochen Kögel. 2011. One-way delay measurement based on flow data: Quantification and compensation of errors by exporter profiling. In Proceedings of IEEE ICOIN 2011. 25–30.Google ScholarCross Ref
- Ramana Kompella, Kirill Levchenko, Alex C. Snoeren, and George Varghese. 2009. Every microsecond counts: Tracking fine-grain latencies with a lossy difference aggregator. ACM SIGCOMM Computer Communication Review 39, 4 (2009), 255–266.Google ScholarDigital Library
- Diego Kreutz, Fernando M. V. Ramos, Paulo Esteves Veríssimo, Christian Esteve Rothenberg, Siamak Azodolmolky, and Steve Uhlig. 2014. Software-defined networking: A comprehensive survey. Proceedings of the IEEE 103, 1 (2014), 14–76.Google ScholarCross Ref
- Gautam Kumar, Nandita Dukkipati, Keon Jang, Hassan M. G. Wassel, Xian Wu, Behnam Montazeri, Yaogong Wang, et al. 2020. Swift: Delay is simple and effective for congestion control in the datacenter. In Proceedings of the ACM SIGCOMM Conference. 514–528.Google ScholarDigital Library
- James F. Kurose and Keith W. Ross. 2012. End-to-end delay. In Computer Networking: A Top-Down Approach (6th ed.). Pearson, 42–43.Google Scholar
- Myungjin Lee, Nick Duffield, and Ramana Ella. 2012. MAPLE: A scalable architecture for maintaining packet latency measurements. In Proceedings of ACM IMC2012. 101–114.Google ScholarDigital Library
- Myungjin Lee, Nick Duffield, and Ramana Kompella. 2010. Not all microseconds are equal: Fine-grained per-flow measurements with reference latency interpolation. In Proceedings of the ACM SIGCOMM Conference. 27–38.Google ScholarDigital Library
- Myungjin Lee, Nick Duffield, and Ramana Kompella. 2010. Two samples are enough: Opportunistic flow-level latency estimation using netflow. In Proceedings of IEEE INFOCOM 2010. 1–9.Google ScholarCross Ref
- Myungjin Lee, Sharon Goldberg, Ramana Kompella, and George Varghese. 2011. Fine-grained latency and loss measurements in the presence of reordering. In Proceedings of the ACM SIGMETRICS Conference. 329–340.Google ScholarDigital Library
- Young Lee, Ricard Vilalta, Ramon Casellas, Ricardo Martínez, and Raul Muñoz. 2017. Scalable telemetry and network autonomics in ACTN SDN controller hierarchy. In Proceedings of IEEE ICTON 2017. 1–4.Google ScholarCross Ref
- Lingxia Liao and Victor C. M. Leung. 2016. LLDP based link latency monitoring in software defined networks. In Proceedings of IEEE CNSM 2016. 330–335.Google Scholar
- Lingxia Liao, Victor C. M. Leung, and Min Chen. 2018. An efficient and accurate link latency monitoring method for low-latency software-defined networks. IEEE Transactions on Instrumentation and Measurement 68, 2 (2018), 377–391.Google ScholarCross Ref
- László Lovász. 1993. Random walks on graphs: A survey. Combinatorics, Paul Erdos Is Eighty 2, 1 (1993), 1–46.Google Scholar
- Wei-Zhou Lu, Wei-Xuan Gu, and Shun-Zheng Yu. 2009. One-way queuing delay measurement and its application on detecting DDoS attack. Journal of Network and Computer Applications 32, 2 (2009), 367–376.Google ScholarDigital Library
- Jan Medved, Robert Varga, Anton Tkacik, and Ken Gray. 2014. OpenDaylight: Towards a model-driven SDN controller architecture. In Proceedings of IEEE WoWMoM 2014. 1–6.Google ScholarCross Ref
- Mininet Team. 2018. Mininet: An Instant Virtual Network on Your Laptop (or Other PC). Retrieved July 14, 2020 from http://mininet.org/.Google Scholar
- Tal Mizrahi and Yoram Moses. 2016. The Case for Data Plane Timestamping in SDN. Technical Report. Technion. http://arxiv.org/pdf/1602.03342v1.Google Scholar
- Tal Mizrahi and Yoram Moses. 2016. Time Capability in NETCONF. RFC 7758. IETF. https://doi.org/10.17487/RFC7758Google ScholarDigital Library
- Maurizio Molina, Fredric Raspall, Saverio Niccolini, Nick Duffield, and Tanja Zseby. 2009. Sampling and Filtering Techniques for IP Packet Selection. RFC 5475. IETF. https://doi.org/10.17487/RFC5475Google ScholarDigital Library
- Srinivas Narayana, Anirudh Sivaraman, Vikram Nathan, Prateesh Goyal, Venkat Arun, Mohammad Alizadeh, Vimalkumar Jeyakumar, and Changhoon Kim. 2017. Language-directed hardware design for network performance monitoring. In Proceedings of the ACM SIGCOMM Conference. 85–98.Google ScholarDigital Library
- Huu-Nghi Nguyen, Thomas Begin, Anthony Busson, and Isabelle Guérin Lassous. 2016. Evaluation of an end-to-end delay estimation in the case of multiple flows in SDN networks. In Proceedings of IEEE CNSM 2016. 336–341.Google ScholarCross Ref
- Kathleen Nichols and Van Jacobson. 2012. Controlling queue delay. Communications of the ACM 55, 7 (2012), 42–50.Google ScholarDigital Library
- Anne-Cécile Orgerie, Paulo Gonçalves, Matthieu Imbert, Julien Ridoux, and Darryl Veitch. 2012. Survey of network metrology platforms. In Proceedings of IEEE/IPSJ SAINT 2012. 220–225.Google ScholarDigital Library
- Abhinav Pathak, Himabindu Pucha, Ying Zhang, Y. Charlie Hu, and Z. Morley Mao. 2008. A measurement study of internet delay asymmetry. In Proceedings of the PAM Conference. 182–191.Google Scholar
- Kevin Phemius and Mathieu Bouet. 2013. Monitoring latency with OpenFlow. In Proceedings of IEEE CNSM 2013. 122–125.Google ScholarCross Ref
- Sivaramakrishnan Ramanathan, Yaron Kanza, and Balachander Krishnamurthy. 2018. SDProber: A software defined prober for SDN. In Proceedings of ACM SOSR 2018. 1–7.Google ScholarDigital Library
- Alon Riesenberg, Yonnie Kirzon, Michael Bunin, Elad Galili, Gidi Navon, and Tal Mizrahi. 2019. Time-multiplexed parsing in marking-based network telemetry. In Proceedings of ACM SYSTOR 2019. 80–85.Google ScholarDigital Library
- Elisa Rojas, Roberto Doriguzzi-Corin, Sergio Tamurejo, Andres Beato, Arne Schwabe, Kevin Phemius, and Carmen Guerrero. 2018. Are we ready to drive software-defined networks? A comprehensive survey on management tools and techniques. ACM Computing Surveys 51, 2 (2018), 1–35.Google ScholarDigital Library
- Charalampos Rotsos, Nadi Sarrar, Steve Uhlig, Rob Sherwood, and Andrew W. Moore. 2012. OFLOPS: An open framework for OpenFlow switch evaluation. In Proceedings of the PAM Conference. 85–95.Google Scholar
- Josep Sanjuàs-Cuxart, Pere Barlet-Ros, Nick G. Duffield, and Ramana Kompella. 2011. Sketching the delay: Tracking Temporally uncorrelated flow-level latencies. In Proceedings of ACM IMC 2011.Google ScholarDigital Library
- Yaron Schwartz, Yuval Shavitt, and Udi Weinsberg. 2010. A measurement study of the origins of end-to-end delay variations. In Proceedings of the PAM Conference. 21–30.Google ScholarCross Ref
- Muhammad Shahzad and Alex X. Liu. 2014. Noise can help: Accurate and efficient per-flow latency measurement without packet probing and time stamping. ACM SIGMETRICS Performance Evaluation Review 42, 1 (2014), 207–219.Google ScholarDigital Library
- Stanislav Shalunov, Greg Hazel, Jana Iyengar, and Mirja Kuehlewind. 2012. Low Extra Delay Background Transport (LEDBAT). RFC 6817. IETF. https://doi.org/10.17487/RFC6817Google ScholarDigital Library
- Minsu Shin, Mankyu Park, Deockgil Oh, Byungchul Kim, and Jaeyong Lee. 2011. Clock synchronization for one-way delay measurement: A survey. In Proceedings of the International Conference on Advanced Communication and Networking. 1–10.Google ScholarCross Ref
- Zhaogang Shu, Jiafu Wan, Jiaxiang Lin, Shiyong Wang, Di Li, Seungmin Rho, and Changcai Yang. 2016. Traffic engineering in software-defined networking: Measurement and management. IEEE Access 4 (2016), 3246–3256.Google Scholar
- Debanshu Sinha, K. Haribabu, and Sundar Balasubramaniam. 2015. Real-time monitoring of network latency in software defined networks. In Proceedings of the IEEE ANTS Conference. 1–3.Google ScholarCross Ref
- Peerapon Siripongwutikorn and Sujata Banerjee. 2002. Per-flow delay performance in traffic aggregates. In Proceedings of IEEE GLOBECOM 2002, Vol. 3. 2634–2638.Google ScholarCross Ref
- Joel Sommers, Paul Barford, Nick Duffield, and Amos Ron. 2007. Accurate and efficient SLA compliance monitoring. In Proceedings of the ACM SIGCOMM Conference. 109–120.Google ScholarDigital Library
- Philipp Svoboda, Markus Laner, Joachim Fabini, Markus Rupp, and Fabio Ricciato. 2012. Packet delay measurements in reactive IP networks. IEEE Instrumentation & Measurement Magazine 15, 6 (2012), 36–44.Google ScholarCross Ref
- The Open Networking Foundation. 2015. OpenFlow Switch Specification, Version 1.5.1. Retrieved July 2, 2020 from https://www.opennetworking.org/wp-content/uploads/2014/10/openflow-switch-v1.5.1.pdf.Google Scholar
- The OpenConfig Operators Group. 2015. OpenConfig Repository. Retrieved January 8, 2021 from https://github.com/openconfig/public.Google Scholar
- The OpenConfig Operators Group. 2018. gNMI—gRPC Network Management Interface, Version: 0.6.0. Retrieved January 8, 2021 from https://github.com/openconfig/reference/tree/master/rpc/gnmi.Google Scholar
- Amin Tootoonchian, Sergey Gorbunov, Yashar Ganjali, Martin Casado, and Rob Sherwood. 2012. On controller performance in software-defined networks. In Proceedings of USENIX Hot-ICE 2012.Google Scholar
- Pang-Wei Tsai, Chun-Wei Tsai, Chia-Wei Hsu, and Chu-Sing Yang. 2018. Network monitoring in software-defined networking: A review. IEEE Systems Journal 12, 4 (2018), 3958–3969.Google ScholarCross Ref
- Ahmad Vakili and Jean-Charles Gregoire. 2012. Accurate one-way delay estimation: Limitations and improvements. IEEE Transactions on Instrumentation and Measurement 61, 9 (2012), 2428–2435.Google ScholarCross Ref
- Niels L. M. Van Adrichem, Christian Doerr, and Fernando A. Kuipers. 2014. OpenNetNon: Network monitoring in openflow software-defined networks. In Proceedings of IEEE NOMS. 1–8.Google Scholar
- Darryl Veitch, Julien Ridoux, and Satish Babu Korada. 2008. Robust synchronization of absolute and difference clocks over networks. IEEE/ACM Transactions on Networking 17, 2 (2008), 417–430.Google ScholarDigital Library
- Junfeng Wang, Mingtian Zhou, and Yuxia Li. 2004. Survey on the end-to-end Internet delay measurements. In Proceedings of HSNMC 2004. 155–166.Google ScholarCross Ref
- Abdulsalam Yassine, Hesam Rahimi, and Shervin Shirmohammadi. 2015. Software defined network traffic measurement: Current trends and challenges. IEEE Instrumentation & Measurement Magazine 18, 2 (2015), 42–50.Google ScholarCross Ref
- Curtis Yu, Cristian Lumezanu, Abhishek Sharma, Qiang Xu, Guofei Jiang, and Harsha V. Madhyastha. 2015. Software-defined latency monitoring in data center networks. In Proceedings of the PAM Conference. 360–372.Google Scholar
- Matthew Zekauskas, Anatoly Karp, Stanislav Shalunov, Jeff Boote, and Benjamin Teitelbaum. 2006. A One-Way Active Measurement Protocol (OWAMP). RFC 4656. IETF. https://doi.org/10.17487/RFC4656Google ScholarDigital Library
- Ting Zhang and Bin Liu. 2019. Exposing end-to-end delay in software-defined networking. International Journal of Reconfigurable Computing 2019 (2019), 7363901.Google ScholarDigital Library
- Yusu Zhao, Pengfei Zhang, and Yaohui Jin. 2016. Netography: Troubleshoot your network with packet behavior in SDN. In Proceedings of IEEE NOMS 2016. 878–882.Google ScholarDigital Library
- Tanja Zseby and Florian Schreiner. 2002. QoS Monitoring and Measurement Benchmarking. Research Report. Fraunhofer FOKUS.Google Scholar
- Tanja Zseby, Sebastian Zander, and Georg Carle. 2001. Evaluation of building blocks for passive one-way-delay measurements. In Proceedings of the PAM Conference.Google Scholar
Index Terms
- One-Way Delay Measurement From Traditional Networks to SDN: A Survey
Recommendations
Available bandwidth measurement in software defined networks
SAC '16: Proceedings of the 31st Annual ACM Symposium on Applied ComputingSoftware Defined Networking (SDN) is an emerging paradigm that is expected to revolutionize computer networks. With the decoupling of data and control plane and the introduction of open communication interfaces between layers, SDN enables ...
Measuring one-way delay in wireless mesh networks: an experimental investigation
PM2HW2N '09: Proceedings of the 4th ACM workshop on Performance monitoring and measurement of heterogeneous wireless and wired networksWireless Mesh networks are multi-hop networks mostly based on IEEE 802.11 technology and are considered as a viable alternative for providing broadband wireless Internet access. As a consequence, they require support for Quality of Service or advanced ...
Efficient topology discovery in OpenFlow-based Software Defined Networks
Software Defined Networking (SDN) is a new networking paradigm, with a great potential to increase network efficiency, ease the complexity of network control and management, and accelerate the rate of technology innovation. One of the core concepts of ...
Comments