With the increasing trend in softwarization and virtualization of network functions and systems, NFV management and orchestration (MANO) solutions are being developed to meet the agile and flexible management requirements of virtualized network services in the 5G era and beyond. In this regard, ETSI ISG NFV has specified a standard MANO system that is used as a reference by vendors as well as open-source MANO projects. These MANO systems are inherently very complex and have a direct impact on the overall performance of NFV systems. However, unlike traditional networking functions and systems, there are no well-defined test methods and KPIs based on which the performance of the NFV-MANO system can be tested, validated and benchmarked. Given the absence of formal MANO specific evaluation techniques based on which the performance and features of a MANO system can be quantified, and compared against, we introduce in this paper a formal benchmarking methodology and KPIs for MANO systems. For illustration purposes, we analyze and compare the performance of the two most popular open-source NFV MANO projects, namely ONAP and OSM, using a complex open-source virtual customer premises equipment (vCPE) VNF. Our results show the current features support, performance to be expected and gaps to be covered in future releases.

随着网络功能和系统的软化和虚拟化趋势的发展，正在开发NFV管理和编排（MANO）解决方案，以满足5G时代及以后时代对虚拟化网络服务的敏捷和灵活管理要求。在这方面，ETSI ISG NFV已指定了标准MANO系统，供卖方以及开源MANO项目用作参考。这些MANO系统本质上非常复杂，并且直接影响NFV系统的整体性能。但是，与传统的网络功能和系统不同，没有定义明确的测试方法和KPI，可以基于这些方法和KPI对NFV-MANO系统的性能进行测试，验证和基准测试。鉴于缺乏正式的MANO专用评估技术，可以基于该技术来量化MANO系统的性能和功能，并与之进行比较，在本文中，我们介绍了用于MANO系统的正式基准测试方法和KPI。出于说明目的，我们使用复杂的开源虚拟客户驻地设备（vCPE）VNF分析并比较了两个最受欢迎的开源NFV MANO项目（即ONAP和OSM）的性能。我们的结果表明，当前的功能支持，预期的性能以及将来的版本中需要弥补的差距。即ONAP和OSM，使用复杂的开源虚拟客户驻地设备（vCPE）VNF。我们的结果表明，当前的功能支持，预期的性能以及将来的版本中需要弥补的差距。即ONAP和OSM，使用复杂的开源虚拟客户驻地设备（vCPE）VNF。我们的结果表明，当前的功能支持，预期的性能以及将来的版本中需要弥补的差距。