Prudent and meaningful performance evaluation of algorithms is essential for the progression of any research field. In the field of Non-Intrusive Load Monitoring (NILM), performance evaluation can be conducted on real-world aggregate signals, provided by smart energy meters or artificial superpositions of individual load signals (i.e., denoised aggregates). It has long been suspected that testing on these denoised aggregates provides better evaluation results mainly due to the fact that the signal is less complex. Complexity in real-world aggregate signals increases with the number of unknown/untracked loads. Although this is a known performance reporting problem, an investigation into the actual performance gap between real and denoised testing is still pending. In this paper, we examine the performance gap between testing on real-world and denoised aggregates with the aim of bringing clarity into this matter. Starting with an assessment of noise levels in datasets, we find significant differences in test cases. We give broad insights into our evaluation setup comprising three load disaggregation algorithms, two of them relying on neural network architectures. The results presented in this paper, based on studies covering three scenarios with ascending noise levels, show a strong tendency towards load disaggregation algorithms providing significantly better performance on denoised aggregate signals. A closer look at the outcome of our studies reveals that all appliance types could be subject to this phenomenon. We conclude the paper by discussing aspects that could be causing these considerable gaps between real and denoised testing in NILM.

中文翻译：

---

研究非侵入式负载监控中实值和去噪聚合测试之间的性能差距

对于任何研究领域的进展，对算法进行审慎而有意义的性能评估都是必不可少的。在非侵入式负载监控（NILM）领域，可以对由智能电表或各个负载信号（即去噪后的聚合）的人工叠加提供的现实世界中的聚合信号进行性能评估。长期以来人们一直怀疑对这些去噪后的集合进行测试可以提供更好的评​​估结果，这主要是由于信号的复杂程度较低。现实情况下，聚合信号的复杂性随未知/未跟踪负载的数量而增加。尽管这是一个已知的性能报告问题，但是对实际测试和去噪测试之间的实际性能差距的调查仍在进行中。在本文中，我们研究了在真实世界和去噪聚合测试之间的性能差距，目的是使这一问题变得清晰。从评估数据集中的噪声水平开始，我们发现测试用例存在显着差异。我们对评估设置提供了广泛的见解，其中包括三种负载分解算法，其中两种依赖于神经网络架构。本文提出的结果基于对三种噪声水平不断提高的情况的研究，显示了一种趋势，即负载分解算法有很强的趋势，可在去噪的聚合信号上提供明显更好的性能。仔细研究我们的研究结果可以发现，所有类型的设备都可能会遇到这种现象。