Prudent and meaningful performance evaluation of algorithms is essential for the progression of any research field. In the field of Non-Intrusive Load Monitoring (NILM), performance evaluation can be conducted on real-world aggregate signals, provided by smart energy meters or artificial superpositions of individual load signals (i.e., denoised aggregates). It has long been suspected that testing on these denoised aggregates provides better evaluation results mainly due to the the fact that the signal is less complex. Complexity in real-world aggregate signals increases with the number of unknown/untracked load. Although this is a known performance reporting problem, an investigation in the actual performance gap between real and denoised testing is still pending. In this paper, we examine the performance gap between testing on real-world and denoised aggregates with the aim of bringing clarity into this matter. Starting with an assessment of noise levels in datasets, we find significant differences in test cases. We give broad insights into our evaluation setup comprising three load disaggregation algorithms, two of them relying on neural network architectures. The results presented in this paper, based on studies covering three scenarios with ascending noise levels, show a strong tendency towards load disaggregation algorithms providing significantly better performance on denoised aggregate signals. A closer look into the outcome of our studies reveals that all appliance types could be subject to this phenomenon. We conclude the paper by discussing aspects that could be causing these considerable gaps between real and denoised testing in NILM.

中文翻译：

---

调查非侵入式负载监控中真实和去噪聚合测试之间的性能差距

对算法进行审慎而有意义的性能评估对于任何研究领域的进展都是必不可少的。在非侵入式负载监控 (NILM) 领域，可以对由智能电表或人工叠加单个负载信号（即去噪聚合）提供的真实世界的聚合信号进行性能评估。长期以来，人们一直怀疑对这些去噪聚合进行测试会提供更好的评​​估结果，这主要是因为信号不太复杂。真实世界聚合信号的复杂性随着未知/未跟踪负载的数量而增加。尽管这是一个已知的性能报告问题，但对真实测试和去噪测试之间的实际性能差距的调查仍在进行中。在本文中，我们研究了真实世界和去噪聚合测试之间的性能差距，目的是让这个问题变得清晰。从评估数据集中的噪声水平开始，我们发现测试用例存在显着差异。我们对包含三种负载分解算法的评估设置提供了广泛的见解，其中两种算法依赖于神经网络架构。本文中提出的结果基于涵盖噪声水平上升的三个场景的研究，表明负载分解算法的强烈趋势在去噪聚合信号上提供显着更好的性能。仔细研究我们的研究结果表明，所有电器类型都可能受到这种现象的影响。