Sleep stage classification constitutes an important element of sleep disorder diagnosis. It relies on the visual inspection of polysomnography records by trained sleep technologists. Automated approaches have been designed to alleviate this resource-intensive task. However, such approaches are usually compared to a single human scorer annotation despite an inter-rater agreement of about 85% only. The present study introduces two publicly-available datasets, DOD-H including 25 healthy volunteers and DOD-O including 55 patients suffering from obstructive sleep apnea (OSA). Both datasets have been scored by 5 sleep technologists from different sleep centers. We developed a framework to compare automated approaches to a consensus of multiple human scorers. Using this framework, we benchmarked and compared the main literature approaches to a new deep learning method, SimpleSleepNet, which reach state-of-the-art performances while being more lightweight. We demonstrated that many methods can reach human-level performance on both datasets. SimpleSleepNet achieved an F1 of 89.9% vs 86.8% on average for human scorers on DOD-H, and an F1 of 88.3% vs 84.8% on DOD-O. Our study highlights that state-of-the-art automated sleep staging outperforms human scorers performance for healthy volunteers and patients suffering from OSA. Considerations could be made to use automated approaches in the clinical setting.

中文翻译：

---

Dreem开放式数据集：多得分睡眠数据集，用于比较人为和自动睡眠阶段

睡眠阶段分类是睡眠障碍诊断的重要元素。它依靠训练有素的睡眠技术人员对多导睡眠图记录进行视觉检查。已设计出自动化方法来减轻这种资源密集型任务。然而，尽管评分者之间的协议仅约85％，但通常将此类方法与单个人类评分者注释进行比较。本研究介绍了两个公开可用的数据集，包括25名健康志愿者的DOD-H和包括55名患有阻塞性睡眠呼吸暂停（OSA）的患者的DOD-O。这两个数据集均由来自不同睡眠中心的5位睡眠技术专家进行了评分。我们开发了一个框架，用于比较自动化方法与多个人类记分员的共识。使用这个框架，我们对主要文献方法与新的深度学习方法SimpleSleepNet进行了基准测试，并进行了比较，该方法不仅具有最先进的性能，而且更轻巧。我们证明了许多方法都可以在两个数据集上达到人类水平的性能。SimpleSleepNet的F1得分为89.9％，而人类得分者的平均得分为86.8％，而DOD-H的得分为88.3％vs 84.8％。我们的研究强调，对于健康志愿者和患有OSA的患者，最先进的自动睡眠分阶段系统优于人类评分者的表现。可以考虑在临床环境中使用自动化方法。在DOD-H上，人类得分者的平均得分分别为9％和86.8％，在DOD-O上的F1为88.3％和84.8％。我们的研究强调，对于健康志愿者和患有OSA的患者，最先进的自动睡眠分阶段系统优于人类评分者的表现。可以考虑在临床环境中使用自动化方法。在DOD-H上，人类得分者的平均得分分别为9％和86.8％，在DOD-O上的F1为88.3％和84.8％。我们的研究强调，对于健康志愿者和患有OSA的患者，最先进的自动睡眠分阶段系统优于人类评分者的表现。可以考虑在临床环境中使用自动化方法。