Duty cycle management is critical for the energy-neutral operation of batteryless devices. Many efforts have been made to develop an effective duty cycling method, including machine-learning-based approaches, but existing methods can barely handle the dynamic harvesting environments of batteryless devices. Specifically, most machine-learning-based methods require the harvesting patterns to be collected in advance, as well as manual configuration of the duty-cycle boundaries. In this article, we propose a configuration-free duty cycling scheme for batteryless devices, called CTRL, with which energy harvesting nodes tune the duty cycle themselves adapting to the surrounding environment without user intervention. This approach combines reinforcement learning (RL) with a control system to allow the learning algorithm to explore all possible search space automatically. The learning algorithm sets the target State of Charge (SoC) of the energy storage, instead of explicitly setting the target task frequency at a given time. The control system then satisfies the target SoC by controlling the duty cycle. An evaluation based on the real implementation of the system using publicly available trace data shows that CTRL outperforms state-of-the-art approaches, resulting in 40% less frequent power failures in energy-scarce environments while achieving more than ten times the task frequency in energy-rich environments.

中文翻译：

---

无电池应用中基于强化学习的能量管理的控制动作空间


占空比管理对于无电池设备的能源中性运行至关重要。人们已经做出了许多努力来开发有效的工作循环方法，包括基于机器学习的方法，但现有方法几乎无法处理无电池设备的动态采集环境。具体来说，大多数基于机器学习的方法需要提前收集收获模式，以及手动配置工作周期边界。在本文中，我们提出了一种用于无电池设备的免配置占空比方案，称为 CTRL，通过该方案，能量收集节点可以自行调整占空比以适应周围环境，而无需用户干预。这种方法将强化学习（RL）与控制系统相结合，使学习算法能够自动探索所有可能的搜索空间。学习算法设置能量存储的目标充电状态（SoC），而不是在给定时间明确设置目标任务频率。然后控制系统通过控制占空比来满足目标SoC。使用公开跟踪数据基于系统的实际实施进行的评估表明，CTRL 的性能优于最先进的方法，可在能源稀缺环境中将电源故障频率降低 40%，同时实现十倍以上的任务频率在能源丰富的环境中。