Bi-level optimization and reinforcement learning (RL) constitute the state-of-the-art frameworks for modeling strategic bidding decisions in deregulated electricity markets. However, the former neglects the market participants' physical non-convex operating characteristics, while conventional RL methods require discretization of state and/or action spaces and thus suffer from the curse of dimensionality. This paper proposes a novel deep reinforcement learning (DRL) based methodology, combining a deep deterministic policy gradient (DDPG) method with a prioritized experience replay (PER) strategy. This approach sets up the problem in multi-dimensional continuous state and action spaces, enabling market participants to receive accurate feedback regarding the impact of their bidding decisions on the market clearing outcome, and devise more profitable bidding decisions by exploiting the entire action domain, also accounting for the effect of non-convex operating characteristics. Case studies demonstrate that the proposed methodology achieves a significantly higher profit than the alternative state-of-the-art methods, and exhibits a more favourable computational performance than benchmark RL methods due to the employment of the PER strategy.

中文翻译：

---

电力市场战略招标的深度强化学习

双层优化和强化学习（RL）构成了用于在放松管制的电力市场中对战略投标决策进行建模的最新框架。但是，前者忽略了市场参与者的物理非凸操作特性，而常规的RL方法要求离散状态和/或动作空间，因此遭受了维数的诅咒。本文提出了一种新颖的基于深度强化学习（DRL）的方法，将深度确定性策略梯度（DDPG）方法与优先体验重放（PER）策略相结合。这种方法在多维连续状态和操作空间中设置了问题，使市场参与者可以收到有关其出价决策对市场清算结果的影响的准确反馈，并通过利用整个行动领域来设计更具利润的出价决策，同时还要考虑非凸面运营特征的影响。案例研究表明，由于采用了PER策略，因此与替代的最新技术相比，该方法可实现更高的利润，并且与基准RL方法相比，具有更好的计算性能。