While maximizing expected return is the goal in most reinforcement learning approaches, risk-sensitive objectives such as conditional value at risk (CVaR) are more suitable for many high-stakes applications. However, relatively little is known about how to explore to quickly learn policies with good CVaR. In this paper, we present the first algorithm for sample-efficient learning of CVaR-optimal policies in Markov decision processes based on the optimism in the face of uncertainty principle. This method relies on a novel optimistic version of the distributional Bellman operator that moves probability mass from the lower to the upper tail of the return distribution. We prove asymptotic convergence and optimism of this operator for the tabular policy evaluation case. We further demonstrate that our algorithm finds CVaR-optimal policies substantially faster than existing baselines in several simulated environments with discrete and continuous state spaces.

中文翻译：

---

乐观保守：快速学习 CVaR 策略

虽然最大化预期回报是大多数强化学习方法的目标，但风险敏感目标（如条件风险价值 (CVaR)）更适合许多高风险应用。然而，关于如何探索以快速学习具有良好 CVaR 的策略的知识相对较少。在本文中，我们基于面对不确定性原理的乐观主义，提出了第一个在马尔可夫决策过程中对 CVaR 最优策略进行样本有效学习的算法。该方法依赖于分布 Bellman 算子的新颖乐观版本，该算子将概率质量从收益分布的下尾移至上尾。我们证明了该算子在表格策略评估案例中的渐近收敛性和乐观性。