Rate-distortion optimization (RDO) is conventionally based on the analysis of rate-distortion (R-D) curve to minimize the coding distortion under the coding bits constraint. However, it is necessary to consider the computational complexity in the RDO process. In this paper, we obtain the Confidence LEvel - Computational complexity (CLEC) curves which indicate the characteristics of coding tree units (CTUs). Based on the CLEC curves, a rate-distortion-complexity optimization (RDCO) algorithm is proposed to optimize R-D under given computational complexity and achieve the optimal coding performance for x265. Experimental results demonstrate that the proposed algorithm can achieve a wide range of encoding speed under a given quantization parameter (QP) whereas the original x265 can only achieve a few fixed encoding speeds, and the proposed algorithm can reduce the BD-rate and increase the BD-PSNR by 6.59% and 0.13 dB on average under the same requirements of encoding speeds as the original x265.

速率失真优化（RDO）通常是基于速率失真（RD）曲线的分析，以最大程度地减少编码位约束下的编码失真。但是，有必要考虑RDO过程中的计算复杂性。在本文中，我们获得了置信水平-计算复杂度（CLEC）曲线，该曲线表明了编码树单元（CTU）的特征。在CLEC曲线的基础上，提出了一种率失真复杂度优化（RDCO）算法，在给定的计算复杂度下优化RD，实现了x265的最优编码性能。实验结果表明，该算法在给定的量化参数（QP）下可以实现较宽的编码速度，而原始的x265仅能实现几种固定的编码速度，