The class of quaternion matrix optimization (QMO) problems, with quaternion matrices as decision variables, has been widely used in color image processing and other engineering areas in recent years. However, optimization theory for QMO is far from adequate. The main objective of this paper is to provide necessary theoretical foundations on optimality analysis, in order to enrich the contents of optimization theory and to pave way for the design of efficient numerical algorithms as well. We achieve this goal by conducting a thorough study on the first-order and second-order (sub)differentiation of real-valued functions in quaternion matrices, with a newly introduced operation called R-product as the key tool for our calculus. Combining with the classical optimization theory, we establish the first-order and the second-order optimality analysis for QMO. Particular treatments on convex functions, the \(\ell _0\)-norm and the rank function in quaternion matrices are tailored for a sparse low rank QMO model, arising from color image denoising, to establish its optimality conditions via stationarity.

一类以四元数矩阵为决策变量的四元数矩阵优化（QMO）问题，近年来在彩色图像处理等工程领域得到广泛应用。然而，QMO 的优化理论还远远不够。本文的主要目的是为优化性分析提供必要的理论基础，以丰富优化理论的内容，也为设计高效的数值算法铺平道路。我们通过对四元数矩阵中实值函数的一阶和二阶（亚）微分进行深入研究来实现这一目标，并使用新引入的称为 R 乘积的运算作为我们微积分的关键工具。结合经典优化理论，我们建立了QMO的一阶和二阶最优性分析。凸函数的特殊处理，\(\ell _0\) -norm 和四元数矩阵中的秩函数是为彩色图像去噪产生的稀疏低秩 QMO 模型量身定制的，以通过平稳性建立其最优条件。