In this paper, a direction of arrival (DOA) estimation algorithm for coherent distributed (CD) sources considering unknown mutual coupling in massive MIMO systems is proposed. By utilizing an enhanced sample covariance matrix (SCM) that matches well with massive MIMO systems, the proposed algorithm constructs a perturbed model and further exploits the sparse total east squares (STLS) technique to achieve DOA and angular spread estimation. With the DOA and angular spread estimates, the mutual coupling coefficients are given by solving the constrained minimization problem. Compared with the other state-of-the-art algorithms, the proposed algorithm not only can provide improved DOA estimation performance, but also can conquer the deficient scenario in a much easier way. Theoretical performance analysis and numerical examples validate the effectiveness of the proposed algorithm.

本文提出了一种在大规模MIMO系统中考虑未知互耦合的相干分布式（CD）源的到达方向（DOA）估计算法。通过利用与大规模MIMO系统良好匹配的增强样本协方差矩阵（SCM），该算法构造了一个扰动模型，并进一步利用稀疏总方差（STLS）技术来实现DOA和角扩展估计。利用DOA和角度扩展估计，可以通过求解约束最小化问题来给出互耦合系数。与其他最新算法相比，所提出的算法不仅可以提供改进的DOA估计性能，而且可以以更容易的方式克服缺陷情况。