The output signal-to-clutter-plus-noise ratio (SCNR) of space-time adaptive processing (STAP) decreases due to the dispersion of the transmit energy for traditional airborne multiple-input-multiple-output (MIMO) radar. Moreover the sufficient training samples cannot be provided to estimate the clutter covariance matrix (CCM) in the non-stationary environment. To solve these problems, a novel STAP method based on the atomic norm minimization (ANM) for transmit beamspace-based (TB-based) airborne MIMO radar is proposed. Firstly, the signal model of TB-based MIMO-STAP is established, then the optimizing principle based on the ANM is presented to design TB matrix used to focus the transmit energy in a certain spatial sector. Moreover, the beampattern corresponding to the TB matrix is close to the desired beampattern in different working modes. Meanwhile, to further reduce the number of the training samples, an accurate CCM estimation with the low-rank property is yielded by applying the ANM theory into the TB-based MIMO-STAP. Compared with the conventional MIMO-STAP with the orthogonal waveforms, simulation results show that the output SCNR of TB-based MIMO-STAP is improved. Moreover, since the CCM of the proposed method is estimated in continuous domain, the off-grid problem can be avoided. Thus, the clutter suppression performance is superior over the conventional sparse representation based STAP (SR-STAP). Meanwhile, simulation results show that the performance of the proposed method is close to that of the statistical classical MIMO-STAP and outperforms that of SR-STAP. Furthermore, it can be shown from the simulation results that the clutter suppression performance is still robust when the low-rank property of the CCM is destroyed.

由于传统机载多输入多输出（MIMO）雷达发射能量的分散，空时自适应处理（STAP）的输出信噪比（SCNR）降低。此外，不能提供足够的训练样本来估计非平稳环境中的杂波协方差矩阵（CCM）。为了解决这些问题，提出了一种基于原子范数最小化（ANM）的新型STAP方法，用于基于发射波束空间的（基于TB的）机载MIMO雷达。首先建立了基于TB的MIMO-STAP信号模型，然后提出了基于ANM的优化原理，设计了用于将发射能量集中在特定空间扇区的TB矩阵。而且，在不同的工作模式下，对应于TB矩阵的波束图接近于期望的波束图。同时，为了进一步减少训练样本的数量，通过将ANM理论应用于基于TB的MIMO-STAP中，可以得到具有低秩特性的准确CCM估计。仿真结果表明，与传统的正交波形MIMO-STAP相比，基于TB的MIMO-STAP的输出SCNR得到了改善。而且，由于所提出方法的CCM是在连续域中估计的，因此可以避免离网问题。因此，杂波抑制性能优于传统的基于稀疏表示的STAP（SR-STAP）。同时，仿真结果表明，所提方法的性能接近统计经典MIMO-STAP，性能优于SR-STAP。此外，