Background and objective: Owing to the variable shapes, large size difference, uneven grayscale and dense distribution among biological cells in an image, it is still a challenging task for the standard Mask R-CNN to accurately detect and segment cells. Especially, the state-of-the-art anchor-based methods fail to generate the anchors of sufficient scales effectively according to the various sizes and shapes of cells, thereby hardly covering all scales of cells.

Methods: We propose an adaptive approach to learn the anchor shape priors from data samples, and the aspect ratios and the number of anchor boxes can be dynamically adjusted by using ISODATA clustering algorithm instead of human prior knowledge in this work. To solve the identification difficulties for small objects owing to the multiple down-samplings in a deep learning-based method, a densification strategy of candidate anchors is presented to enhance the effects of identifying tinny size cells. Finally, a series of comparative experiments are conducted on various datasets to select appropriate a network structure and verify the effectiveness of the proposed methods.

Results: The results show that the ResNet-50-FPN combining the ISODATA method and densification strategy can obtain better performance than other methods in multiple metrics (including AP, Precision, Recall, Dice and PQ) for various biological cell datasets, such as U373, GoTW1, SIM+ and T24.

Conclusions: Our adaptive algorithm could effectively learn the anchor shape priors from the various sizes and shapes of cells. It is promising and encouraging for a real-world anchor-based detection and segmentation application of biomedical engineering in the future.

背景与目的：由于图像中生物细胞形状多变、尺寸差异大、灰度不均匀且分布密集，标准Mask R-CNN准确检测和分割细胞仍然是一项具有挑战性的任务。尤其是目前最先进的基于anchor的方法无法根据细胞的各种大小和形状有效地生成足够尺度的anchors，从而难以覆盖所有尺度的细胞。

方法：我们提出了一种自适应方法来从数据样本中学习锚形状先验，并且在这项工作中可以通过使用 ISODATA 聚类算法而不是人类先验知识来动态调整长宽比和锚框的数量。为了解决基于深度学习的方法中由于多次下采样而导致的小物体识别困难，提出了候选锚点的致密化策略以增强识别微小细胞的效果。最后，在各种数据集上进行了一系列对比实验，以选择合适的网络结构并验证所提出方法的有效性。

结果：结果表明，结合ISODATA方法和致密化策略的ResNet-50-FPN在U373等各种生物细胞数据集的多个指标（包括AP、Precision、Recall、Dice和PQ）上获得了比其他方法更好的性能、GoTW1、SIM+ 和 T24。

结论：我们的自适应算法可以有效地从各种大小和形状的细胞中学习锚形状先验。这对于未来生物医学工程在现实世界中基于锚的检测和分割应用是有希望和鼓舞的。