The accurate and reliable fruit detection in orchards is one of the most crucial tasks for supporting higher level agriculture tasks such as yield mapping and robotic harvesting. However, detecting and counting small fruit is a very challenging task under variable lighting conditions, low-resolutions and heavy occlusion by neighboring fruits or foliage. To robustly detect small fruits, an improved method is proposed based on multiple scale faster region-based convolutional neural networks (MS-FRCNN) approach using the color and depth images acquired with an RGB-D camera. The architecture of MS-FRCNN is improved to detect lower-level features by incorporating feature maps from shallower convolution feature maps for regions of interest (ROI) pooling. The detection framework consists of three phases. Firstly, multiple scale feature extractors are used to extract low and high features from RGB and depth images respectively. Then, RGB-detector and depth-detector are trained separately using MS-FRCNN. Finally, late fusion methods are explored for combining the RGB and depth detector. The detection framework was demonstrated and evaluated on two datasets that include passion fruit images under variable illumination conditions and occlusion. Compared with the faster R-CNN detector of RGB-D images, the recall, the precision and F1-score of MS-FRCNN method increased from 0.922 to 0.962, 0.850 to 0.931 and 0.885 to 0.946, respectively. Furthermore, the MS-FRCNN method effectively improves small passion fruit detection by achieving 0.909 of the F1 score. It is concluded that the detector based on MS-FRCNN can be applied practically in the actual orchard environment.

中文翻译：

---

基于RGB-D图像的多尺度faster R-CNN百香果检测与计数

果园中准确可靠的水果检测是支持更高级别农业任务（例如产量测绘和机器人收获）的最关键任务之一。然而，在可变光照条件、低分辨率和相邻水果或树叶的严重遮挡下，检测和计数小水果是一项非常具有挑战性的任务。为了稳健地检测小水果，提出了一种基于多尺度更快的基于区域的卷积神经网络 (MS-FRCNN) 方法的改进方法，该方法使用 RGB-D 相机获取的颜色和深度图像。MS-FRCNN 的架构得到改进，通过合并来自感兴趣区域 (ROI) 池的较浅卷积特征图的特征图来检测较低级别的特征。检测框架由三个阶段组成。首先，多尺度特征提取器分别用于从 RGB 和深度图像中提取低特征和高特征。然后，使用 MS-FRCNN 分别训​​练 RGB 检测器和深度检测器。最后，探索了结合 RGB 和深度检测器的后期融合方法。该检测框架在两个数据集上进行了演示和评估，其中包括可变光照条件和遮挡下的百香果图像。与 RGB-D 图像的更快 R-CNN 检测器相比，MS-FRCNN 方法的召回率、精度和 F1 分数分别从 0.922 增加到 0.962、0.850 到 0.931 和 0.885 到 0.946。此外，MS-FRCNN 方法通过实现 0.909 的 F1 分数有效地提高了小百香果检测。