A method for organs classification and fruit counting on pomegranate trees based on multi-features fusion and support vector machine by 3D point cloud

Highlights

• 3D point cloud was used for organs classification and fruits recognition for the whole fruit tree on the natural environment.

• The shape descriptor used for identification and detection of industrial robots be used for agriculture.

• The KNN smoothing based on weighted Euclidean distance was used to smooth the results of SVM classifier.

• The agglomerative-divisive hierarchical clustering makes the clustering effect better through double threshold adjustment.

Abstract

Organs classification and fruit counting on pomegranate trees are of great significance for horticulture works and robotic picking. However, there are still some challenges: (1) illumination is uncontrollable in the natural environment; (2) traditional 2D image-based methods for classification and recognition are limited by occlusion on pomegranate trees. In this paper, a method for organs classification and fruit counting on pomegranate trees based on multi-features fusion and Support Vector Machine (SVM) was proposed. It was constructed by the following steps: (1) Three-dimensional point clouds of pomegranate trees were obtained by an RGB-D camera; (2) Three-dimensional point clouds were preprocessed; (3) Color and shape features were extracted to train the SVM classifier; (4) The obtained classifier model was used for organs classification on pomegranate trees; (5) A K-nearest neighbor (KNN) smoothing based on weighted Euclidean distance was used to improve the accuracy of classification; (6) An agglomerative-divisive hierarchical clustering was used to count pomegranate fruit. The experiment results showed that the SVM classifier based on color and shape feature had an accuracy of 0.75 for fruit and 0.99 for non-fruit. The fruit counting based on agglomerative-divisive hierarchical clustering had a recall of 87.74 % and a precision of 78.15 %. Compared with density-based spatial clustering of applications with noise (DBSCAN), the recall has improved significantly. This method was aimed at the whole fruit tree, so it has advantages in the completeness of information. The results indicated that the proposed method was effective and feasible for organs classification and yield estimation on pomegranate trees in the natural environment.

Introduction

Organs classification and fruit counting are important for horticulture works in orchard management. Recognition and counting of fruit can be used to identify the growing stage and estimate yield. Spatial localization of fruit can be used to guide robotic picking. The identification of leaf can be used for precision spraying (Westling et al., 2018). Trunk and branch recognition can be used for the pruning, which affects the quantity and quality of fruit (Rosell et al., 2012). In recent years, many researchers have researched organs classification and recognition on fruit trees.

Some researchers have researched the fruit recognition methods based on 2D image (such as colors, shapes and textures). These methods have achieved good results. In these works, different color spaces (such as RGB, YCbCr and YUV) were used to extract different color features (Mohammadi et al., 2015; Lei et al., 2019; Liu et al., 2019a,b; Li et al., 2018). Shape descriptors were used to extract shape features, such as histogram of oriented gradient (HOG) (Liu et al., 2019a,b), Hough transformation and shape context algorithm (Gongal et al., 2016; Lu et al., 2018; Linker, 2017; Niu, 2017). Lv et al. (2019a, b) proposed a method based on control limited adaptive histogram equalization (CLAHE) algorithm for bagged green apple image segmentation. In some works, template matching with weighted Euclidean distance (TMWE) (Tan et al., 2018), weighted relevance vector machine (RVM) (Wu et al., 2019), classification and regression tree classifier (Yamamoto et al., 2014), etc. were used to identify fruit. Recently, methods for fruit recognition based on deep learning have been used in researchers' works, such as convolutional neural network (CNN) (Bargoti et al., 2017), pulse coupled neural network (PCNN) (Xu et al., 2018) and faster regions with CNN features (faster RCNN) (Stein et al., 2016). However, deep learning requires a lot of data and takes a lot of time to train models. Fruit counting is an important part of yield estimation. Researchers mostly counted fruit by clustering. Such as X-means (Yamamoto et al., 2014), clustering algorithm based on Euclidian distance (Gongal et al., 2016), kernel fuzzy C-means (Lei et al., 2019). In addition, thermal images (Gan et al., 2018) was used to identify fruit. In Lv et al., 2019a, b, a apple image segmentation method based on convex hull center priori and Markov adsorption chain was proposed with a ultimate measurement accuracy of 3.15 % and a IoU of 95.65 %. In summary, although features (such as color, shape, texture) by 2D image were used for fruit recognition and organs classification, there were still some challenges: (1) Shape features extracted from 2D images were incomplete, and then, (2) fruit by 2D images were occluded, which was an important factor affecting fruit recognition and organs classification in horticulture works.

Compared with 2D images, 3D point clouds shows more complete plant information. There are three methods for acquiring 3D point cloud: (1) Large-scale, high-precision and high-cost equipment, such as 3D laser scanners (Pearse et al., 2019; Calders et al., 2015; Yang et al., 2018; Jamayet et al., 2018; Park et al., 2018), laser radar (Madec et al., 2017; Yan et al., 2015; Li et al., 2019; Krishna et al., 2019; Qiu et al., 2019), and ultrasonic (Hu et al., 2018; Gangadharan et al., 2019). These devices are generally used for three-dimensional reconstruction of large-scale scenes, such as forest reconstruction, building large-scale three-dimensional maps, etc., and have high accuracy in the reconstruction of large-scale scenes. However, they are expensive, so their application in general scene reconstruction is limited. (2) Small-range, moderate-precision and low-cost RGB-D camera (Wang et al., 2017; Chiu et al., 2019; Tu et al., 2018; Gené-Mola et al., 2019). Depth cameras are generally used for 3D reconstruction of small scenes, such as indoor scene reconstruction, human body 3D reconstruction, etc. They generally have a shooting range not more than ten meters, and has high accuracy in small-scale scene reconstruction. (3) Reconstruct 3D point cloud based on multiple images (Dey et al., 2012; Zhou et al., 2019; Jay et al., 2015). This method can directly use the ordinary camera to obtain the two-dimensional image of the scene, but its reconstruction accuracy is affected by the algorithm and the resolution of the camera. In recent years, the SVM classifiers based on color and 3D shape features have been commonly used by researchers (Dey et al., 2012; Tao et al., 2017; Avendano et al., 2017; Ramos et al., 2018). A method based on color-fast point feature histogram (Color-FPFH) descriptors and SVM classifier for organs classification and fruit recognition on apple trees was proposed in Tao et al. (2017). Some researchers identified fruit by only 3D features. A method based on convex template Instance (CTI) descriptor for various fruit (such as tomato, apple) recognition was proposed in Nyarko et al. (2018). In addition, clustering was used by researchers to detect individual fruit. In Díaz et al. (2018), an approach based on density-based spatial clustering of applications with noise (DBSCAN) for grape buds detection in winter was proposed with a precision of 100 % and a recall of 45 %. In Lin et al. (2019), a method for guava segmentation based on Euclidean clustering was proposed with a precision of 98.3 % and a recall of 94.8 %. In summary, the 3D point cloud shows more complete shape features of fruit trees than the 2D image, and has some improvement on occlusion condition, therefore, the 3D point cloud is better to identify fruit and classify organs in horticulture works.

In this paper, a method for organs classification and fruit counting on pomegranate trees was proposed. This method uses the fusion of the shape and color features of the three-dimensional point cloud to classify organs. Previously, the shape descriptor (Kleppe et al., 2018) were mostly used for identification and detection of industrial robots, and their applications in agriculture were few. The proposed method can be used for various operations such as pruning, spraying and yield estimation in orchards in the natural environment.