Abstract
The detection and localization of pineapple fruit must be successfully conducted to realize intelligent picking. This paper proposed a method for detecting and localizing pineapples in natural environments based on binocular stereo vision and an improved YOLO (You Only Look Once) v3 model. In comparison with the original YOLOv3, the improved algorithm did the following two improvement: the DenseNet was added into the Darknet-53 backbone network to optimize the 13 × 13 and 26 × 26 feature layer, and the SPP-net was fused in the 52 × 52 dimension detection module to strengthen the information representation ability of feature map. A binocular camera acquired left and right images. The left image was then input to the improved YOLOv3 model to obtain the position information of pineapples in the image. Stereo matching and parallax calculation of the target pineapple region were completed via the stereo matching algorithm. Finally, the three-dimensional co-ordinates of pineapples were calculated based on the triangulation principle of binocular stereo vision. A series of experiments were run to compare the detection result of this method against YOLOv3, Faster-RCNN (Region Convolutional Neural Network) and Mobilenet-SSD (Single Shot MultiBox Detector) as per their respective F1 score (Balanced Score) and AP (Average Precision) values. On test set with slight occlusion, the F1 score and AP values of the improved YOLOv3 model were 93.18% and 97.55%, respectively. As the occlusion grew severe, the F1 score and AP values decreased to 89.15% and 91.47%, respectively. The improved YOLOv3 model developed in this study had the best detection effect among all models tested. The binocular stereo vision localization experiment showed an average absolute error of 24.414 mm and average relative error of 1.17% at a distance of 1.7–2.7 m. The proposed method may thus be suitable for picking robots detecting and localizing pineapple fruit in natural environments.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bargoti, S., & Underwood, J. P. (2017). Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 34(6), 1039–1060. https://doi.org/10.1002/rob.21699
Bisong, E. (2019) TensorFlow 2.0 and Keras. In Building machine learning and deep learning models on Google cloud platform. Berkeley, CA, USA: Apress. https://doi.org/10.1007/978-1-4842-4470-830
Chaivivatrakul, S., & Dailey, M. N. (2014). Texture-based fruit detection. Precision Agriculture, 15(6), 662–683. https://doi.org/10.1007/s11119-014-9361-x
Duong, L. T., Nguyen, P. T., Sipio, C. D., & Ruscio, D. D. (2020). Automated fruit recognition using efficientnet and mixnet. Computers and Electronics in Agriculture, 171, 105326. https://doi.org/10.1016/j.compag.2020.105326
Fu, L. S., Liu, Z. H., Majeed, Y., & Cui, Y. J. (2018). Kiwifruit yield estimation using image processing by an Android mobile phone. IFAC-PapersOnLine, 51(17), 185–190. https://doi.org/10.1016/j.ifacol.2018.08.137
Fu, L. S., Feng, Y. L., Wu, J. Z., Liu, Z. H., Gao, F. F., Majeed, Y., et al. (2021). Fast and accurate detection of kiwifruit in orchard using improved YOLOv3-tiny model. Precision Agriculture, 22, 754–776. https://doi.org/10.1007/s11119-020-09754-y
Gao, F. F., Fu, L., Zhang, X., Majeed, Y., & Zhang, Q. (2020). Multi-class fruit-on-plant detection for apple in snap system using faster r-cnn. Computers and Electronics in Agriculture, 176, 105634. https://doi.org/10.1016/j.compag.2020.105634
Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE international conference on computer vision (pp. 1440–1448). https://arxiv.org/abs/1504.08083v2
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society (pp. 580–587).
Hamuda, E., Ginley, B. M., Glavin, M., & Jones, E. (2018). Improved image processing-based crop detection using Kalman filtering and the Hungarian algorithm. Computers and Electronics in Agriculture, 148, 37–44. https://doi.org/10.1016/j.compag.2018.02.027
He, D. J., Zhang, L. Z., Li, X., Li, P., & Wang, T. Y. (2019). Design of automatic pineapple harvesting machine based on binocular machine vision. Anhui Agricultural Science, 13, 207–210. (in Chinese with English abstract).
He, K., Zhang, X., Ren, S., & Sun, J. (2014). Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence, 37(9), 1904–1916.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 770–778). https://doi.org/10.1109/CVPR.2016.90
Huang, G., Liu, Z., Laurens, V., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In IEEE conference on computer vision and pattern recognition, IEEE Computer Society (pp. 2261–2269). https://arxiv.org/abs/1608.06993
Jia, W. K., Tian, Y. Y., Luo, R., Zhang, Z. H., Jian, L., & Zheng, Y. J. (2020). Detection and segmentation of overlapped fruits based on optimized mask r-cnn application in apple harvesting robot. Computers and Electronics in Agriculture, 172, 105380. https://doi.org/10.1016/j.compag.2020.105380
Jiang, Z. Y. (2019). TensorFlow deep learning algorithm principle and programming practice. China Water Resources and Hydropower Press. (In Chinese).
Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. https://arxiv.org/1412.6980.
Koirala, A., Walsh, K. B., Wang, Z., & Mccarthy, C. (2019a). Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of “MangoYOLO.” Precision Agriculture, 20, 1107–1135. https://doi.org/10.1007/s11119-019-09642-0
Koirala, A., Walsh, K. B., Wang, Z. L., & McCarthy, C. (2019b). Deep learning-Method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 162, 219–234. https://doi.org/10.1016/j.compag.2019.04.017
Li, B., Wang, H. F., Huang, W. Q., & Zhang, C. (2012). Construction and in-field experiment of low-cost binocular vision platform for pineapple harvesting robot. Transactions of the Chinese Society of Agricultural Engineering, 28(S2), 188–192.
Liang, C., Xiong, J., Zheng, Z., Zhong, Z., & Yang, Z. (2020). A visual detection method for nighttime litchi fruits and fruiting stems. Computers and Electronics in Agriculture, 169, 105192. https://doi.org/10.1016/j.compag.2019.105192
Linker, R., Cohen, O., & Naor, A. (2012). Determination of the number of green apples in RGB images recorded in orchards. Computers and Electronics in Agriculture, 81(1), 45–57. https://doi.org/10.1016/j.compag.2011.11.007
Liu, F., Liu, Y. K., Lin, S., Guo, W. Z., Xu, F., & Zhang, B. (2020a). Fast recognition method for tomatoes under complex environments based on improved YOLO. Journal of Agricultural Machinery, 51(06), 236–244. https://doi.org/10.6041/j.issn.1000-1298.2020.06.024 (in Chinese with English abstract).
Liu, T. H., Ehsani, R., Toudeshki, A., Zou, X. J., & Wang, H. J. (2019). Identifying immature and mature pomelo fruits in trees by elliptical model fitting in the Cr-Cb color space. Precision Agriculture, 20(1), 138–156. https://doi.org/10.1007/s11119-018-9586-1
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., et al. (2016). SSD: Single shot MultiBox detector. In European conference on computer vision. Springer. https://arxiv.org/abs/1512.02325
Liu, X. G., Fan, C., Li, J. N., Gao, Y. L., Zhang, Y. Y., & Yang, Q. L. (2020b). Identification method of strawberry based on convolutional neural network. Journal of Agricultural Machinery, 051(002), 237–244. https://doi.org/10.6041/j.issn.1000-1298.2020.02.026 (in Chinese with English abstract).
Montserrat, D. M., Lin, Q., Allebach, J., & Delp, E. (2017). Training object detection and recognition cnn models using data augmentation. Electronic Imaging, 10, 27–36. https://doi.org/10.2352/ISSN.2470-1173.2017.10.IMAWM-163
Rahnemoonfar, M., & Sheppard, C. (2017). Deep count: fruit counting based on deep simulated learning. Sensors, 17(4), 905. https://doi.org/10.3390/s17040905
Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: unified, real-time object detection. In Proceedings of the IEEE computer society conference on computer vision and pattern recognition (pp. 779–788). https://arxiv.org/abs/1506.02640
Redmon, J., & Farhadi, A. (2017). Yolo9000: better, faster, stronger. In IEEE conference on computer vision & pattern recognition (pp. 6517–6525). https://doi.org/10.1109/CVPR.2017.690
Redmon, J., & Farhadi, A. (2018). YOLOv3: an incremental improvement. In IEEE conference on computer vision and pattern recognition. https://arxiv.org/abs/1804.02767
Ren, S., He, K., Girshick, R., & Sun, J. (2016). Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 1137–1149.
Wang, Z., James, U., & Walsh, K. B. (2018). Machine vision assessment of mango orchard flowering. Computers and Electronics in Agriculture, 151, 501–511. https://doi.org/10.1016/j.compag.2018.06.040
Wang, Z. L., Walsh, K. B., & Verma, B. (2017). On-tree mango fruit size estimation using RGB-D images. Sensors, 17, 2738. https://doi.org/10.3390/s17122738
Wu, X., Qi, Z. Y., Wang, L. J., Yang, J. J., & Xia, X. (2020). Apple Detection Method Based on Light-YOLOv3 Convolutional Neural Network. Journal of Agricultural Machinery, 8, 17–25. https://doi.org/10.6041/j.issn.1000-1298.2020.08.002 (in Chinese with English abstract).
Xiong, J. T., Zheng, Z. H., Liang, J. E., Zhong, Z., Liu, B. L., & Sun, B. X. (2020). Citrus detection method in night environment based on improved YOLO v3 network. Journal of Agricultural Machinery, 51(04), 206–213. https://doi.org/10.6041/j.issn.1000-1298.2020.04.023 (in Chinese with English abstract).
Zhang, X., Gao, Q., Pan, D., Cao, P. C., & Huang, D. H. (2021). Research on spatial positioning system of fruits to be picked in field based on binocular vision and SSD Model. Journal of Physics, 1748(4), 042011. https://doi.org/10.1088/1742-6596/1748/4/042011
Acknowledgements
This research is supported by National Science Project of China with Research Grant 52175229.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Liu, TH., Nie, XN., Wu, JM. et al. Pineapple (Ananas comosus) fruit detection and localization in natural environment based on binocular stereo vision and improved YOLOv3 model. Precision Agric 24, 139–160 (2023). https://doi.org/10.1007/s11119-022-09935-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11119-022-09935-x