Skip to main content

Advertisement

Log in

Optimization strategies of fruit detection to overcome the challenge of unstructured background in field orchard environment: a review

  • Review
  • Published:
Precision Agriculture Aims and scope Submit manuscript

Abstract

The demand for intelligent agriculture is increasing due to the continuous impact of world food and environmental crises. Focusing on fruit detection, with the rapid development of object detection technology, it is now possible to achieve high efficiency and high accuracy in fruit detection systems. However, detecting fruit with high precision in unstructured orchard environments remains particularly challenging. Such environments, which are composed of varying illumination conditions and degrees of occlusion, can be mitigated by certain strategies. To our knowledge, this is the first time that optimization strategies used in fruit detection have been reviewed. This review aims to explore methods for improving fruit detection in complex environments. First, we describe the common types of complex backgrounds found in outdoor orchard environments. Subsequently, we divide the improvement measures into two categories: optimization before and after image sampling. Next, we compare the test results obtained before and after the application of these improved methods. Finally, we describe the future development trends of fruit detection optimization technology in complex backgrounds. We hope that this review will inspire researchers to design their optimization strategies and help explore lower-cost and more robust fruit detection systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  • Apolo-Apolo, O. E., Martínez-Guanter, J., Egea, G., Raja, P., & Pérez-Ruiz, M. (2020a). Deep learning techniques for estimation of the yield and size of citrus fruits using a UAV. European Journal of Agronomy, 115, 126030. https://doi.org/10.1016/j.eja.2020.126030

    Article  Google Scholar 

  • Apolo-Apolo, O. E., Pérez-Ruiz, M., Martínez-Guanter, J., & Valente, J. (2020b). A cloud-based environment for generating yield estimation maps from apple orchards using UAV imagery and a deep learning technique. Frontiers in Plant Science, 11, 1086. https://doi.org/10.3389/fpls.2020.01086

    Article  PubMed  PubMed Central  Google Scholar 

  • Aquino, A., Millan, B., Diago, M., & Tardaguila, J. (2018). Automated early yield prediction in vineyards from on-the-go image acquisition. Computers and Electronics in Agriculture, 144, 26–36. https://doi.org/10.1016/j.compag.2017.11.026

    Article  Google Scholar 

  • Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). SegNet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615

    Article  PubMed  Google Scholar 

  • Bargoti, S., & Underwood, J. (2017). Deep fruit detection in orchards. In 2017 IEEE International Conference on Robotics and Automation (ICRA) (pp. 3626–3633). IEEE.

  • Bay, H., Ess, A., Tuytelaars, T., & Van Gool, L. (2008). Speeded-up robust features (SURF). Computer Vision and Image Understanding, 110(3), 346–359. https://doi.org/10.1016/j.cviu.2007.09.014

    Article  Google Scholar 

  • Behera, S. K., Rath, A. K., & Sethy, P. K. (2021). Fruits yield estimation using Faster R-CNN with MIoU. Multimedia Tools and Applications, 80(12), 19043–19056. https://doi.org/10.1007/s11042-021-10704-7

    Article  Google Scholar 

  • Bochkovskiy, A., Wang, C., & Liao, H. M. (2020). Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934.

  • Bresilla, K., Perulli, G. D., Boini, A., Morandi, B., Corelli Grappadelli, L., & Manfrini, L. (2019). Single-shot convolution neural networks for real-time fruit detection within the tree. Frontiers in Plant Science, 10, 611. https://doi.org/10.3389/fpls.2019.00611

    Article  PubMed  PubMed Central  Google Scholar 

  • Bulanon, D. M., Burks, T. F., & Alchanatis, V. (2009). Image fusion of visible and thermal images for fruit detection. Biosystems Engineering, 103(1), 12–22. https://doi.org/10.1016/j.biosystemseng.2009.02.009

    Article  Google Scholar 

  • Cabrera Ardila, C. E., Alberto Ramirez, L., & Prieto Ortiz, F. A. (2020). Spectral analysis for the early detection of anthracnose in fruits of Sugar Mango (Mangifera indica). Computers and Electronics in Agriculture, 173, 105357. https://doi.org/10.1016/j.compag.2020.105357

    Article  Google Scholar 

  • Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., & Zagoruyko, S. (2020). End-to-end object detection with transformers. In European conference on computer vision (pp. 213–229). Springer, Cham.

  • Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014). Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062.

  • Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2017a). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848.

    Article  PubMed  Google Scholar 

  • Chen, L., Papandreou, G., Schroff, F., & Adam, H. (2017b). Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587.

  • Chen, L. C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018). Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) (pp. 801–818).

  • Chen, M., Tang, Y., Zou, X., Huang, K., Huang, Z., Zhou, H., Wang, C., & Lian, G. (2020). Three-dimensional perception of orchard banana central stock enhanced by adaptive multi-vision technology. Computers and Electronics in Agriculture, 174, 105508. https://doi.org/10.1016/j.compag.2020.105508

    Article  Google Scholar 

  • Chen, M., Tang, Y., Zou, X., Huang, Z., Zhou, H., & Chen, S. (2021). 3D global mapping of large-scale unstructured orchard integrating eye-in-hand stereo vision and SLAM. Computers and Electronics in Agriculture, 187, 106237. https://doi.org/10.1016/j.compag.2021.106237

    Article  Google Scholar 

  • Chen, Y., Lee, W. S., Gan, H., Peres, N., Fraisse, C., Zhang, Y., & He, Y. (2019). Strawberry yield prediction based on a deep neural network using high-resolution aerial orthoimages. Remote Sensing, 11(13), 1584. https://doi.org/10.3390/rs11131584

    Article  Google Scholar 

  • Chinchuluun, R., Lee, W. S., & Ehsani, R. (2009). Machine vision system for determining citrus count and size on a canopy shake and catch harvester. Applied Engineering in Agriculture, 25(4), 451–458. https://doi.org/10.13031/2013.27459

    Article  Google Scholar 

  • Chu, P., Li, Z., Lammers, K., Lu, R., & Liu, X. (2021). Deep learning-based apple detection using a suppression mask R-CNN. Pattern Recognition Letters, 147, 206–211. https://doi.org/10.1016/j.patrec.2021.04.022

    Article  Google Scholar 

  • Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297.

    Article  Google Scholar 

  • Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27.

    Article  Google Scholar 

  • Dalal, N., & Triggs, B. (2005). Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05) (Vol. 1, pp. 886–893). IEEE.

  • Das, J., Cross, G., Qu, C., Makineni, A., Tokekar, P., Mulgaonkar, Y., & Kumar, V. (2015). Devices, systems, and methods for automated monitoring enabling precision agriculture. In 2015 IEEE International Conference on Automation Science and Engineering (CASE) (pp. 462–469). IEEE.

  • Dong, W., Roy, P., Peng, C., & Isler, V. (2021). Ellipse R-CNN: Learning to infer elliptical object from clustering and occlusion. IEEE Transactions on Image Processing, 30, 1. https://doi.org/10.1109/TIP.2021.3050673

    Article  Google Scholar 

  • Dorj, U., Lee, M., & Yun, S. (2017). An yield estimation in citrus orchards via fruit detection and counting using image processing. Computers and Electronics in Agriculture, 140, 103–112. https://doi.org/10.1016/j.compag.2017.05.019

    Article  Google Scholar 

  • Eizentals, P., & Oka, K. (2016). 3D pose estimation of green pepper fruit for automated harvesting. Computers and Electronics in Agriculture, 128, 127–140. https://doi.org/10.1016/j.compag.2016.08.024

    Article  Google Scholar 

  • Elharrouss, O., Almaadeed, N., Al-Maadeed, S., & Akbari, Y. (2020). Image inpainting: A review. Neural Processing Letters, 51(2), 2007–2028.

    Article  Google Scholar 

  • FAO, F. (2017). The future of food and agriculture—Trends and challenges. Annual Report, 296.

  • Felzenszwalb, P., McAllester, D., & Ramanan, D. (2008a). A discriminatively trained, multiscale, deformable part model. Paper presented at the 2008a IEEE conference on computer vision and pattern recognition.

  • Felzenszwalb, P., McAllester, D., & Ramanan, D. (2008b). A discriminatively trained, multiscale, deformable part model. In 2008b IEEE conference on computer vision and pattern recognition (pp. 1–8). IEEE.

  • Felzenszwalb, P. F., Girshick, R. B., & McAllester, D. (2010). Cascade object detection with deformable part models. In 2010 IEEE computer society conference on computer vision and pattern recognition (pp. 2241–2248). IEEE.

  • Felzenszwalb, P. F., Girshick, R. B., McAllester, D., & Ramanan, D. (2009). Object detection with discriminatively trained part-based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32(9), 1627–1645. https://doi.org/10.1109/TPAMI.2009.167

    Article  Google Scholar 

  • Feng, J., Zeng, L., & He, L. (2019). Apple fruit recognition algorithm based on multi-spectral dynamic image analysis. Sensors, 19(4), 949. https://doi.org/10.3390/s19040949

    Article  PubMed  PubMed Central  Google Scholar 

  • Font, D., Pallejà, T., Tresanchez, M., Teixidó, M., Martinez, D., Moreno, J., & Palacín, J. (2014). Counting red grapes in vineyards by detecting specular spherical reflection peaks in RGB images obtained at night with artificial illumination. Computers and Electronics in Agriculture, 108, 105–111. https://doi.org/10.1016/j.compag.2014.07.006

    Article  Google Scholar 

  • Font, D., Tresanchez, M., Martínez, D., Moreno, J., Clotet, E., & Palacín, J. (2015). Vineyard yield estimation based on the analysis of high resolution images obtained with artificial illumination at night. Sensors, 15(4), 8284–8301. https://doi.org/10.3390/s150408284

    Article  PubMed  PubMed Central  Google Scholar 

  • Fu, L., Feng, Y., Wu, J., Liu, Z., Gao, F., Majeed, Y., & Cui, Y. (2021). Fast and accurate detection of kiwifruit in orchard using improved YOLOv3-tiny model. Precision Agriculture, 22(3), 754–776.

    Article  CAS  Google Scholar 

  • Fu, L., Tola, E., Al-Mallahi, A., Li, R., & Cui, Y. (2019). A novel image processing algorithm to separate linearly clustered kiwifruits. Biosystems Engineering, 183, 184–195. https://doi.org/10.1016/j.biosystemseng.2019.04.024

    Article  Google Scholar 

  • Fu, L., Gao, F., Wu, J., Li, R., Karkee, M., & Zhang, Q. (2020). Application of consumer RGB-D cameras for fruit detection and localization in field: A critical review. Computers and Electronics in Agriculture, 177, 105687. https://doi.org/10.1016/j.compag.2020.105687

    Article  Google Scholar 

  • Gai, R., Chen, N., & Yuan, H. (2021). A detection algorithm for cherry fruits based on the improved YOLO-v4 model. Neural Computing & Applications. https://doi.org/10.1007/s00521-021-06029-z

    Article  Google Scholar 

  • Gan, H., Lee, W. S., Alchanatis, V., & Abd-Elrahman, A. (2020). Active thermal imaging for immature citrus fruit detection. Biosystems Engineering, 198, 291–303. https://doi.org/10.1016/j.biosystemseng.2020.08.015

    Article  Google Scholar 

  • Gan, H., Lee, W. S., Alchanatis, V., Ehsani, R., & Schueller, J. K. (2018). Immature green citrus fruit detection using color and thermal images. Computers and Electronics in Agriculture, 152, 117–125. https://doi.org/10.1016/j.compag.2018.07.011

    Article  Google Scholar 

  • Gebbers, R., & Adamchuk, V. I. (2010). Precision agriculture and food security. Science, 327(5967), 828–831. https://doi.org/10.1126/science.1183899

    Article  CAS  PubMed  Google Scholar 

  • Gené-Mola, J., Gregorio, E., Cheein, F. A., Guevara, J., Llorens, J., Sanz-Cortiella, R., & Rosell-Polo, J. R. (2020a). Fruit detection, yield prediction and canopy geometric characterization using LiDAR with forced air flow. Computers and Electronics in Agriculture, 168, 105121.

    Article  Google Scholar 

  • Gené-Mola, J., Sanz-Cortiella, R., Rosell-Polo, J. R., Morros, J., Ruiz-Hidalgo, J., Vilaplana, V., & Gregorio, E. (2020b). Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry. Computers and Electronics in Agriculture, 169, 105165. https://doi.org/10.1016/j.compag.2019.105165

    Article  Google Scholar 

  • Gené-Mola, J., Gregorio, E., Guevara, J., Auat, F., Sanz-Cortiella, R., Escolà, A., & Rosell-Polo, J. R. (2019). Fruit detection in an apple orchard using a mobile terrestrial laser scanner. Biosystems Engineering, 187, 171–184. https://doi.org/10.1016/j.biosystemseng.2019.08.017

    Article  Google Scholar 

  • Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 1440–1448).

  • Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 580–587).

  • Gongal, A., Amatya, S., Karkee, M., Zhang, Q., & Lewis, K. (2015). Sensors and systems for fruit detection and localization: A review. Computers and Electronics in Agriculture, 116, 8–19. https://doi.org/10.1016/j.compag.2015.05.021

    Article  Google Scholar 

  • Gongal, A., Silwal, A., Amatya, S., Karkee, M., Zhang, Q., & Lewis, K. (2016). Apple crop-load estimation with over-the-row machine vision system. Computers and Electronics in Agriculture, 120, 26–35. https://doi.org/10.1016/j.compag.2015.10.022

    Article  Google Scholar 

  • Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., & Bengio, Y. (2014). Generative adversarial nets. Advances in Neural Information Processing Systems, 27.

  • Han, Y., Batra, R., Boyd, N., Zhao, T., She, Y., Hutchinson, S., Zhao, Y. (2021). Learning generalizable vision-tactile robotic grasping strategy for deformable objects via transformer. arXiv preprint arXiv:2112.06374.

  • He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 2961–2969).

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778).

  • Hemming, J., Ruizendaal, J., Hofstee, J. W., & Van Henten, E. J. (2014). Fruit detectability analysis for different camera positions in sweet-pepper. Sensors, 14(4), 6032–6044. https://doi.org/10.3390/s140406032

    Article  PubMed  PubMed Central  Google Scholar 

  • Hemming, S., de Zwart, F., Elings, A., Righini, I., & Petropoulou, A. (2019). Remote control of greenhouse vegetable production with artificial intelligence—Greenhouse climate, irrigation, and crop production. Sensors, 19(8), 1807. https://doi.org/10.3390/s19081807

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Hou, L., Wu, Q., Sun, Q., Yang, H., & Li, P. (2016, August). Fruit recognition based on convolution neural network. In 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD) (pp. 18–22). IEEE. https://doi.org/10.1109/FSKD.2016.7603144

  • Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.

  • Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132–7141).

  • Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700–4708).

  • Hung, C., Nieto, J., Taylor, Z., Underwood, J., & Sukkarieh, S. (2013). Orchard fruit segmentation using multi-spectral feature learning. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 5314–5320). IEEE.

  • Jia, W., Tian, Y., Luo, R., Zhang, Z., Lian, J., & Zheng, Y. (2020). Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot. Computers and Electronics in Agriculture, 172, 105380. https://doi.org/10.1016/j.compag.2020.105380

    Article  Google Scholar 

  • Jiao, L., Wu, H., Wang, H., & Bie, R. (2019). Multi-scale semantic image inpainting with residual learning and GAN. Neurocomputing, 331, 199–212.

    Article  Google Scholar 

  • Ju, Y. J., Lee, G. H., Hong, J. H., & Lee, S. W. (2022). Complete face recovery GAN: unsupervised joint face rotation and de-occlusion from a single-view image. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 3711–3721).

  • Kamilaris, A., & Prenafeta-Boldú, F. X. (2018). Deep learning in agriculture: A survey. Computers and Electronics in Agriculture, 147, 70–90. https://doi.org/10.1016/j.compag.2018.02.016

    Article  Google Scholar 

  • Kirk, R., Cielniak, G., & Mangan, M. (2020). L* a* b* fruits: A rapid and robust outdoor fruit detection system combining bio-inspired features with one-stage deep learning networks. Sensors, 20(1), 275. https://doi.org/10.3390/s20010275

    Article  PubMed  PubMed Central  Google Scholar 

  • Kleynen, O., Leemans, V., & Destain, M. F. (2005). Development of a multi-spectral vision system for the detection of defects on apples. Journal of Food Engineering, 69(1), 41–49. https://doi.org/10.1016/j.jfoodeng.2004.07.008

    Article  Google Scholar 

  • Koirala, A., Walsh, K. B., & Wang, Z. (2021). Attempting to estimate the unseen—Correction for occluded fruit in tree fruit load estimation by machine vision with deep learning. Agronomy, 11(2), 347. https://doi.org/10.3390/agronomy11020347

    Article  Google Scholar 

  • Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019a). Deep learning–method overview and review of use for fruit detection and yield estimation. Computers and Electronics in Agriculture, 162, 219–234. https://doi.org/10.1016/j.compag.2019.04.017

    Article  Google Scholar 

  • Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019b). Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO.’ Precision Agriculture, 20(6), 1107–1135. https://doi.org/10.1007/s11119-019-09642-0

    Article  Google Scholar 

  • Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25, 1097–1105.

    Google Scholar 

  • Kuznetsova, A., Maleva, T., & Soloviev, V. (2020). Using YOLOv3 algorithm with pre- and post-processing for apple detection in fruit-harvesting robot. Agronomy, 10(7), 1016. https://doi.org/10.3390/agronomy10071016

    Article  Google Scholar 

  • LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444.

    Article  CAS  PubMed  Google Scholar 

  • LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.

    Article  Google Scholar 

  • Li, C., Li, L., Jiang, H., Weng, K., Geng, Y., Li, L., Ke, Z., Li, Q., Cheng, M., & Nie, W. (2022). YOLOv6: A single-stage object detection framework for industrial applications. arXiv preprint arXiv:2209.02976.

  • Li, H., Li, C., Li, G., & Chen, L. (2021a). A real-time table grape detection method based on improved YOLOv4-tiny network in complex background. Biosystems Engineering, 212, 347–359. https://doi.org/10.1016/j.biosystemseng.2021.11.011

    Article  CAS  Google Scholar 

  • Li, X., Pan, J., Xie, F., Zeng, J., Li, Q., Huang, X., & Wang, X. (2021b). Fast and accurate green pepper detection in complex backgrounds via an improved Yolov4-tiny model. Computers and Electronics in Agriculture, 191, 106503. https://doi.org/10.1016/j.compag.2021.106503

    Article  Google Scholar 

  • Liakos, K. G., Busato, P., Moshou, D., Pearson, S., & Bochtis, D. (2018). Machine learning in agriculture: A review. Sensors, 18(8), 2674. https://doi.org/10.3390/s18082674

    Article  PubMed  PubMed Central  Google Scholar 

  • Liang, C., Xiong, J., Zheng, Z., Zhong, Z., Li, Z., Chen, S., & Yang, Z. (2020). A visual detection method for nighttime litchi fruits and fruiting stems. Computers and Electronics in Agriculture, 169, 105192. https://doi.org/10.1016/j.compag.2019.105192

    Article  Google Scholar 

  • Lin, G., Tang, Y., Zou, X., Xiong, J., & Li, J. (2019). Guava detection and pose estimation using a low-cost RGB-D sensor in the field. Sensors, 19(2), 428. https://doi.org/10.3390/s19020428

    Article  PubMed  PubMed Central  Google Scholar 

  • Lin, G., Tang, Y., Zou, X., & Wang, C. (2021). Three-dimensional reconstruction of guava fruits and branches using instance segmentation and geometry analysis. Computers and Electronics in Agriculture, 184, 106107. https://doi.org/10.1016/j.compag.2021.106107

    Article  Google Scholar 

  • Linker, R. (2018). Machine learning based analysis of night-time images for yield prediction in apple orchard. Biosystems Engineering, 167, 114–125. https://doi.org/10.1016/j.biosystemseng.2018.01.003

    Article  Google Scholar 

  • Linker, R., & Kelman, E. (2015). Apple detection in nighttime tree images using the geometry of light patches around highlights. Computers and Electronics in Agriculture, 114, 154–162. https://doi.org/10.1016/j.compag.2015.04.005

    Article  Google Scholar 

  • Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., & Sánchez, C. I. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60–88. https://doi.org/10.1016/j.media.2017.07.005

    Article  PubMed  Google Scholar 

  • Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C. Y., & Berg, A. C. (2016). Ssd: Single shot multibox detector. In European conference on computer vision (pp. 21–37). Springer, Cham.

  • Liu, X., Chen, S. W., Aditya, S., Sivakumar, N., Dcunha, S., Qu, C., Kumar, V. (2018). Robust fruit counting: combining deep learning, tracking, and structure from motion. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 1045–1052). IEEE.

  • Liu, X., Chen, S. W., Liu, C., Shivakumar, S. S., Das, J., Taylor, C. J., & Kumar, V. (2019). Monocular camera based fruit counting and mapping with semantic data association. IEEE Robotics and Automation Letters, 4(3), 2296–2303.

    Article  Google Scholar 

  • Llorens, J., Gil, E., Llop, J., & Escolà, A. (2011). Ultrasonic and LIDAR sensors for electronic canopy characterization in vineyards: Advances to improve pesticide application methods. Sensors, 11(2), 2177–2194. https://doi.org/10.3390/s110202177

    Article  PubMed  PubMed Central  Google Scholar 

  • Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431–3440).

  • Longsheng, F., Bin, W., Yongjie, C., Shuai, S., Gejima, Y., & Kobayashi, T. (2015). Kiwifruit recognition at nighttime using artificial lighting based on machine vision. International Journal of Agricultural and Biological Engineering, 8(4), 52.

    Google Scholar 

  • Lowe, D. G. (1999). Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision (Vol. 2, pp. 1150–1157). IEEE.

  • Lu, J., Lee, W. S., Gan, H., & Hu, X. (2018). Immature citrus fruit detection based on local binary pattern feature and hierarchical contour analysis. Biosystems Engineering, 171, 78–90. https://doi.org/10.1016/j.biosystemseng.2018.04.009

    Article  Google Scholar 

  • Mack, J., Lenz, C., Teutrine, J., & Steinhage, V. (2017). High-precision 3D detection and reconstruction of grapes from laser range data for efficient phenotyping based on supervised learning. Computers and Electronics in Agriculture, 135, 300–311. https://doi.org/10.1016/j.compag.2017.02.017

    Article  Google Scholar 

  • MacQueen, J. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, No. 14, pp. 281–297).

  • Misra, D. (2019). Mish: A self regularized non-monotonic activation function. arXiv preprint arXiv:1908.0868. https://doi.org/10.48550/arXiv.1908.08681

  • Nguyen, T. T., Vandevoorde, K., Wouters, N., Kayacan, E., De Baerdemaeker, J. G., & Saeys, W. (2016). Detection of red and bicoloured apples on tree with an RGB-D camera. Biosystems Engineering, 146, 33–44. https://doi.org/10.1016/j.biosystemseng.2016.01.007

    Article  Google Scholar 

  • Nuske, S., Wilshusen, K., Achar, S., Yoder, L., Narasimhan, S., & Singh, S. (2014). Automated visual yield estimation in vineyards. Journal of Field Robotics, 31(5), 837–860.

    Article  Google Scholar 

  • Payne, A., Walsh, K., Subedi, P., & Jarvis, D. (2014). Estimating mango crop yield using image analysis using fruit at ‘stone hardening’ stage and night time imaging. Computers and Electronics in Agriculture, 100, 160–167. https://doi.org/10.1016/j.compag.2013.11.011

    Article  Google Scholar 

  • Pourdarbani, R., Sabzi, S., Kalantari, D., Karimzadeh, R., Ilbeygi, E., & Arribas, J. I. (2020). Automatic non-destructive video estimation of maturation levels in Fuji apple (Malus Malus pumila) fruit in orchard based on colour (Vis) and spectral (NIR) data. Biosystems Engineering, 195, 136–151. https://doi.org/10.1016/j.biosystemseng.2020.04.015

    Article  Google Scholar 

  • Pretty, J. (2008). Agricultural sustainability: concepts, principles and evidence. Philosophical Transactions of the Royal Society B: Biological Sciences, 363(1491), 447–465. https://doi.org/10.1098/rstb.2007.2163

    Article  Google Scholar 

  • Qin, J., Burks, T. F., Zhao, X., Niphadkar, N., & Ritenour, M. A. (2012). Development of a two-band spectral imaging system for real-time citrus canker detection. Journal of Food Engineering, 108(1), 87–93. https://doi.org/10.1016/j.jfoodeng.2011.07.022

    Article  Google Scholar 

  • Rahnemoonfar, M., & Sheppard, C. (2017). Deep count: Fruit counting based on deep simulated learning. Sensors, 17(4), 905. https://doi.org/10.3390/s17040905

    Article  PubMed  PubMed Central  Google Scholar 

  • Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 779–788).

  • Redmon, J., & Farhadi, A. (2017). YOLO9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7263–7271).

  • Redmon, J., & Farhadi, A. (2018). Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767.

  • Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems. https://doi.org/10.1109/TPAMI.2016.2577031

    Article  Google Scholar 

  • Ripley, B. D. (2007). Pattern recognition and neural networks. Cambridge University Press.

    Google Scholar 

  • Robinson, T. L., Hoying, S. A., & Reginato, G. H. (2008). The tall spindle planting system: principles and performance. In IX International Symposium on Integrating Canopy, Rootstock and Environmental Physiology in Orchard Systems 903 (pp. 571–579).

  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention (pp. 234–241). Springer, Cham.

  • Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985). Learning internal representations by error propagation. California Univ San Diego La Jolla Inst for Cognitive Science.

  • Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., & McCool, C. (2016). Deepfruits: A fruit detection system using deep neural networks. Sensors, 16(8), 1222. https://doi.org/10.3390/s16081222

    Article  PubMed  PubMed Central  Google Scholar 

  • Schonberger, J. L., & Frahm, J. M. (2016). Structure-from-motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4104–4113).

  • Senthilnath, J., Dokania, A., Kandukuri, M., Ramesh, K. N., Anand, G., & Omkar, S. N. (2016). Detection of tomatoes using spectral-spatial methods in remotely sensed RGB images captured by UAV. Biosystems Engineering, 146, 16–32. https://doi.org/10.1016/j.biosystemseng.2015.12.003

    Article  Google Scholar 

  • Shi, X., Chen, Z., Wang, H., Yeung, D. Y., Wong, W. K., & Woo, W. C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in Neural Information Processing Systems, 28.

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

  • Song, Z., Fu, L., Wu, J., Liu, Z., Li, R., & Cui, Y. (2019). Kiwifruit detection in field images using Faster R-CNN with VGG16. IFAC-Papers OnLine, 52(30), 76–81. https://doi.org/10.1016/j.ifacol.2019.12.500

    Article  Google Scholar 

  • Stajnko, D., Lakota, M., & Hočevar, M. (2004). Estimation of number and diameter of apple fruits in an orchard during the growing season by thermal imaging. Computers and Electronics in Agriculture, 42(1), 31–42. https://doi.org/10.1016/S0168-1699(03)00086-3

    Article  Google Scholar 

  • Stein, M., Bargoti, S., & Underwood, J. (2016). Image based mango fruit detection, localisation and yield estimation using multiple view geometry. Sensors, 16(11), 1915. https://doi.org/10.3390/s16111915

    Article  PubMed  PubMed Central  Google Scholar 

  • Steinbrener, J., Posch, K., & Leitner, R. (2019). Hyperspectral fruit and vegetable classification using convolutional neural networks. Computers and Electronics in Agriculture, 162, 364–372. https://doi.org/10.1016/j.compag.2019.04.019

    Article  Google Scholar 

  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–9).

  • Taigman, Y., Yang, M., Ranzato, M. A., & Wolf, L. (2014). Deepface: closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1701–1708).

  • Tang, Y., Chen, M., Wang, C., Luo, L., Li, J., Lian, G., & Zou, X. (2020). Recognition and localization methods for vision-based fruit picking robots: A review. Frontiers in Plant Science, 11, 510. https://doi.org/10.3389/fpls.2020.00510

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Tang, Y., Zhou, H., Wang, H., & Zhang, Y. (2023). Fruit detection and positioning technology for a Camellia oleifera C. Abel orchard based on improved YOLOv4-tiny model and binocular stereo vision. Expert Systems with Applications, 211, 118573. https://doi.org/10.1016/j.eswa.2022.118573

    Article  Google Scholar 

  • Tebaldi, C., & Lobell, D. B. (2008). Towards probabilistic projections of climate change impacts on global crop yields. Geophysical Research Letters, 35(8).

  • Tian, Y., Yang, G., Wang, Z., Li, E., Liang, Z., Padilla-Medina, J. A. (2019a). Detection of apple lesions in orchards based on deep learning methods of CycleGAN and YOLOV3-Dense. Journal of Sensors, 1–13. https://doi.org/10.1155/2019/7630926.

  • Tian, Y., Yang, G., Wang, Z., Wang, H., Li, E., & Liang, Z. (2019b). Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Computers and Electronics in Agriculture, 157, 417–426. https://doi.org/10.1016/j.compag.2019.01.012

    Article  Google Scholar 

  • Tsoulias, N., Paraforos, D. S., Xanthopoulos, G., & Zude-Sasse, M. (2020). Apple shape detection based on geometric and radiometric features using a LiDAR laser scanner. Remote Sensing, 12(15), 2481. https://doi.org/10.3390/rs12152481

    Article  Google Scholar 

  • Tu, S., Pang, J., Liu, H., Zhuang, N., Chen, Y., Zheng, C., & Xue, Y. (2020). Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images. Precision Agriculture, 21(5), 1072–1091. https://doi.org/10.1007/s11119-020-09709-3

    Article  Google Scholar 

  • Tu, S., Xue, Y., Zheng, C., Qi, Y., Wan, H., & Mao, L. (2018). Detection of passion fruits and maturity classification using Red-Green-Blue Depth images. Biosystems Engineering, 175, 156–167. https://doi.org/10.1016/j.biosystemseng.2018.09.004

    Article  Google Scholar 

  • Tyagi, A. C. (2016). Towards a second green revolution. Irrigation and Drainage, 65(4), 388–389. https://doi.org/10.1002/ird.2076

    Article  Google Scholar 

  • Underwood, J. P., Hung, C., Whelan, B., & Sukkarieh, S. (2016). Mapping almond orchard canopy volume, flowers, fruit and yield using lidar and vision sensors. Computers and Electronics in Agriculture, 130, 83–96. https://doi.org/10.1016/j.compag.2016.09.014

    Article  Google Scholar 

  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems30.

  • Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. In Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001 (Vol. 1, p. I). IEEE.

  • Viola, P., & Jones, M. J. (2004). Robust real-time face detection. International Journal of Computer Vision, 57(2), 137–154.

    Article  Google Scholar 

  • Wachs, J. P., Stern, H. I., Burks, T., & Alchanatis, V. (2010). Low and high-level visual feature-based apple detection from multi-modal images. Precision Agriculture, 11(6), 717–735. https://doi.org/10.1007/s11119-010-9198-x

    Article  Google Scholar 

  • Wang, C., Bochkovskiy, A., & Liao, H. M. (2022a). YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696.

  • Wang, H., Lin, Y., Xu, X., Chen, Z., Wu, Z., & Tang, Y. (2022b). A study on long-close distance coordination control strategy for Litchi picking. Agronomy, 12(7), 1520. https://doi.org/10.3390/agronomy12071520

    Article  Google Scholar 

  • Wang, C., Lee, W. S., Zou, X., Choi, D., Gan, H., & Diamond, J. (2018a). Detection and counting of immature green citrus fruit based on the Local Binary Patterns (LBP) feature using illumination-normalized images. Precision Agriculture, 19(6), 1062–1083. https://doi.org/10.1007/s11119-018-9574-5

    Article  Google Scholar 

  • Wang, X., Xiao, T., Jiang, Y., Shao, S., Sun, J., & Shen, C. (2018b). Repulsion loss: Detecting pedestrians in a crowd. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 7774–7783). https://doi.org/10.1109/CVPR.2018.00811.

  • Wang, C., Tang, Y., Zou, X., Luo, L., & Chen, X. (2017a). Recognition and matching of clustered mature litchi fruits using binocular charge-coupled device (CCD) color cameras. Sensors, 17(11), 2564. https://doi.org/10.3390/s17112564

    Article  PubMed  PubMed Central  Google Scholar 

  • Wang, Z., Walsh, K. B., & Verma, B. (2017b). On-tree mango fruit size estimation using RGB-D images. Sensors, 17(12), 2738. https://doi.org/10.3390/s17122738

    Article  PubMed  PubMed Central  Google Scholar 

  • Wang, C., Luo, T., Zhao, L., Tang, Y., & Zou, X. (2019). Window zooming–based localization algorithm of fruit and vegetable for harvesting robot. IEEE Access, 7, 103639–103649. https://doi.org/10.1109/ACCESS.2019.2925812

    Article  Google Scholar 

  • Wang, C. Y., Bochkovskiy, A., & Liao, H. Y. M. (2021a). Scaled-yolov4: scaling cross stage partial network. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition (pp. 13029–13038).

  • Wang, H., Dong, L., Zhou, H., Luo, L., Lin, G., Wu, J., Garg, A. (2021b). YOLOv3-Litchi detection method of densely distributed litchi in large vision scenes. Mathematical Problems in Engineering, 1–11. https://doi.org/10.1155/2021/8883015.

  • Wang, J., Zhang, Z., Luo, L., Zhu, W., Chen, J., & Wang, W. (2021c). SwinGD: A robust grape bunch detection model based on swin transformer in complex vineyard environment. Horticulturae, 7(11), 492. https://doi.org/10.3390/horticulturae7110492

    Article  Google Scholar 

  • Wang, Q., Nuske, S., Bergerman, M., & Singh, S. (2013). Automated crop yield estimation for apple orchards. In Experimental robotics (pp. 745–758). Springer.

  • Wang, Y., Lv, J., Xu, L., Gu, Y., Zou, L., & Ma, Z. (2020). A segmentation method for waxberry image under orchard environment. Scientia Horticulturae, 266, 109309. https://doi.org/10.1016/j.scienta.2020.109309

    Article  Google Scholar 

  • Wei, X., Jia, K., Lan, J., Li, Y., Zeng, Y., & Wang, C. (2014). Automatic method of fruit object extraction under complex agricultural background for vision system of fruit picking robot. Optik, 125(19), 5684–5689. https://doi.org/10.1016/j.ijleo.2014.07.001

    Article  Google Scholar 

  • Wu, F., Duan, J., Ai, P., Chen, Z., Yang, Z., & Zou, X. (2022). Rachis detection and three-dimensional localization of cut off point for vision-based banana robot. Computers and Electronics in Agriculture, 198, 107079. https://doi.org/10.1016/j.compag.2022.107079

    Article  Google Scholar 

  • Xiang, R. (2018). Image segmentation for whole tomato plant recognition at night. Computers and Electronics in Agriculture, 154, 434–442. https://doi.org/10.1016/j.compag.2018.09.034

    Article  Google Scholar 

  • Xiong, J., Lin, R., Liu, Z., He, Z., Tang, L., Yang, Z., & Zou, X. (2018). The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment. Biosystems Engineering, 166, 44–57. https://doi.org/10.1016/j.biosystemseng.2017.11.005

    Article  Google Scholar 

  • Xu, W., Chen, H., Su, Q., Ji, C., Xu, W., Memon, M. S., & Zhou, J. (2019). Shadow detection and removal in apple image segmentation under natural light conditions using an ultrametric contour map. Biosystems Engineering, 184, 142–154. https://doi.org/10.1016/j.biosystemseng.2019.06.016

    Article  Google Scholar 

  • Yan, B., Fan, P., Lei, X., Liu, Z., & Yang, F. (2021). A real-time apple targets detection method for picking robot based on improved YOLOv5. Remote Sensing, 13(9), 1619. https://doi.org/10.3390/rs13091619

    Article  Google Scholar 

  • Yu, F., & Koltun, V. (2015). Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122.

  • Yu, Y., Zhang, K., Liu, H., Yang, L., & Zhang, D. (2020). Real-time visual localization of the picking points for a ridge-planting strawberry harvesting robot. IEEE Access, 8, 116556–116568. https://doi.org/10.1109/ACCESS.2020.3003034

    Article  Google Scholar 

  • Yu, Y., Zhang, K., Yang, L., & Zhang, D. (2019). Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Computers and Electronics in Agriculture, 163, 104846. https://doi.org/10.1016/j.compag.2019.06.001

    Article  Google Scholar 

  • Yurtsever, E., Lambert, J., Carballo, A., & Takeda, K. (2020). A survey of autonomous driving: Common practices and emerging technologies. IEEE Access, 8, 58443–58469. https://doi.org/10.1109/ACCESS.2020.2983149

    Article  Google Scholar 

  • Zemmour, E., Kurtser, P., & Edan, Y. (2019). Automatic parameter tuning for adaptive thresholding in fruit detection. Sensors, 19(9), 2130. https://doi.org/10.3390/s19092130

    Article  PubMed  PubMed Central  Google Scholar 

  • Zhang, W., Chen, K., Wang, J., Shi, Y., & Guo, W. (2021). Easy domain adaptation method for filling the species gap in deep learning-based fruit detection. Horticulture Research, 8(1). https://doi.org/10.1038/s41438-021-00553-8.

  • Zhang, Q., Liu, Y., Gong, C., Chen, Y., & Yu, H. (2020). Applications of deep learning for dense scenes analysis in agriculture: A review. Sensors, 20(5), 1520. https://doi.org/10.3390/s20051520

    Article  PubMed  PubMed Central  Google Scholar 

  • Zhang, Y., Lee, W. S., Li, M., Zheng, L., & Ritenour, M. A. (2018). Non-destructive recognition and classification of citrus fruit blemishes based on ant colony optimized spectral information. Postharvest Biology and Technology, 143, 119–128. https://doi.org/10.1016/j.postharvbio.2018.05.004

    Article  Google Scholar 

  • Zheng, Z., Zheng, L., & Yang, Y. (2017). Unlabeled samples generated by gan improve the person re-identification baseline in vitro. In Proceedings of the IEEE international conference on computer vision (pp. 3754–3762).

  • Zhou, Y., Tang, Y., Zou, X., Wu, M., Tang, W., Meng, F., Zhang, Y., & Kang, H. (2022). Adaptive active positioning of Camellia oleifera Fruit Picking Points: Classical image processing and YOLOv7 fusion algorithm. Applied Sciences, 12(24), 12959. https://doi.org/10.3390/app122412959

    Article  CAS  Google Scholar 

  • Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision (pp. 2223–2232).

  • Zou, Z., Shi, Z., Guo, Y., & Ye, J. (2019). Object detection in 20 years: A survey. arXiv preprint arXiv:1905.05055.

Download references

Acknowledgements

This work was supported by Guangdong Basic and Applied Basic Research Foundation (2022A1515140162, 2022A1515140013), the Key-area Research and Development Program of Guangdong Province (2019B020223003), Science and Technology Planning Project of Guangzhou (202102080269), the Natural Science Foundation of China (32171909).

Author information

Authors and Affiliations

Authors

Contributions

YT: Conceptualization, Reviewing, Editing. JQ: Conceptualization, Writing—original draft. YZ: Chart drawing and beautification. DW: Investigation. YC: Chart beautification. KZ: Editing. LZ: Supervision.

Corresponding author

Correspondence to Yunchao Tang.

Ethics declarations

Competing interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tang, Y., Qiu, J., Zhang, Y. et al. Optimization strategies of fruit detection to overcome the challenge of unstructured background in field orchard environment: a review. Precision Agric 24, 1183–1219 (2023). https://doi.org/10.1007/s11119-023-10009-9

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11119-023-10009-9

Keywords

Navigation