Abstract
Deep learning techniques and computer vision systems offer effective fruit counting solutions for farm yield estimation. However, the performance of these solutions drops when identifying different cultivars of the same fruit species. This study clarified the differences between mango fruit detection and mango cultivar identification. An original double-threshold-based classification method for fruit cultivar identification, with estimation of the misidentification rate was proposed in order to significantly increase the performance of a specialised mango fruit detection method known as Faster R-CNN. This method was applied on images of mango trees of three cultivars taken in Senegalese orchards of different existing cropping systems, with varying tree features, planting patterns and acquisition contexts. Analysis of the results focused on the contributions of fruit detection errors and fruit cultivar confusion to the overall error of the network for fruit counts by cultivar class. The shift from fruit detection to cultivar identification resulted in a drop in the average prediction rate from 92 to 68%. With its explicitly independent fruit detection and cultivar identification steps, the double-threshold-based classification method increased the prediction rate to 86%, with a maximum identification error of 0.05%. This setting also led to relative equality between the recall and the precision of each cultivar class, making the network well suited for fruit counting by cultivar class. This work opened new perspectives for decision support tools for fruit growers that could provide more appropriate yield estimates per cultivar.
Similar content being viewed by others
Data availability
The datasets generated and analysed during the current study are available from the corresponding author upon reasonable request.
Code availability
Not applicable.
References
Anderson, N. T., Walsh, K. B., & Wulfsohn, D. (2021). Technologies for forecasting Tree Fruit load and Harvest timing—from Ground, Sky and Time. Agronomy, 11(7), 1409. https://doi.org/10.3390/agronomy11071409.
Bargoti, S., & Underwood, J. (2017). Deep fruit detection in orchards. Robotics and Automation (ICRA), 2017 IEEE International Conference on, 3626–3633. http://ieeexplore.ieee.org/abstract/document/7989417/
Behera, S. K., Rath, A. K., & Sethy, P. K. (2021). Fruits yield estimation using faster R-CNN with MIoU. Multimedia Tools and Applications, 80(12), 19043–19056. https://doi.org/10.1007/s11042-021-10704-7.
Blok, P. M., van Evert, F. K., Tielen, A. P., van Henten, E. J., & Kootstra, G. (2021). The effect of data augmentation and network simplification on the image-based detection of broccoli heads with Mask R‐CNN. Journal of Field Robotics, 38(1), 85–104.
Borianne, P., Sarron, J., Borne, F., & Faye, É. (2019). Deep mangoes: From fruit detection to cultivar identification in colour images of mango trees. ArXiv:1909.10939.
Burke, M., & Lobell, D. B. (2017). Satellite-based assessment of yield variation and its determinants in smallholder African systems. Proceedings of the National Academy of Sciences, 114(9), 2189–2194.
Cetinic, E., Lipic, T., & Grgic, S. (2018). Fine-tuning convolutional neural networks for fine art classification. Expert Systems with Applications, 114, 107–118. https://doi.org/10.1016/j.eswa.2018.07.026.
Dambreville, A., Lauri, P. E., Trottier, C., Guedon, Y., & Normand, F. (2013). Deciphering structural and temporal interplays during the architectural development of mango trees. Journal of Experimental Botany, 64(8), 2467–2480. https://doi.org/10.1093/jxb/ert105.
Deng, X., Lan, Y., Hong, T., & Chen, J. (2016). Citrus greening detection using visible spectrum imaging and C-SVC. Computers and Electronics in Agriculture, 130, 177–183. https://doi.org/10.1016/j.compag.2016.09.005.
Gao, F., Fu, L., Zhang, X., Majeed, Y., Li, R., Karkee, M., & Zhang, Q. (2020). Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN. Computers and Electronics in Agriculture. https://doi.org/10.1016/j.compag.2020.105634
Girshick, R., Donahue, J., Darrell, T., Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. 2014 IEEE Conference on Computer Vision and, & Recognition, 580–587. https://doi.org/10.1109/CVPR.2014.81
Girshick, R., & Fast, R. C. N. N. (2015). 2015 IEEE International Conference on Computer Vision (ICCV), 1440–1448. https://doi.org/10.1109/ICCV.2015.169
Gongal, A., Amatya, S., Karkee, M., Zhang, Q., & Lewis, K. (2015). Sensors and systems for fruit detection and localization: A review. Computers and Electronics in Agriculture, 116, 8–19. https://doi.org/10.1016/j.compag.2015.05.021.
Hou, L., Wu, Q., Sun, Q., Yang, H., & Li, P. (2016). Fruit recognition based on convolution neural network. 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), 18–22. https://doi.org/10.1109/FSKD.2016.7603144
Kang, H., & Chen, C. (2019). Fast implementation of real-time fruit detection in apple orchards using deep learning. Computers and Electronics in Agriculture. https://doi.org/10.1016/j.compag.2019.105108
Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019). Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO’. Precision Agriculture, 20(6), 1107–1135. https://doi.org/10.1007/s11119-019-09642-0.
Mbow, C., Van Noordwijk, M., Luedeling, E., Neufeldt, H., Minang, P. A., & Kowero, G. (2014). Agroforestry solutions to address food security and climate change challenges in Africa. Current Opinion in Environmental Sustainability, 6, 61–67. https://doi.org/10.1016/j.cosust.2013.10.014.
Mettleq, A. S. A., Dheir, I. M., Elsharif, A. A., & Abu-Naser, S. S. (2019). Mango classification using deep learning. International Journal of Academic Engineering Research, 3(12), 8.
Mukherjee, S. K., & Litz, R. E. (2009). Introduction Botany and Importance. In R. E. Litz (Ed.), The mango: Botany, production and uses (2nd ed., pp. 1–18). CABI.
Osako, Y., Yamane, H., Lin, S. Y., Chen, P. A., & Tao, R. (2020). Cultivar discrimination of litchi fruit images using deep learning. Scientia Horticulturae, 269, 109360. https://doi.org/10.1016/j.scienta.2020.109360.
Payne, A., Walsh, K., Subedi, P., & Jarvis, D. (2014). Estimating mango crop yield using image analysis using fruit at ‘stone hardening’ stage and night time imaging. Computers and Electronics in Agriculture, 100, 160–167. https://doi.org/10.1016/j.compag.2013.11.011.
Qureshi, W. S., Payne, A., Walsh, K. B., Linker, R., Cohen, O., & Dailey, M. N. (2016). Machine vision for counting fruit on mango tree canopies. Precision Agriculture. https://doi.org/10.1007/s11119-016-9458-5.
Rahnemoonfar, M., & Sheppard, C. (2017). Deep Count: Fruit counting based on deep simulated learning. Sensors (Basel, Switzerland), 17(4), 905. https://doi.org/10.3390/s17040905.
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in neural information processing systems. 91–99. http://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks
Rey, J. Y., Diallo, T. M., Vannière, H., Didier, C., Kéita, S., & Sangaré, M. (2007). The mango in french-speaking West Africa: Varieties and varietal composition of the orchards. Fruits, 62(1), 57–73. https://doi.org/10.1051/fruits:2006051.
Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., & McCool, C. (2016). DeepFruits: A fruit detection system using deep neural networks. Sensors (Basel, Switzerland), 16(8), 1222. https://doi.org/10.3390/s16081222
Sarron, J., Malézieux, É., Sané, C., & Faye, É. (2018). Mango yield mapping at the orchard scale based on tree structure and land cover assessed by UAV. Remote Sensing, 10(12), 1900. https://doi.org/10.3390/rs10121900
Sarron, J., Beillouin, D., Huat, J., et al. (2022). Digital agriculture to fulfil the shortage of horticultural data and achieve food security in sub-saharan Africa. Acta Horticulture. https://doi.org/10.17660/ActaHortic.2022.1348.33
Sarron, J., Sané, C. A. B., Borianne, P., Malézieux, E., Nordey, T., Normand, F., Diatta, P., Niang, Y., & Faye, E. (2020). Is machine learning efficient for mango yield estimation when used under heterogeneous field conditions? Acta Horticulturae, 1279, 201–208. https://doi.org/10.17660/ActaHortic.2020.1279.30.
Schneider, C. A., Rasband, W. S., & Eliceiri, K. W. (2012). NIH Image to ImageJ: 25 years of image analysis. Nature Methods, 9(7), 671–675. https://doi.org/10.1038/nmeth.2089.
Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. ArXiv. http://arxiv.org/abs/1409.1556
Tadesse, D., & Blank, S. C. (2003). Cultivar Diversity: A neglected risk management strategy. Journal of Agricultural and Resource Economics, 28(2), 217–232. http://www.jstor.org/stable/40987183.
Wan, S., & Goudos, S. (2020). Faster R-CNN for multi-class fruit detection using a robotic vision system. Computer Networks, 168, 107036. https://doi.org/10.1016/j.comnet.2019.107036.
Wang, L. (2017). Heterogeneous data and big data analytics. Automatic Control and Information Sciences, 3(1), 8–15. https://doi.org/10.12691/acis-3-1-3
Wang, C., & Peng, Z. (2019). Design and implementation of an object detection system using faster R-CNN. In 2019 International Conference on Robots & Intelligent System (ICRIS) (pp. 204–206). IEEE.
Zhang, S., Wen, L., Bian, X., Lei, Z., & Li, S. Z. (2018). Occlusion-aware R-CNN: Detecting pedestrians in a crowd. ArXiv. http://arxiv.org/abs/1807.08407
Acknowledgements
This work has been financially supported by the French National Research Agency (ANR) under the programme “Investissements d’avenir”, ANR-16-CONV-0004 #Digitag. It was also part of the PixFruit App project of the Occitanie Region (ESR Premat 00224). The authors wish to thank Dr. Sané for his expert work and active participation in the annotation of visual data.
Author information
Authors and Affiliations
Contributions
EF was responsible for funding acquisition. All authors contributed to the conception and design of the study. Data collection and expert annotation were performed by JS and EF. PB and FB conducted methodology, neural network implementation, experiments and result analysis. PB wrote the first draft of the manuscript and JS and EF reviewed and commented on the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare no conflict of interest.
Ethics approval
Not applicable.
Consent to participate
Not applicable.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Borianne, P., Sarron, J., Borne, F. et al. Deep mango cultivars: cultivar detection by classification method with maximum misidentification rate estimation. Precision Agric 24, 1619–1637 (2023). https://doi.org/10.1007/s11119-023-10012-0
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11119-023-10012-0