Abstract
Citrus fruits do not ripen at the same time in natural environments and exhibit different maturity stages on trees, hence it is necessary to realize selective harvesting of citrus picking robots. The visual attention mechanism reveals a physiological phenomenon that human eyes usually focus on a region that is salient from its surround. The degree to which a region contrasts with its surround is called visual saliency. This study proposes a novel citrus fruit maturity method combining visual saliency and convolutional neural networks to identify three maturity levels of citrus fruits. The proposed method is divided into two stages: the detection of citrus fruits on trees and the detection of fruit maturity. In stage one, the object detection network YOLOv5 was used to identify the citrus fruits in the image. In stage two, a visual saliency detection algorithm was improved and generated saliency maps of the fruits; The information of RGB images and the saliency maps were combined to determine the fruit maturity class using 4-channel ResNet34 network. The comparison experiments were conducted around the proposed method and the common RGB-based machine learning and deep learning methods. The experimental results show that the proposed method yields an accuracy of 95.07%, which is higher than the best RGB-based CNN model, VGG16, and the best machine learning model, KNN, about 3.14% and 18.24%, respectively. The results prove the validity of the proposed fruit maturity detection method and that this work can provide technical support for intelligent visual detection of selective harvesting robots.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abeywickrama, T., Cheema, M. A., & Taniar, D. (2016). K-nearest neighbors on road networks: A journey in experimentation and in-memory implementation. Proceedings of the VLDB Endowment, 9(6), 1–18.
Achanta, R., & Susstrunk, S. (2010). Saliency detection using maximum symmetric surround. In 2010 IEEE International Conference on Image Processing (ICIP) (pp. 2653–2656). https://doi.org/10.1109/icip.2010.5652636.
Ali, M. M., Hashim, N., & Hamid, A. S. A. (2020). Combination of laser-light backscattering imaging and computer vision for rapid determination of oil palm fresh fruit bunches maturity. Computers and Electronics in Agriculture, 169, 1–9. https://doi.org/10.1016/j.compag.2020.105235
Bochkovskiy, A., Wang, C., & Liao, H. (2020). YOLOv4: optimal speed and accuracy of object detection. arXiv: 2004.10934[cs.CV].
Cardenas-Perez, S., Chanona-Perez, J., Mendez-Mendez, J., et al. (2017). Evaluation of the ripening stages of apple (golden delicious) by means of computer vision system. Biosystems Engineering, 159, 46–58. https://doi.org/10.1016/j.biosystemseng.2017.04.009
Chang, C.-C., & Lin, C.-J. (2011). LIBSVM : A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(27), 1–27.
Chien-Yao Wang, Hong-Yuan Mark Liao, Yueh-Hua Wu, et al. (2020). CSPNet: A new backbone that can enhance learning capability of CNN. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE Computer Society, 2020: 1571–1580. https://doi.org/10.1109/cvprw50498.2020.00203
Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297.
Das, P., & Yadav, J. (2020). Automated Tomato Maturity Grading System using CNN. In 2020 International Conference on Smart Electronics and Communication (ICOSEC). https://doi.org/10.1109/icosec49089.2020.9215451
El-Bendary, N., El Hariri, E., Hassanien, A., & Badr, A. (2015). Using machine learning techniques for evaluating tomato ripeness. Expert Systems with Applications, 42, 1892–1905. https://doi.org/10.1016/j.eswa.2014.09.057
Faisal, M., Alsulaiman, M., Arafah, M., & Mekhtiche, M. A. (2020). IHDS: Intelligent harvesting decision system for date fruit based on maturity stage using deep learning and computer vision. IEEE Access, 8, 167985–167997. https://doi.org/10.1109/access.2020.3023894
Gupta, A. K., Pathak, U., Tongbram, T., Medhi, M., et al. (2021). Emerging approaches to determine maturity of citrus fruit. Critical Reviews in Food Science and Nutrition. https://doi.org/10.1080/10408398.2021.1883547
Glenn J, Alex S, Jirka B, et al. (2021). YOLOv5. Retrieved January 15, 2021, from https://github.com/ultralytics/yoloV5.
Goel, N., & Sehgal, P. (2015). Fuzzy classification of pre-harvest tomatoes for ripeness estimation—An approach based on automatic rule learning using decision tree. Applied Soft Computing, 36, 45–56. https://doi.org/10.1016/j.asoc.2015.07.009
Habaragamuwa, H., Ogawa, Y., Suzuki, T., Shiigi, T., Ono, M., & Kondo, N. (2018). Detecting greenhouse strawberries (mature and immature), using deep convolutional neural network. Engineering in Agriculture, Environment and Food. https://doi.org/10.1016/j.eaef.2018.03.001
Harel, B., Essen, R. V., Parmet, Y., & Edan, Y. (2020a). Viewpoint analysis for maturity classification of sweet peppers. Sensors, 20(13), 3783. https://doi.org/10.3390/s20133783
Harel, B., Parmet, Y., & Edan, Y. (2020b). Maturity classification of sweet peppers using image datasets acquired in different times. Computers in Industry, 121, 1–10. https://doi.org/10.1016/j.compind.2020.103274
He, K., Zhang, X., Ren, S., et al. (2016). Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, 770–778. https://doi.org/10.1109/CVPR.2016.90
Huang, G., Liu, Z., Laurens, V., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition (pp. 1–28). https://doi.org/10.1109/CVPR.2017.243
Kao, I. H., Hsu, Y. W., Yang, Y. Z., Chen, Y. L., Lai, Y. H., & Perng, J. W. (2019). Determination of lycopersicon maturity using convolutional autoencoders. Scientia Horticulturae, 256(C), 108538–108538. https://doi.org/10.1016/j.scienta.2019.05.065
Khojastehnazhand, M., Mohammadi, V., & Minaei, S. (2019). Maturity detection and volume estimation of apricot using image processing technique. Scientia Horticulturae, 251, 247–251. https://doi.org/10.1016/j.scienta.2019.03.033
Lecun, Y., & Bottou, L. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324. https://doi.org/10.1109/5.726791
Li, D., Li, X., Han, Q., Zhou, Y., Dong, J., & Duan, Z. (2020). Phosphorus application improved the yield of citrus plants grown for three years in an acid soil in the Three Gorges Reservoir Area. Scientia Horticulturae, 273, 1–7. https://doi.org/10.1016/j.scienta.2020.109596
Li, H., Lee, W., & Ku, W. (2014). Identifying blueberry fruit of different growth stages using natural outdoor color images. Computers & Electronics in Agriculture, 106, 91–101. https://doi.org/10.1016/j.compag.2014.05.015
Liu, S., Qi, L., Qin, H., Shi, J., & Jia, J. (2018). Path aggregation network for instance segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018, 8759–8768. https://doi.org/10.1109/CVPR.2018.00913
Mim, F. S., Galib, S. M., Hasan, M. F., et al. (2018). Automatic detection of mango ripening stages—An application of information technology to botany. Scientia Horticulturae, 237, 156–163. https://doi.org/10.1016/j.scienta.2018.03.057
Nandi, C. S., Tudu, B., & Koley, C. (2014). Machine vision based techniques for automatic mango fruit sorting and grading based on maturity level and size. Sensing Technology: Current Status and Future Trends II (Eds.), 8(265), 27–46. https://doi.org/10.1007/978-3-319-02315-1_2
Nasiri, A., Taheri-Garavand, A., & Dong Zhang, Yu. (2019). Image-based deep learning automated sorting of date fruit. Postharvest Biology and Technology, 153, 133–141. https://doi.org/10.1016/j.postharvbio.2019.04.003
Pereira, S., Fernando, L., Barbon, S., Valous, A., & Barbin, D. (2018). Predicting the ripening of papaya fruit with digital imaging and random forests. Computers & Electronics in Agriculture, 145, 76–82. https://doi.org/10.1016/j.compag.2017.12.029
Prabha, D. S., & Kumar, J. S. (2015). Assessment of banana fruit maturity by image processing technique. Journal of Food Science & Technology, 52(3), 1316. https://doi.org/10.1007/s13197-013-1188-3
Ramos, P., Avendaño, J., & Prieto, F. (2018). Measurement of the ripening rate on coffee branches by using 3d images in outdoor environments. Computers in Industry, 99, 83–95. https://doi.org/10.1016/j.compind.2018.03.024
Redmon, J., & Farhadi, A. (2018). YOLOv3: An Incremental Improvement. Retrieved 08 April 2018, from https://arxiv.org/abs/1804.02767v1.
Rhodes, M. J. C. (1980). Senescence in plants. Boca Raton: CRC Press.
Rodríguez-Pulido, F. J., Gómez-Robledo, L., Melgosa, M., Gordillo, B., González-Miret, M. L., & Heredia, F. J. (2012). Ripeness estimation of grape berries and seeds by image analysis. Computers & Electronics in Agriculture, 82, 128–133. https://doi.org/10.1016/j.compag.2012.01.004
Seymour, G. B., & Tucker, G. A. (1993). Biochemistry of fruit ripening. Dordrecht: Springer. https://doi.org/10.1002/9781118593714.ch1
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. Retrieved October 20, 2015, from http://arxiv.org/abs/1409.1556v6.
Tan, K., Lee, W., Gan, H., & Wang, S. (2018). Recognising blueberry fruit of different maturity using histogram oriented gradients and colour features in outdoor scenes. Biosystems Engineering, 176, 59–72. https://doi.org/10.1016/j.biosystemseng.2018.08.011
Treisman, A. M., & Gelade, G. (1980). A feature-integration theory of attention. Cognitive Psychology, 12(1), 97–136. https://doi.org/10.1016/0010-0285(80)90005-5
Tu, S., Xue, Y., Zheng, C., Qi, Yu., & Hua, W. (2018). Detection of passion fruits and maturity classification using red-green-blue depth images. Biosystems Engineering, 175, 156–167. https://doi.org/10.1016/j.biosystemseng.2018.09.004
Vélez-Rivera, N., Blasco, J., Chanona-Pérez, J., Calderón-Domínguez, et al. (2014). Computer vision system applied to classification of “Manila” mangoes during ripening process. Food Bioprocess Technology, 7, 1183–1194. https://doi.org/10.1007/s11947-013-1142-4.
Wan, P., Toudeshki, A., Tan, H., & Ehsani, R. (2018). A methodology for fresh tomato maturity detection using computer vision. Computers and Electronics in Agriculture, 146, 43–50. https://doi.org/10.1016/j.compag.2018.01.011
Yuan, J. (2020). Research progress analysis of robotics selective harvesting technologies. Transactions of the Chinese Society for Agricultural Machinery, 51(9), 1–17. https://doi.org/10.6041/j.issn.1000-1298.2020.09.001
Acknowledgements
This work is funded by National Natural Science Foundation of China (Project No. 32071912), Science and Technology Program of Guangzhou (202002030423), Special Funds for the Cultivation of Guangdong College Students’ Scientific and Technological Innovation. (“Climbing Program” Special Funds.) (pdjh2020a0082). The authors wish to thank the useful comments of the anonymous reviewers to this paper.
Author information
Authors and Affiliations
Contributions
Chen, S. and Xiong, J. conceived and designed the experiments; Chen, S., Jiao, J. and Xie, Z. performed the experiments and analyzed the data; Chen, S., Xiong, J., Jiao, J., Xie, Z., Huo, Z., and Hu, W. wrote the paper.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no conflicts of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Chen, S., Xiong, J., Jiao, J. et al. Citrus fruits maturity detection in natural environments based on convolutional neural networks and visual saliency map. Precision Agric 23, 1515–1531 (2022). https://doi.org/10.1007/s11119-022-09895-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11119-022-09895-2