Abstract
Automatic detection of kiwifruit in the orchard is challenging because illumination varies through the day and night and because of color similarity between kiwifruit and the complex background of leaves, branches and stems. Also, kiwifruits grow in clusters, which may result in having occluded and touching fruits. A fast and accurate object detection algorithm was developed to automatically detect kiwifruits in the orchard by improving the YOLOv3-tiny model. Based on the characteristics of kiwifruit images, two convolutional kernels of 3 × 3 and 1 × 1 were added to the fifth and sixth convolution layers of the YOLOv3-tiny model, respectively, to develop a deep YOLOv3-tiny (DY3TNet) model. It takes multiple 1 × 1 convolutional layers in intermediate layers of the network to reduce the computational complexity. Testing images captured from day and night and comparing with other deep learning models, namely, Faster R-CNN with ZFNet, Faster R-CNN with VGG16, YOLOv2 and YOLOv3-tiny, the DY3TNet model achieved the highest average precision of 0.9005 with the smallest data weight of 27 MB. Furthermore, it took only 34 ms on average to process an image of a resolution of 2352 × 1568 pixels. The DY3TNet model, along with the YOLOv3-tiny model, showed better performance on images captured with flash than those without. Moreover, the experiments indicated that the image augmentation process could improve the detection performance, and a simple lighting arrangement could improve the success rate of detection in the orchard. The experimental results demonstrated that the improved DY3TNet model is small and efficient and that it would increase the applicability of real-time kiwifruit detection in the orchard even when small hardware devices are used.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akcay, S., Kundegorski, M. E., Willcocks, C. G., & Breckon, T. P. (2018). Using deep convolutional neural network architectures for object classification and detection within x-ray baggage security imagery. IEEE Transactions on Information Forensics and Security, 13(9), 2203–2215. https://doi.org/10.1109/TIFS.2018.2812196.
Al-masni, M. A., Al-antari, M. A., Park, J. M., Gi, G., Kim, T. Y., Rivera, P., et al. (2018). Simultaneous detection and classification of breast masses indigital mammograms via a deep learning YOLO-based CAD system. Computer Methods and Programs in Biomedicine, 157, 85–94. https://doi.org/10.1016/j.cmpb.2018.01.017.
Bargoti, S., & Underwood, J. (2017). Image segmentation for fruit detection and yield estimation in apple orchards. Journal of Field Robotics, 34, 1039–1060. https://doi.org/10.1002/rob.21699.
Fu, L., Feng, Y., Elkamil, T., Liu, Z., Li, R., & Cui, Y. (2018a). Image recognition method of multi-cluster kiwifruit in field based on convolutional neural networks. Transactions of the Chinese Society of Agricultural Engineering, 34(2), 205–211. https://doi.org/10.11975/j.issn.1002-6819.2018.02.028 (in Chinese with English abstract).
Fu, L., Feng, Y., Majeed, Y., Zhang, X., Zhang, J., Karkee, M., et al. (2018b). Kiwifruit detection in field images using faster R-CNN with ZFNet. IFAC-PapersOnLine, 51(17), 45–50. https://doi.org/10.1016/j.ifacol.2018.08.059.
Fu, L., Liu, Z., Majeed, Y., & Cui, Y. (2018c). Kiwifruit yield estimation using image processing by an android mobile phone. IFAC-PapersOnLine, 51(17), 185–190. https://doi.org/10.1016/j.ifacol.2018.08.137.
Fu, L., Majeed, Y., Zhang, X., Karkee, M., & Zhang, Q. (2020). Faster R-CNN-based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting. Biosystems Engineering, 197, 245–256. https://doi.org/10.1016/j.biosystemseng.2020.07.007.
Fu, L., Sun, S., Li, R., & Wang, S. (2016). Classification of kiwifruit grades based on fruit shape using a single camera. Sensors, 16(7), 1012. https://doi.org/10.3390/s16071012.
Fu, L., Sun, S., Vázquezarellano, M., Li, S., Li, R., & Cui, Y. (2017). Kiwifruit recognition method at night based on fruit calyx image. Transactions of the Chinese Society of Agricultural Engineering, 33(2), 199–204. https://doi.org/10.11975/j.issn.1002-6819.2017.02.027 (in Chinese with English abstract).
Fu, L., Tola, E., Al-Mallahi, A., Li, R., & Cui, Y. (2019). A novel image processing algorithm to separate linearly clustered kiwifruits. Biosystems Engineering, 183, 184–195. https://doi.org/10.1016/j.biosystemseng.2019.04.024.
Fu, L., Wang, B., Cui, Y., Su, S., Gejima, Y., & Kobayashi, T. (2015). Kiwifruit recognition at nighttime using artificial lighting based on machine vision. International Journal of Agricultural and Biological Engineering, 8(4), 52–59. https://doi.org/10.3965/j.ijabe.20150804.1576.
Gao, F., Fu, L., Zhang, X., Majeed, Y., Li, R., Karkee, M., et al. (2020). Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN. Computers and Electronics in Agriculture, 176, 105634. https://doi.org/10.1016/j.compag.2020.105634.
Gené-Mola, J., Sanz-Cortiella, R., Rosell-Polo, J. R., Morros, J.-R., Ruiz-Hidalgo, J., Vilaplana, V., et al. (2020). Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry. Computers and Electronics in Agriculture, 30, 105591. https://doi.org/10.1016/j.compag.2019.105165.
He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 27-30 June 2016
Hu, F., Shi, L., Li, R., Li, S., Li, X., Wang, X., et al. (2017). Fertilization evaluation of kiwifruit in Guanzhong region of Shaanxi province. Soils and Fertilizers Sciences in China, 54(3), 44–49. https://doi.org/10.11838/sfsc.20170308 (in Chinese with English abstract)
Huang, R., Pedoeem, J., Chen, C. (2018). YOLO-LITE: a real-time object detection algorithm optimized for non-GPU computers. https://arxiv.org/pdf/1811.05588v1.pdf. Accessed 30 August 2020.
Jia, W., Tian, Y., Luo, R., Zhang, Z., Lian, J., & Zheng, Y. (2020). Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot. Computers and Electronics in Agriculture, 172, 105380. https://doi.org/10.1016/j.compag.2020.105380.
Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019). Deep learning for real-time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’. Precision Agriculture, 20(6), 1107–1135. https://doi.org/10.1007/s11119-019-09642-0.
Lin, G., Tang, Y., Zou, X., Xiong, J., & Fang, Y. (2020). Color-, depth-, and shape-based 3D fruit detection. Precision Agriculture, 21(1), 1–17. https://doi.org/10.1007/s11119-019-09654-w.
Lin, M., Chen, Q., Yan, S. (2013). Network in network. https://arxiv.org/pdf/1312.4400.pdf. Accessed 30 August 2020.
Linker, R. (2018). Machine learning based analysis of night-time images for yield prediction in apple orchard. Biosystems Engineering, 167, 114–125. https://doi.org/10.1016/j.biosystemseng.2018.01.003.
Liu, G., Nouaze, J. C., Mbouembe, P. L. T., & Kim, J. H. (2020). YOLO-tomato: a robust algorithm for tomato detection based on YOLOv3. Sensors, 20(7), 2145. https://doi.org/10.3390/s20072145.
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C., et al. (2016). SSD: single shot multibox detector. In B. Leibe, et al. (Eds.), Lecture notes in computer science (pp. 21–37). Switzerland: Springer International Publishing.
Liu, X., Jia, W., Ruan, C., Zhao, D., Gu, Y., & Chen, W. (2018). The recognition of apple fruits in plastic bags based on block classification. Precision Agriculture, 19(4), 735–749. https://doi.org/10.1007/s11119-017-9553-2.
Lu, Y., Chen, Z., Kang, T., Zhang, X., Bellarby, J., & Zhou, J. (2016). Land-use changes from arable crop to kiwi-orchard increased nutrient surpluses and accumulation in soils. Agriculture Ecosystems and Environment, 223, 270–277. https://doi.org/10.1016/j.agee.2016.03.019.
Mu, L., Liu, H., Cui, Y., Fu, L., & Gejima, Y. (2018). Mechanized technologies for scaffolding cultivation in the kiwifruit industry: a review. Information Processing in Agriculture, 5(4), 401–410. https://doi.org/10.1016/j.inpa.2018.07.005.
Peng, H., Huang, B., Shao, Y., Li, Z., Zhang, C., Chen, Y., et al. (2018). General improved SSD model for picking object recognition of multiple fruits in natural environment. Transactions of the Chinese Society of Agricultural Engineering, 34(6), 155–162. https://doi.org/10.11975/j.issn.1002-6819.2018.16.020 (in Chinese with English abstract)
Pizer, S. M., Amburn, E. P., Austin, J. D., Cromartie, R., Geselowitz, A., Greer, T., et al. (1987). Adaptive histogram equalization and its variations. Computer vision, graphics, and image processing, 39(3), 355–368. https://doi.org/10.1016/S0734-189X(87)80186-X.
Redmon, J., Divvala, S., Girshick, R., Farhadi, A. (2016). You only look once: Unified, real-time object detection, In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 27–30 June 2016
Redmon, J., Farhadi, A. (2017). YOLO9000: Better, faster, stronger, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 21–26 July 2017
Redmon, J., Farhadi, A. (2018). YOLOv3: An incremental improvement. https://arxiv.org/pdf/1804.02767.pdf. Accessed 30 August 2020.
Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Roy, S., Haque, A., Neubert, J. (2018). Automatic diagnosis of melanoma from dermoscopic image using real-time object detection. In Annual Conference on Information Sciences and Systems, Princeton, USA, 21–23 March 2018
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., et al. (2015). ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252. https://doi.org/10.1007/s11263-015-0816-y.
Sa, I., Ge, Z., Dayoub, F., Upcroft, B., Perez, T., & McCool, C. (2016). Deepfruits: a fruit detection system using deep neural networks. Sensors, 16(8), 1222. https://doi.org/10.3390/s16081222.
Scarfe, A. J. (2012). Development of an autonomous kiwifruit harvester, PhD thesis. Manawatu, New Zealand: Massey University.
Shinde, S., Kothari, A., & Gupta, V. (2018). YOLO based human action recognition and localization. Procedia Computer Science, 133, 831–835. https://doi.org/10.1016/j.procs.2018.07.112.
Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 1–48. https://doi.org/10.1186/s40537-019-0197-0.
Silwal, A., Davidson, J. R., Karkee, M., Mo, C. K., Zhang, Q., & Lewis, K. (2017). Design, integration, and field evaluation of a robotic apple harvester. Journal of Field Robotics, 34, 1140–1159. https://doi.org/10.1002/rob.21715.
Simonyan, K., & Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. Advances in Neural Information Processing Systems, 1, 568–576. https://doi.org/10.1002/14651858.CD001941.pub3.
Smith, A. R. (1978). Color Gamut transform pairs. Computer Graphics (ACM), 12(3), 12–19. https://doi.org/10.1145/965139.807361.
Sun, C., Shrivastava, A., Singh, S., Gupta, A. (2017). Revisiting unreasonable effectiveness of data in deep learning era, In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 Oct 2017
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., et al. (2015). Going deeper with convolutions, In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Boston, USA, 7-12 June 2015
Tani, J., Mishra, S., & Wen, J. T. (2016). Motion blur-based state estimation. IEEE Transactions on Control Systems Technology, 24(3), 1012–1019. https://doi.org/10.1109/TCST.2015.2473004.
Taylor, L., Nitschke, G. (2018). Improving deep learning with generic data augmentation. https://arxiv.org/pdf/1708.06020.pdf. Accessed 30 August 2020.
Tian, Y., Yang, G., Wang, Z., Wang, H., Li, E., & Liang, Z. (2019). Apple detection during different growth stages in orchards using the improved YOLO-V3 model. Computers and Electronics in Agriculture, 157, 417–426. https://doi.org/10.1016/j.compag.2019.01.012.
UN Food & Agriculture Organization (2018). Production of kiwi (fruit) by countries. https://www.fao.org/faostat/en/#data. Accessed 30 August 2020.
Vasconez, J. P., Delpiano, J., Vougioukas, S., & Auat Cheein, F. (2020). Comparison of convolutional neural networks in fruit detection and counting: a comprehensive evaluation. Computers and Electronics in Agriculture, 173, 105348. https://doi.org/10.1016/j.compag.2020.105348.
Wang, C., Lee, W. S., Zou, X., Choi, D., Gan, H., & Diamond, J. (2018). Detection and counting of immature green citrus fruit based on the Local Binary Patterns (LBP) feature using illumination-normalized images. Precision Agriculture, 19(6), 1062–1083. https://doi.org/10.1007/s11119-018-9574-5.
Wang, R. (2017). Kiwifruit recognition in natural scene. Master thesis, Northwest Agriculture and Forestry University: Shanxi, China (in Chinese with English abstract)
Williams, H., Jones, M., Nejati, M., Seabright, M., Bell, J., Penhall, N., et al. (2019). Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms. Biosystems Engineering, 181, 140–156. https://doi.org/10.1016/j.biosystemseng.2019.03.007.
Williams, H., Ting, C., Nejati, M., Jones, M., Penhall, N., Lim, J. Y., et al. (2020). Improvements to and large-scale evaluation of a robotic kiwifruit harvester. Journal of Field Robotics, 37(2), 187–201. https://doi.org/10.1002/rob.21890.
Xue, Y., Huang, N., Tu, S., Mao, L., Yang, A., Zhu, X., et al. (2018). Immature mango detection based on improved YOLOv2. Transactions of the Chinese Society of Agricultural Engineering, 34(7), 173–179. https://doi.org/10.11975/j.issn.1002-6819.2018.07.022.
Yang, S., Zhang, J., Bo, C., Wang, M., & Chen, L. (2019). Fast vehicle logo detection in complex scenes. Optics and Laser Technology, 110, 196–201. https://doi.org/10.1016/j.optlastec.2018.08.007.
Yu, Y., Zhang, K., Yang, L., & Zhang, D. (2019). Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Computers and Electronics in Agriculture, 163, 104846. https://doi.org/10.1016/j.compag.2019.06.001.
Zhang, J., Huang, M., Jin, X., & Li, X. (2017a). A real-time Chinese traffic sign detection algorithm based on modified YOLOv2. Algorithms, 10(4), 1–13. https://doi.org/10.3390/a10040127.
Zhang, S., Benenson, R., Schiele, B. (2017a). CityPersons: A diverse dataset for pedestrian detection, In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, USA, 21–26 July 2017
Zhou, Y., Xu, T., Deng, H., & Miao, T. (2018). Real-time recognition of main organs in tomato based on channel wise group convolutional network. Transactions of the Chinese Society of Agricultural Engineering, 34(10), 153–162. https://doi.org/10.11975/j.issn.1002-6819.2018.10.019 (in Chinese with English abstract)
Zhuang, J., Luo, S., Hou, C., Tang, Y., He, Y., & Xue, X. (2018). Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications. Computers and Electronics in Agriculture, 152, 64–73. https://doi.org/10.1016/j.compag.2018.07.004.
Acknowledgments
The authors would like to express their deep gratitude to the “Young Faculty Study Abroad Program” of the Northwest A&F University Scholarship who sponsored Dr. Longsheng Fu in conducting post-doctoral research at the Centre for Precision and Automated Agricultural Systems, Washington State University; and to the Mexian Kiwifruit Experimentation Station of Northwest A&F University for providing the experimental orchard.
Funding
This work was supported by the Key Research and Development Program in Shaanxi Province of China (Grant Number 2018TSCXL-NY-05-04, 2019ZDLNY02-04); Fund of the Beijing Key Laboratory of Big Data Technology for Food Safety, Beijing Technology & Business University (Grant Number BTBD-2019KF03); the International Scientific and Technological Cooperation Foundation of Northwest A&F University (Grant Number A213021803).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Fu, L., Feng, Y., Wu, J. et al. Fast and accurate detection of kiwifruit in orchard using improved YOLOv3-tiny model. Precision Agric 22, 754–776 (2021). https://doi.org/10.1007/s11119-020-09754-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11119-020-09754-y