Skip to main content
SpringerLink
Log in

Segmentation and motion parameter estimation for robotic Medjoul-date thinning

  • Published:
Precision Agriculture Aims and scope Submit manuscript

Abstract

Laborious fruit thinning is required for attaining high-quality Medjoul dates. Thinning automation can significantly reduce labor and improve efficiency. An image processing apparatus developed for robotic Medjoul thinning is presented. Instance segmentation based on Mask R-CNN was applied to identify the fruit bunch components: spikelets and rachis. Motion planning parameters were extracted using the derived masks: rachis center point (RCP), rachis orientation angle, and spikelets remaining length. RCP and rachis orientation angle were computed geometrically, spikelets remaining length was estimated with a convolutional neural network (CNN) and a deep neural network (DNN). Instance segmentation results were accurate, especially for spikelets, for low intersection over union (IoU) (0.3 IoU, fruit determined for thinning identification, spikelets: 98%, rachises: 73%). However, only 66% of the rachises were correctly matched to spikelets. The segmentation of all spikelets and rachises in the images was of medium quality for low IoU (0.3 IoU, F1, spikelets: 0.67, rachis: 0.77), where both precision and recall dropped for higher IoUs. RCP and rachis orientation angle were accurately estimated (0.3 IoU, error, RCP: 2.2 cm, rachis orientation angle: 5.0°). Spikelets remaining length estimation using CNN resulted in better performance than DNN (0.3 IoU, error, CNN: 19.7%, DNN: 24.6%). Spikelets segmentation results are suitable for thinning automation. However, rachis segmentation and matching the rachis and spikelets may still require human intervention during run-time. RCP and rachis orientation angle estimation errors are acceptable, while spikelets remaining length estimation errors are acceptable only for preliminary motion planning and mandate additional tuning during motion execution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availability

The data collected within the current work is available online (Shoshan & Berman, 2021).

Code availability

The algorithms developed in the current work are based on available code as detailed in the text.

Notes

  1. Mask R-CNN pre-trained weights based on the ‘COCO’ dataset can be easily loaded into the model from https://github.com/matterport.

  2. https://www.tensorflow.org.

  3. https://keras.io.

  4. https://github.com/facebookresearch/Detectron.

  5. https://pytorch.org.

References

  • Abdulla, W. (2017). Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow. GitHub repository. Github. Retrieved September 15, 2020, from https://github.com/matterport/Mask_RCNN

  • Aljanobi, A. A., Al-Hamed, S. A., & Al-Suhaibani, S. A. (2010). A setup of mobile robotic unit for fruit harvesting. In 19th international workshop on robotics in Alpe-Adria-Danube Region (RAAD 2010), Budapest, 2010, pp. 105-108. https://doi.org/10.1109/RAAD.2010.5524602

  • Altaheri, H., Alsulaiman, M., & Muhammad, G. (2019). Date fruit classification for robotic harvesting in a natural environment using deep learning. IEEE Access, 7, 117115–117133. https://doi.org/10.1109/ACCESS.2019.2936536

    Article  Google Scholar 

  • Bar-Shira, O., Cohen, Y., Shoshan, T., Sadowky, A., Cohen, Y., Bechar, A., Berman, S. (2019). Learning motion parameters for a Medjoul-date thinning robot. In XXXVIII CIOSTA & CIGR section V conference.

  • Barth, R., IJsselmuiden, J., Hemming, J., & Van Henten, E. J. (2018). Data synthesis methods for semantic segmentation in agriculture: A Capsicum annuum dataset. Computers and Electronics in Agriculture, 144, 284–296. https://doi.org/10.1016/j.compag.2017.12.001

    Article  Google Scholar 

  • Bechar, A., Bloch, V., Finkelshtain, R., Levi, S., Hoffman, A., Egozi, H., & Schmilovitch, Z. (2014). Visual servoing methodology for selective tree pruning by human-robot collaborative system. In International conference of agricultural engineering (AgEng).

  • Bechar, A., & Vigneault, C. (2017). Agricultural robots for field operations. Part 2: Operations and systems. Biosystems Engineering, 153, 110–128. https://doi.org/10.1016/j.biosystemseng.2016.11.004

    Article  Google Scholar 

  • Bergstra, J., & Bengio, Y. (2012). Random search for hyper-parameter optimization. The Journal of Machine Learning Research, 13(1), 281–305.

    Google Scholar 

  • Bolya, D., Zhou, C., Xiao, F., & Lee, Y. J. (2019). YOLACT: Real-time instance segmentation, arXiv:1904.

  • Chao, C. T., & Krueger, R. R. (2007). The date palm (Phoenix dactylifera L.): overview of biology, uses, and cultivation. HortScience, 42(5), 1077–1082. https://doi.org/10.21273/HORTSCI.42.5.1077

    Article  Google Scholar 

  • Cohen, Y., Bar-Shira, O., & Berman, S. (2020). Motion adaptation based on learning the manifold of task and dynamic movement primitive parameters. Robotica. https://doi.org/10.1017/s0263574720001186

    Article  Google Scholar 

  • Cohen, Y., & Glasner, B. B. (2015). Date palm status and perspective in Israel. In J. M. Al-Khayri, S. M. Jain, & D. V. Johnson (Eds.), Date palm genetic resources, cultivar assessment, cultivation practices and novel products (pp. 265–298). Dordrecht: Springer. https://doi.org/10.1007/978-94-017-9707-8_8

    Chapter  Google Scholar 

  • Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., & Fei-Fei, L. (2009). Imagenet: A large-scale hierarchical image database. In IEEE computer society conference on computer vision and pattern recognition (CVPR 2009), 20–25 June 2009, Miami, Florida, USA https://doi.org/10.1109/CVPR.2009.5206848.‏

  • Duckett, T., Pearson, S., Blackmore, S., & Grieve, B. (2018). Agricultural robotics: The future of robotic agriculture. arXiv preprint arXiv:1806.06762

  • Eizicovits, D., van Tuijl, B., Berman, S., & Edan, Y. (2016). Integration of perception capabilities in gripper design using graspability maps. Biosystems Engineering, 146, 98–113. https://doi.org/10.1016/j.biosystemseng.2015.12.016

    Article  Google Scholar 

  • Forman, G., & Scholz, M. (2010). Apples-to-apples in cross-validation studies: Pitfalls in classifier performance measurement. Acm Sigkdd Explorations Newsletter, 12(1), 49–57. https://doi.org/10.1145/1882471.1882479

    Article  Google Scholar 

  • Glorot, X., Bordes, A., & Bengio, Y. (2011). Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics (pp. 315–323).‏ PMLR.

  • Gongal, A., Amatya, S., Karkee, M., Zhang, Q., & Lewis, K. (2015). Sensors and systems for fruit detection and localization: A review. Computers and Electronics in Agriculture, 116, 8–19. https://doi.org/10.1016/j.compag.2015.05.021

    Article  Google Scholar 

  • He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017). Mask R-CNN. In Proceedings of the IEEE international conference on computer vision (pp. 2961–2969).‏ New York: IEEE.‏

  • Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., & Schaal, S. (2013). Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25(2), 328–373.

    Article  Google Scholar 

  • Kamilaris, A., & Prenafeta-Boldú, F. X. (2018). A review of the use of convolutional neural networks in agriculture. The Journal of Agricultural Science, 156, 312–322. https://doi.org/10.1017/S0021859618000436

    Article  Google Scholar 

  • Kapach, K., Barnea, E., Mairon, R., Edan, Y., & Ben-Shahar, O. (2012). Computer vision for fruit harvesting robots—State of the art and challenges ahead. International Journal of Computational Vision and Robotics. https://doi.org/10.1504/IJCVR.2012.046419

    Article  Google Scholar 

  • Kestur, R., Meduri, A., & Narasipura, O. (2019). MangoNet: A deep semantic segmentation architecture for a method to detect and count mangoes in an open orchard. Engineering Applications of Artificial Intelligence, 77, 59–69. https://doi.org/10.1016/j.engappai.2018.09.011

    Article  Google Scholar 

  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.‏

  • Koirala, A., Walsh, K. B., Wang, Z., & McCarthy, C. (2019). Deep learning for real-time fruit detection and orchard fruit load estimation: Benchmarking of ‘MangoYOLO.’ Precision Agriculture, 20, 1107–1135. https://doi.org/10.1007/s11119-019-09642-0

    Article  Google Scholar 

  • Legland, D. (2014). Feret diameter and oriented box. MATLAB Central File Exchange. Retrieved September 15, 2020, from https://www.mathworks.com/matlabcentral/fileexchange/30402-feret-diameter-and-oriented-box

  • Lin, T. Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014). Microsoft COCO: Common objects in context. In D. Fleet, T. Pajdla, B. Schiele, & T. Tuytelaars (Eds.), Computer vision—ECCV 2014. Lecture Notes in Computer Science. (Vol. 8693). Cham: Springer. https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  • Lin, G., Tang, Y., Zou, X., Cheng, J., & Xiong, J. (2020). Fruit detection in natural environment using partial shape matching and probabilistic Hough transform. Precision Agriculture, 21, 160–177. https://doi.org/10.1007/s11119-019-09662-w

    Article  Google Scholar 

  • Linker, R., & Neer, O. (2014). Tree skeleton extraction by close-range laser telemetry as first step toward robotic pruning. In International conference of agricultural engineering (AgEng).

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60, 91–110. https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  • Marani, R., Milella, A., Petitti, A., & Reina, G. (2021). Deep neural networks for grape bunch segmentation in natural images from a consumer-grade camera. Precision Agriculture, 22, 387–413. https://doi.org/10.1007/s11119-020-09736-0

    Article  Google Scholar 

  • Morris, J. R. (2007). Development and commercialization of a complete vineyard mechanization system. HortTechnology, 17(4), 411–420. https://doi.org/10.21273/HORTTECH.17.4.411

    Article  Google Scholar 

  • Mortensen, A. K., Dyrmann, M., Karstoft, H., Jørgensen, R. N., & Gislum, R. (2016). Semantic segmentation of mixed crops using deep convolutional neural network. In Proceedings of the International Conference of Agricultural Engineering (CIGR).‏

  • Qian, N. (1999). On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1), 145–151. https://doi.org/10.1016/S0893-6080(98)00116-6

    Article  CAS  PubMed  Google Scholar 

  • Razzaghi, E., Massah, J., & Vakilian, K. A. (2015). Mechanical analysis of a robotic date harvesting manipulator. Russian Agricultural Sciences, 41(1), 80–85. https://doi.org/10.3103/S1068367415010176

    Article  Google Scholar 

  • Redmon, J., Divvala, S., Girshick, R., Farhadi, A. (2016). You only look once: Unified, real-time object detection. In IEEE conference on computer vision and pattern recognition (CVPR) (pp. 779–788), https://doi.org/10.1109/CVPR.2016.91

  • Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031

    Article  PubMed  Google Scholar 

  • Rosa, U. A., Cheetancheri, K. G., Gliever, C. J., Lee, S. H., Thompson, J., & Slaughter, D. C. (2008). An electro-mechanical limb shaker for fruit thinning. Computers and Electronics in Agriculture, 61(2), 213–221. https://doi.org/10.1016/j.compag.2007.11.008

    Article  Google Scholar 

  • Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747.‏

  • Saleem, M. H., Potgieter, J., & Arif, K. M. (2021). Automation in agriculture by machine and deep learning techniques: A review of recent developments. Precision Agriculture. https://doi.org/10.1007/s11119-021-09806-x

    Article  Google Scholar 

  • Santos, T. T., de Souza, L. L., dos Santos, A. A., & Avila, S. (2020). Grape detection, segmentation, and tracking using deep neural networks and three-dimensional association. Computers and Electronics in Agriculture, 170, 105247. https://doi.org/10.1016/j.compag.2020.105247

    Article  Google Scholar 

  • Shapiro, A., Korkidi, E., Demri, A., Ben-Shahar, O., Riemer, R., & Edan, Y. (2009). Toward elevated agrobotics: Development of a scaled-down prototype for visually guided date palm tree sprayer. Journal of Field Robotics, 26(6–7), 572–590. https://doi.org/10.1002/rob.20299

    Article  Google Scholar 

  • Shin, H. C., Roth, H. R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., & Summers, R. M. (2016). Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging, 35(5), 1285–1298. https://doi.org/10.1109/TMI.2016.2528162

    Article  PubMed  Google Scholar 

  • Shoshan, T., Berman, S. (2021), Medjoul-date images—Annotated, Mendeley Data, V1, https://doi.org/10.17632/k7xk2nwgrh.1

  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.‏

  • Tian, Y., Yang, G., Wang, Z., Li, E., & Liang, Z. (2020). Instance segmentation of apple flowers using the improved mask R-CNN model. Biosystems Engineering, 193, 264–278. https://doi.org/10.1016/j.biosystemseng.2020.03.008

    Article  Google Scholar 

  • Tu, S., Pang, J., Liu, H., Zhuang, N., Chen, Y., Zheng, C., Wan, H., & Xue, Y. (2020). Passion fruit detection and counting based on multiple scale faster R-CNN using RGB-D images. Precision Agriculture, 21, 1072–1091. https://doi.org/10.1007/s11119-020-09709-3

    Article  Google Scholar 

  • Wibisono, J. K., & Hang, H. M. (2017). RGBD image segmentation using deep edge. In 2017 international symposium on intelligent signal processing and communication systems (ISPACS) (pp. 565–569), Xiamen. https://doi.org/10.1109/ISPACS.2017.8266542

  • Zapotezny-Anderson, P., & Lehnert, C. (2019). Towards active robotic vision in agriculture: a deep learning approach to visual servoing in occluded and unstructured protected cropping environments. IFAC-PapersOnLine, 52(30), 120–125. https://doi.org/10.1016/j.ifacol.2019.12.508

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Inbar Ben-David and Dr. Yael Salzer, Dr. Zeev Schmilovitz, and Dekel Meir from the ARO-Volcani Institute for their assistance in the data collection effort, Prof. Yossi Yovel from Tel-Aviv University for his insightful comments, Noam Peles, Nissim Abuhazera, Yossi Zahavi, and Moshe Bardea from Ben-Gurion University for their technical assistance, and the Israeli date growers for their assistance in various stages of the research.

Funding

This research is funded by the Chief Scientist of the Israeli ministry of agriculture and the Israeli Date Grower’s board in the Plant Council (Project # 20-07-0018).

Author information

Authors and Affiliations

  1. Department of Industrial Engineering and Management, Ben-Gurion University of the Negev, Beer-Sheva, Israel

    Tal Shoshan & Sigal Berman

  2. The Institute of Agricultural Engineering, Agricultural Research Organization, The Volcani Institute, Rishon LeZion, Israel

    Avital Bechar

  3. The Institute of Plant Sciences, Agricultural Research Organization, The Volcani Institute, Rishon LeZion, Israel

    Yuval Cohen

  4. Southern Arava R&D, Eilot Region, Israel

    Avraham Sadowsky

Authors
  1. Tal Shoshan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Avital Bechar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yuval Cohen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Avraham Sadowsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sigal Berman
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TS, AB and SB, conceptualized the project and the methodology. All authors contributed to the data collection effort. TS and SB developed the algorithms and performed the data analysis. YC and AS provided consultation regarding Medjoul dates and their thinning operations. TS and SB wrote the initial draft of the manuscript. All authors contributed to the review and editing of the final draft of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Sigal Berman.

Ethics declarations

Conflict of interest

The authors have no conflict of interest to declare which are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shoshan, T., Bechar, A., Cohen, Y. et al. Segmentation and motion parameter estimation for robotic Medjoul-date thinning. Precision Agric 23, 514–537 (2022). https://doi.org/10.1007/s11119-021-09847-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11119-021-09847-2

Keywords

Search

Navigation