Abstract
To ensure litchi fruit yield and quality, reasonable blooming period management such as flower thinning is required during the early flowering period. A combination of the number of litchi flowers and their density map can provide a reference for blooming period management decisions during the flowering period. Flowering intensity is currently largely estimated manually by humans observing the trees in the orchard. Although some automatic computer vision systems have been proposed for estimating flowering intensity, their overall performance is inadequate to meet current flower-thinning needs, and these systems are weak in some situations such as in varying environments and when the target object has a low-density distribution. With male litchi flowers as the research object, the goal of this study was to design a method for calculating the number of flowers. By using an image of the male litchi flower as input to a multicolumn convolutional neural network, a final density map and the number of flowers were generated. Experimental results using a self-constructed male litchi flower dataset demonstrated the feasibility of outputting a density map and flower count. The flower number was estimated from the model with a mean absolute error (MAE) that reached 16.29 and a mean square error (MSE) reaching 25.40 on the test set, which was better than counting by target detection. The proposed method can be used to perform time-saving analyses to help estimate yield and implement follow-up orchard management, and it demonstrates the potential of using density maps as outputs for estimating flowering intensity.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adamsen, F. J., Coffelt, T. A., Nelson, J. M., Barnes, E. M., & Rice, R. C. (2000). Method for using images from a color digital camera to estimate flower number. Crop Science, 40(3), 704–709. https://doi.org/10.2135/cropsci2000.403704x
Aggelopoulou, A. D., Bochtis, D., Fountas, S., Swain, K. C., Gemtos, T. A., & Nanos, G. D. (2011). Yield prediction in apple orchards based on image processing. Precision Agriculture, 12(3), 448–456. https://doi.org/10.1007/s11119-010-9187-0
Bochkovskiy, A., Wang, C. Y., & Liao, H. Y. M. (2020). YOLOv4: Optimal speed and accuracy of object detection. arXiv preprint. arXiv:2004.10934
Boominathan, L., Kruthiventi, S. S., & Babu, R. V. (2016). CrowdNet: A deep convolutional network for dense crowd counting. In Proceedings of the 24th ACM international conference on Multimedia. https://doi.org/10.1145/2964284.2967300
Cai, C., Chen, J., Ou, L., Xi, X., & Sun, Q. (2011). Study on Litchi blooming habits. Guangdong Agricultural Sciences, 38(21), 20–24. https://doi.org/10.16768/j.issn.1004-874x.2011.21.059
Ciregan, D., Meier, U., & Schmidhuber, J. (2012). Multi-column deep neural networks for image classification. 2012 IEEE Conference on Computer Vision and Pattern Recognition (3642–3649). https://doi.org/10.1109/CVPR.2012.6248110
Comas, A., Valente, J., & Kooistra, L. (2019). Automatic apple tree blossom estimation from UAV rgb imagery. International Archives of the Photogrammetry, Remote Sensing & Spatial Information Sciences, XLII-2/W13, 631–635. https://doi.org/10.5194/isprs-archives-XLII-2-W13-631-2019
Dias, P. A., Tabb, A., & Medeiros, H. (2018). Apple flower detection using deep convolutional networks. Computers in Industry, 99, 17–28. https://doi.org/10.1016/j.compind.2018.03.010
Farjon, G., Krikeb, O., Hillel, A. B., & Alchanatis, V. (2020). Detection and counting of flowers on apple trees for better chemical thinning decisions. Precision Agriculture, 21, 503–521. https://doi.org/10.1007/s11119-019-09679-1
Feng, D., Haase-Schütz, C., Rosenbaum, L., Hertlein, H., Glaeser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021). Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges. IEEE Transactions on Intelligent Transportation Systems. https://doi.org/10.1109/TITS.2020.2972974
Hočevar, M., Širok, B., Godeša, T., & Stopar, M. (2014). Flowering estimation in apple orchards by image analysis. Precision Agriculture, 15(4), 466–478. https://doi.org/10.1007/s11119-013-9341-6
Horton, R., Cano, E., Bulanon, D., & Fallahi, E. (2017). Peach flower monitoring using aerial multispectral imaging. Journal of Imaging. https://doi.org/10.3390/jimaging3010002
Huang, X., & Chen, H. (2014). Studies on shoot, flower and fruit development in litchi and strategies for improved litchi production. Acta Horticulturae, 1029, 127–136. https://doi.org/10.17660/ActaHortic.2014.1029.14
Jiang, S., Xu, H., Wang, H., Hu, G., Li, J., Chen, H., & Huang, X. (2012). A comparison of the costs of flowering in “Feizixiao” and “Baitangying” litchi. Scientia Horticulturae, 148, 118–125. https://doi.org/10.1016/j.scienta.2012.09.035
Kentaro, W. (2016) labelme: Image polygonal annotation with python. Git Code. Retrieved from https://github.com/wkentaro/labelme
Koen, B. V. (1988). Toward a definition of the engineering method. European Journal of Engineering Education, 13(3), 307–315. https://doi.org/10.1080/03043798808939429
Lempitsky, V., & Zisserman, A. (2010). Learning to count objects in images. Advances in Neural Information Processing Systems, 23, 1324–1332. https://doi.org/10.5555/2997189.2997337
Li, Y., Zhang, X., & Chen, D. (2018). CSRNet: Dilated convolutional neural networks for understanding the highly congested scenes. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (1091–1100). https://doi.org/10.1109/CVPR.2018.00120
Lin, P., Lee, W. S., Chen, Y. M., Peres, N., & Fraisse, C. (2020). A deep-level region-based visual representation architecture for detecting strawberry flowers in an outdoor field. Precision Agriculture, 21, 387–402. https://doi.org/10.1007/s11119-019-09673-7
Link, H. (2000). Significance of flower and fruit thinning on fruit quality. Plant Growth Regulation, 31(1–2), 17–26. https://doi.org/10.1023/A:1006334110068
Liu, S., Li, X., Wu, H., Xin, B., Tang, J., Petrie, P. R., & Whitty, M. (2018). A robust automated flower estimation system for grape vines. Biosystems Engineering, 172, 110–123. https://doi.org/10.1016/j.biosystemseng.2018.05.009
Liu, Y., Wang, K. J., & Xie, R. J. (2016). Estimating the number of apple tree flowers based on hyperspectral information of a canopy. Scientia Agricultura Sinica, 49(18), 3608–3617. https://doi.org/10.3864/j.issn.0578-1752.2016.18.015
Luo, J., He, F., Wang, X., Hu, F., Wang, Z., & Li, J. (2019). Analyzing the physiological causes of the chemical combined with hand and machine flower-thinning technique to increase Feizixiao litchi yield. South China Fruits, 48(01), 20–24. https://doi.org/10.13938/j.issn.1007-1431.20180126
Mavridou, E., Vrochidou, E., Papakostas, G. A., Pachidis, T., & Kaburlasos, V. G. (2019). Machine vision systems in precision agriculture for crop farming. Journal of Imaging, 5(12), 89. https://doi.org/10.3390/jimaging5120089
Qi, W., Chen, H., Luo, T., & Song, F. (2019). Development status, trend and suggestion of litchi industry in mainland China. Guangdong Agricultural Sciences, 46(10), 132–139. https://doi.org/10.16768/j.issn.1004-874X.2019.10.020
Redmon, J., & Farhadi, A. (2018). YOLOv3: An incremental improvement. Retrieved August 30, 2020, from https://arxiv.org/pdf/1804.02767.pdf
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 91–99.
Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data. https://doi.org/10.1186/s40537-019-0197-0
Srinivas, S., Sarvadevabhatla, R. K., Mopuri, K. R., Prabhu, N., Kruthiventi, S. S., & Babu, R. V. (2017). An introduction to deep convolutional neural nets for computer vision. Deep Learning for Medical Image Analysis. https://doi.org/10.1016/B978-0-12-810408-8.00003-1
Thorp, K. R., & Dierig, D. A. (2011). Color image segmentation approach to monitor flowering in lesquerella. Industrial Crops and Products, 34(1), 1150–1159. https://doi.org/10.1016/j.indcrop.2011.04.002
Viola, P., Jones, M. J., & Snow, D. (2005). Detecting pedestrians using patterns of motion and appearance. International Journal of Computer Vision, 63, 153–161. https://doi.org/10.1007/s11263-005-6644-8
Wu, Y., & Jiang, M. (2018). Face recognition system based on CNN and LBP features for classifier optimization and fusion. The Journal of China Universities of Posts and Telecommunications, 25(01), 37–47. https://doi.org/10.19682/j.cnki.1005-8885.2018.0004
Xiao, C., Zheng, L., & Sun, H. (2014). Estimation of the apple flowers based on aerial multispectral image. 2014 ASABE Annual International Meeting. https://doi.org/10.13031/aim.20141912593
Xu, R., Li, C., Paterson, A. H., Jiang, Y., Sun, S., & Robertson, J. S. (2018). Aerial images and convolutional neural network for cotton bloom detection. Frontiers in Plant Science, 8, 2235. https://doi.org/10.3389/fpls.2017.02235
Yang, B., Li, G., Yang, S., He, Z., Zhou, C., & Yao, L. (2015). Effect of application ratio of potassium over nitrogen on litchi fruit yield, quality, and storability. Hortscience a: Publication of the American Society for Horticultural Science, 50(6), 916–920. https://doi.org/10.21273/HORTSCI.50.6.916
Zeiler, M. D., Ranzato, M., Monga, R., Mao, M., Yang, K., Le, Q. V., Nguyen, P., Senior, A., Vanhoucke, V., Dean, J., & Hinton, G. E. (2013). On rectified linear units for speech processing. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (3517–3521). https://doi.org/10.1109/ICASSP.2013.6638312
Zhang, Y., Zhou, D., Chen, S., Gao, S., & Ma, Y. (2016). Single-image crowd counting via multi-column convolutional neural network. 2016 IEEE Conference on Computer Vision and Pattern Recognition. https://doi.org/10.1109/CVPR.2016.70
Zhu, X. (2020). Analysis of suitable climate resources for the growth of high quality litchi in Yulin City. Journal of Agricultural Catastrophology, 10(03), 126–127. https://doi.org/10.19383/j.cnki.nyzhyj.2020.03.052
Acknowledgements
This research was supported by the earmarked fund for the Special Project of the Rural Vitalization Strategy of Guangdong Academy of Agricultural Sciences (Accession No. TS-1-4), the Key-Area Research and Development Program of Guangdong Province (Accession No. 2019B020223002), and the China Agriculture Research System (Accession No. CARS-32-14).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Lin, J., Li, J., Yang, Z. et al. Estimating litchi flower number using a multicolumn convolutional neural network based on a density map. Precision Agric 23, 1226–1247 (2022). https://doi.org/10.1007/s11119-022-09882-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11119-022-09882-7