Abstract
Plant height, a key agronomic trait, affects crop structure, photosynthesis, and thus the final yield and seed quality. The combination of digital cameras on unmanned aerial vehicles (UAVs) and use of structure from motion have enabled high-throughput crop canopy height estimation. However, the focus of prior research has mainly been on plot-level height prediction, neglecting precise estimations for individual plants. This study aims to explore the potential of UAV RGB images with mask region-based convolutional neural network (Mask-RCNN) for high-throughput phenotyping of individual-level height (IH) in oilseed rape at different growth stages. Field-measured height (FH) of nine sampling plants in each subplot of the 150 subplots was obtained by manual measurement after the UAV flight. An instance segmentation model for oilseed rape with data augmentation based on the Mask-RCNN model was developed. The IHs were then used to obtain plot-level height based on individual-level height (PHIH). The results show that Mask-RCNN performed better than the conventional Otsu method with the F1 score increased by 60.8% and 26.6% under high and low weed pressure, respectively. The trained model with data augmentation achieved accurate crop height estimation based on overexposed and underexposed UAV images, indicating the model’s applicability in practical scenarios. The PHIH can be predicted with the determination coefficient (r2) of 0.992, root mean square error (RMSE) of 4.03 cm, relative root mean square error (rRMSE) of 7.68%, which outperformed the results in the reported studies, especially in the late bolting stage. The IHs of the whole growth stages of oilseed can be predicted by this method with an r2 of 0.983, RMSE of 2.60 cm, and rRMSE of 7.14%. Furthermore, this method enabled a comprehensive Genome-wide association study (GWAS) in a 293-accession genetic population. The GWAS identified 200 and 65 statistically significant single nucleotide polymorphisms (SNPs), which were tightly associated with 28 and 11 candidate genes, at the late bolting and flowering stages, respectively. These findings demonstrated that the proposed method is promising for accurate estimations of IHs in oilseed rape as well as exploring the variations within the subplot, thus providing great potential for high-throughput plant phenotyping in crop breeding.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abdalla, A., Cen, H., Wan, L., Mehmood, K., & He, Y. (2021). Nutrient status diagnosis of infield oilseed rape via deep learning-enabled dynamic model. IEEE Transactions on Industrial Informatics, 17(6), 4379–4389. https://doi.org/10.1109/tii.2020.3009736
Albert, R., & Barabasi, A. L. (2002). Statistical mechanics of complex networks. Reviews of Modern Physics, 74(1), 47–97. https://doi.org/10.1103/RevModPhys.74.47.
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaria, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021). Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. Journal of Big Data, 8(1), 53. https://doi.org/10.1186/s40537-021-00444-8.
Ariza-Sentis, M., Velez, S., & Valente, J. (2023). Dataset on UAV RGB videos acquired over a vineyard including bunch labels for object detection and tracking. Data in Brief, 46, 108848. https://doi.org/10.1016/j.dib.2022.108848.
Attia, Z., Pogoda, C. S., Reinert, S., Kane, N. C., & Hulke, B. S. (2021). Breeding for sustainable oilseed crop yield and quality in a changing climate. Theoretical and Applied Genetics, 134(6), 1817–1827. https://doi.org/10.1007/s00122-021-03770-w.
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). SegNet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495. https://doi.org/10.1109/tpami.2016.2644615
Bendig, J., Yu, K., Aasen, H., Bolten, A., Bennertz, S., Broscheit, J., Gnyp, M. L., & Bareth, G. (2015). Combining UAV-based plant height from crop surface models, visible, and near infrared vegetation indices for biomass monitoring in barley. International Journal of Applied Earth Observation and Geoinformation, 39, 79–87. https://doi.org/10.1016/j.jag.2015.02.012.
Cen, H., Wan, L., Zhu, J., Li, Y., Li, X., Zhu, Y., Weng, H., Wu, W., Yin, W., Xu, C., Bao, Y., Feng, L., Shou, J., & He, Y. (2019). Dynamic monitoring of biomass of rice under different nitrogen treatments using a lightweight UAV with dual image-frame snapshot cameras. Plant Methods, 15, 32. https://doi.org/10.1186/s13007-019-0418-8.
Fan, Y., Feng, H., Jin, X., Yue, J., Liu, Y., Li, Z., Feng, Z., Song, X., & Yang, G. (2022). Estimation of the nitrogen content of potato plants based on morphological parameters and visible light vegetation indices. Frontiers in Plant Science, 13, 1012070. https://doi.org/10.3389/fpls.2022.1012070.
Feldman, A., Wang, H., Fukano, Y., Kato, Y., Ninomiya, S., & Guo, W. (2021). EasyDCP: An affordable, high-throughput tool to measure plant phenotypic traits in 3D. Methods in Ecology and Evolution, 12(9), 1679–1686. https://doi.org/10.1111/2041-210x.13645.
Ghaderizadeh, S., Abbasi-Moghadam, D., Sharifi, A., Tariq, A., & Qin, S. (2022). Multiscale dual-branch residual spectral-spatial network with attention for hyperspectral image classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15, 5455–5467. https://doi.org/10.1109/jstars.2022.3188732
He, K., Gkioxari, G., Dollar, P., & Girshick, R. (2020). Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2), 386–397. https://doi.org/10.1109/tpami.2018.2844175
Holman, F. H., Riche, A. B., Michalski, A., Castle, M., Wooster, M. J., & Hawkesford, M. J. (2016). High throughput field phenotyping of wheat plant height and growth rate in field plot trials using UAV based remote sensing. Remote Sensing, 8(12), 1031. https://doi.org/10.3390/rs8121031
Hu, X., Sun, L., Gu, X., Sun, Q., Wei, Z., Pan, Y., & Chen, L. (2021). Assessing the self-recovery ability of maize after lodging using UAV-LiDAR data. Remote Sensing, 13(12), 2270. https://doi.org/10.3390/rs13122270
Hu, J., Chen, B., Zhao, J., Zhang, F., Xie, T., Xu, K., Gao, G., Yan, G., Li, H., Li, L., Ji, G., An, H., Li, H., Huang, Q., Zhang, M., Wu, J., Song, W., Zhang, X., Luo, Y., Pires, J. C., Batley, J., Tian, S., & Wu, X. (2022). Genomic selection and genetic architecture of agronomic traits during modern rapeseed breeding. Nature Genetics, 54(5), 694–704. https://doi.org/10.1038/s41588-022-01055-6.
Kang, H. M., Sul, J. H., Service, S. K., Zaitlen, N. A., Kong, S., Freimer, N. B., Sabatti, C., & Eskin, E. (2010). Variance component model to account for sample structure in genome-wide association studies. Nature Genetics, 42(4), 348–U110. https://doi.org/10.1038/ng.548.
Klapste, P., Fogl, M., Bartak, V., Gdulova, K., Urban, R., & Moudry, V. (2020). Sensitivity analysis of parameters and contrasting performance of ground filtering algorithms with UAV photogrammetry-based and LiDAR point clouds. International Journal of Digital Earth, 13(12), 1672–1694. https://doi.org/10.1080/17538947.2020.1791267.
Li, D., Li, J., Xiang, S., & Pan, A. (2022). PSegNet: simultaneous semantic and instance segmentation for point clouds of plants. Plant Phenomics. https://doi.org/10.34133/2022/9787643
Liu, S., Yin, D., Feng, H., Li, Z., Xu, X., Shi, L., & Jin, X. (2022). Estimating maize seedling number with UAV RGB images and advanced image processing methods. Precision Agriculture, 23(5), 1604–1632. https://doi.org/10.1007/s11119-022-09899-y.
Long, Z., Tu, M., Xu, Y., Pak, H., Zhu, Y., Dong, J., Lu, Y., & Jiang, L. (2023). Genome-wide-association study and transcriptome analysis reveal the genetic basis controlling the formation of leaf wax in Brassica napus. Journal of Experimental Botany, 74(8), 2726–2739. https://doi.org/10.1093/jxb/erad047
Lu, W., Okayama, T., & Komatsuzaki, M. (2022). Rice height monitoring between different estimation models using UAV photogrammetry and multispectral technology. Remote Sensing, 14(1), 78. https://doi.org/10.3390/rs14010078
Luo, S., Liu, W., Zhang, Y., Wang, C., Xi, X., Nie, S., Ma, D., Lin, Y., & Zhou, G. (2021). Maize and soybean heights estimation from unmanned aerial vehicle (UAV) LiDAR data. Computers and Electronics in Agriculture, 182, 106005. https://doi.org/10.1016/j.compag.2021.106005.
Ma, L., Liu, Y., Zhang, X., Ye, Y., Yin, G., & Johnson, B. A. (2019). Deep learning in remote sensing applications: A meta-analysis and review. ISPRS Journal of Photogrammetry and Remote Sensing, 152, 166–177. https://doi.org/10.1016/j.isprsjprs.2019.04.015.
Muangprakhon, R., & Kaewplang, S. J. E. A. (2021). Estimation of paddy rice plant height using UAV remote sensing. Engineering Access, 7(2), 93–97. https://doi.org/10.14456/mijet.2021.14.
Osco, L. P., Junior, J. M., Ramos, A. P. M., Furuya, D. E. G., Santana, D. C., Teodoro, L. P. R., Gonçalves, W. N., Baio, F. H. R., Pistori, H., & Junior, C. A. S. (2020). Leaf nitrogen concentration and plant height prediction for maize using UAV-based multispectral imagery and machine learning techniques. Remote Sensing, 12(19), 3237. https://doi.org/10.3390/rs12193237.
Otsu, N. (1979). Threshold selection method from gray-level histograms. IEEE Transactions on Systems Man and Cybernetics, 9(1), 62–66. https://doi.org/10.1109/tsmc.1979.4310076
Ren, S., He, K., Girshick, R., & Sun, J. (2017). Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6), 1137–1149. https://doi.org/10.1109/tpami.2016.2577031
Revenga, J. C., Trepekli, K., Oehmcke, S., Jensen, R., Li, L., Igel, C., Gieseke, F. C., & Friborg, T. (2022). Above-ground biomass prediction for croplands at a sub-meter resolution using UAV-LiDAR and machine learning methods. Remote Sensing, 14(16), 3912. https://doi.org/10.3390/rs14163912
Safonova, A., Guirado, E., Maglinets, Y., Alcaraz-Segura, D., & Tabik, S. (2021). Olive tree biovolume from UAV multi-resolution image segmentation with mask R-CNN. Sensors (Basel, Switzerland), 21(5), 1617. https://doi.org/10.3390/s21051617
Schirrmann, M., Hamdorf, A., Giebel, A., Gleiniger, F., Pflanz, M., & Dammer, K. H. (2017). Regression kriging for improving crop height models fusing ultra-sonic sensing with UAV imagery. Remote Sensing, 9(7), 665. https://doi.org/10.3390/rs9070665
Selzner, T., Horn, J., Landl, M., Pohlmeier, A., Helmrich, D., Huber, K., Vanderborght, J., Vereecken, H., Behnke, S., & Schnepf, A. (2023). 3D U-net segmentation improves root system reconstruction from 3D MRI images in automated and manual virtual reality work flows. Plant Phenomics (Washington D C), 5, 0076. https://doi.org/10.34133/plantphenomics.0076
Shi, J., Li, R., Qiu, D., Jiang, C., Long, Y., Morgan, C., Bancroft, I., Zhao, J., & Meng, J. (2009). Unraveling the complex trait of crop yield with quantitative trait loci mapping in Brassica napus. Genetics, 182(3), 851–861. https://doi.org/10.1534/genetics.109.101642
Shi, Y., Zhang, Y., Sun, Y., Xie, Z., Luo, Y., Long, Q., Feng, J., Liu, X., Wang, B., He, D., Ren, J., Guo, P., Xing, J., He, L., Fernie, A. R., Chen, W., Liu, X., Luo, Y., Jin, C., & Luo, J. (2023). Natural variations of OsAUX5, a target gene of OsWRKY78, control the neutral essential amino acid content in rice grains. Molecular Plant, 16(2), 322–336. https://doi.org/10.1016/j.molp.2022.12.013
Sun, Y., Luo, Y., Zhang, Q., Xu, L., Wang, L., & Zhang, P. (2022). Estimation of crop height distribution for mature rice based on a moving surface and 3D point cloud elevation. Agronomy, 12(4), 836. https://doi.org/10.3390/agronomy12040836
Taddia, Y., Stecchi, F., & Pellegrinelli, A. (2020). Coastal mapping using DJI phantom 4 RTK in Post-processing kinematic mode. Drones, 4(2), 9. https://doi.org/10.3390/drones4020009
Wan, L., Li, Y., Cen, H., Zhu, J., Yin, W., Wu, W., Zhu, H., Sun, D., Zhou, W., & He, Y. (2018). Combining UAV-based vegetation indices and image classification to estimate flower number in oilseed rape. Remote Sensing, 10(9), 1484. https://doi.org/10.3390/rs10091484
Wan, L., Cen, H., Zhu, J., Zhang, J., Zhu, Y., Sun, D., Du, X., Zhai, L., Weng, H., & Li, Y. (2020). Grain yield prediction of rice using multi-temporal UAV-based RGB and multispectral images and model transfer—A case study of small farmlands in the South of China. Agricultural and Forest Meteorology, 291, 108096. https://doi.org/10.1016/j.agrformet.2020.108096
Wan, L., Zhang, J., Dong, X., Du, X., Zhu, J., Sun, D., Liu, Y., He, Y., & Cen, H. (2021). Unmanned aerial vehicle-based field phenotyping of crop biomass using growth traits retrieved from PROSAIL model. Computers and Electronics in Agriculture, 187, 106304. https://doi.org/10.1016/j.compag.2021.106304.
Wang, W., Guo, W., Le, L., Yu, J., Wu, Y., Li, D., Wang, Y., Wang, H., Lu, X., Qiao, H., Gu, X., Tian, J., Zhang, C., & Pu, L. (2023). Integration of high-throughput phenotyping, GWAS, and predictive models reveals the genetic architecture of plant height in maize. Molecular Plant, 16(2), 354–373. https://doi.org/10.1016/j.molp.2022.11.016.
Wu, D., Liang, Z., Yan, T., Xu, Y., Xuan, L., Tang, J., Zhou, G., Lohwasser, U., Hua, S., Wang, H., Chen, X., Wang, Q., Zhu, L., Maodzeka, A., Hussain, N., Li, Z., Li, X., Shamsi, I. H., Jilani, G., … Jiang, L. (2019). Whole-genome resequencing of a worldwide collection of rapeseed accessions reveals the genetic basis of ecotype divergence. Molecular Plant, 12(1), 30–43. https://doi.org/10.1016/j.molp.2018.11.007
Wulfmeyer, V., Hardesty, R. M., Turner, D. D., Behrendt, A., Cadeddu, M. P., Di Girolamo, P., Schluessel, P., Van Baelen, J., & Zus, F. (2015). A review of the remote sensing of lower tropospheric thermodynamic profiles and its indispensable role for the understanding and the simulation of water and energy cycles. Reviews of Geophysics, 53(3), 819–895. https://doi.org/10.1002/2014rg000476.
Xie, Y., Tian, J., & Zhu, X. X. (2020). Linking points with labels in 3D: A review of point cloud semantic segmentation. IEEE Geoscience and Remote Sensing Magazine, 8(4), 38–59. https://doi.org/10.1109/mgrs.2019.2937630
Xie, T., Li, J., Yang, C., Jiang, Z., Chen, Y., Guo, L., & Zhang, J. (2021). Crop height estimation based on UAV images: Methods, errors, and strategies. Computers and Electronics in Agriculture, 185, 106155. https://doi.org/10.1016/j.compag.2021.106155
Xuan, L., Yan, T., Lu, L., Zhao, X., Wu, D., Hua, S., & Jiang, L. (2020). Genome-wide association study reveals new genes involved in leaf trichome formation in polyploid oilseed rape (Brassica napus L). Plant Cell and Environment, 43(3), 675–691. https://doi.org/10.1111/pce.13694
Yang, Z., Wang, S., Wei, L., Huang, Y., Liu, D., Jia, Y., Luo, C., Lin, Y., Liang, C., Hu, Y., Dai, C., Guo, L., Zhou, Y., & Yang, Q. Y. (2023). BnIR: A multi-omics database with various tools for Brassica napus research and breeding. Molecular Plant, 16(4), 775–789. https://doi.org/10.1016/j.molp.2023.03.007
Yu, Y., Zhang, K., Yang, L., & Zhang, D. (2019). Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN. Computers and Electronics in Agriculture, 163, 104846. https://doi.org/10.1016/j.compag.2019.06.001.
Yu, X., Yin, D., Nie, C., Ming, B., Xu, H., Liu, Y., Bai, Y., Shao, M., Cheng, M., Liu, Y. J. C., & Agriculture, E. (2022). Maize tassel area dynamic monitoring based on near-ground and UAV RGB images by U-Net model. Computers and Electronics in Agriculture, 203, 107477. https://doi.org/10.1016/j.compag.2022.107477.
Yue, J., Yang, G., Tian, Q., Feng, H., Xu, K., & Zhou, C. (2019). Estimate of winter-wheat above-ground biomass based on UAV ultrahigh-ground-resolution image textures and vegetation indices. Isprs Journal of Photogrammetry and Remote Sensing, 150, 226–244. https://doi.org/10.1016/j.isprsjprs.2019.02.022.
Zajac, T., Oleksy, A., Synowiec, A., Dacko, M., Klimek-Kopyra, A., Ratajczak, K., & Kulig, B. (2019). Simulating the partitioning of winter rape biomass by increasing the cutting height of stems. International Agrophysics, 33(2), 241–253. https://doi.org/10.31545/intagr/109519
Zamani, A., Sharifi, A., Felegari, S., Tariq, A., & Zhao, N. (2022). Agro climatic zoning of saffron culture in Miyaneh City by using WLC method and remote sensing data. Agriculture-Basel, 12(1), 118. https://doi.org/10.3390/agriculture12010118
Zhang, Y., Zhang, Y., Ge, X., Yuan, Y., Jin, Y., Wang, Y., Zhao, L., Han, X., Hu, W., Yang, L., Gao, C., Wei, X., Li, F., & Yang, Z. (2023). Genome-wide association analysis reveals a novel pathway mediated by a dual-TIR domain protein for pathogen resistance in cotton. Genome Biology, 24(1), 111. https://doi.org/10.1186/s13059-023-02950-9.
Zhao, X., Su, Y., Hu, T., Cao, M., Liu, X., Yang, Q., Guan, H., Liu, L., & Guo, Q. (2022). Analysis of UAV lidar information loss and its influence on the estimation accuracy of structural and functional traits in a meadow steppe. Ecological Indicators, 135, 108515. https://doi.org/10.1016/j.ecolind.2021.108515.
Acknowledgements
This work was funded by the National Key R & D Program of China (Grant No. 2021YFD2000104) and the Key R & D Program of Zhejiang Province, China (Grant No. 2021C02057). We would like to thank Xin Yang, Letong Li, Leisen Fang, Yichi Zhang, Mengya Zhao, Li Zhai, and Shuobo Chen for helping with data collection and thank Prof. Weijun Zhou for providing the field experimental materials.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Shen, Y., Lu, X., Lyu, M. et al. High-throughput phenotyping of individual plant height in an oilseed rape population based on Mask-RCNN and UAV images. Precision Agric 25, 811–833 (2024). https://doi.org/10.1007/s11119-023-10095-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11119-023-10095-9