Deep leaf: Mask R-CNN based leaf detection and segmentation from digitized herbarium specimen images
Introduction
Plants are crucial elements of global biodiversity, and they play vital roles in protecting human well-being. The careful study of plant history and development processes is imperative. Among the different parts of a plant, leaves are the essential component where their structures are composed of different parts such as blade (lamina), petiole, midrib, stipules, tip, and veins (Fig. 1).
Besides, leaves are depicted in photosynthesis as the main organ whose size and shape vary from area to area within a plant, ranging from an elliptic to a palmate form. Recent studies on plant morphological traits analyses have been commonly used to know how species respond to biotic and abiotic factors and their impact on the environment. Given various ongoing efforts, morphological trait data generally remain available in multiple ecology and earth system sciences formats. Initiatives such as TRY (https://www.try-db.org/TryWeb/Home.php) attempt to fill gaps between the studied and the latest plant traits. Previously, almost all morphological trait measurement methods have been manual, involving advanced equipment and a high level of expertise. Besides, this process is highly labour-intensive, and the calculation is time-consuming. At present, numerous solutions were developed to determine the leaf outlines and categorize leaf shapes from digital images, such as LeafJ [1] and Easy Leaf Area [2]. However, none of the developed solutions can measure the morphological traits from Digitized Herbarium Specimens (DHS) images. Recently, Gaikwad et al. [3] developed the TraitEx tool to measure the morphological traits such as leaf length, width, area, and size from DHS images stored in herbarium Haussknecht of Jena, Germany (http://www.spezbot.uni-jena.de/herbarium). Nevertheless, TraitEx is a semi-automatic tool where the leaves to be measured should be extracted manually from the DHS images. The semi-automatic applications require applying a pre-processing step on the input images to extract features efficiently when utilizing multiple automation degrees. To complement the existing efforts and overcome the semi-automatic applications’ bottleneck, we propose our Deep Leaf approach to automatically measure the leaf morphological traits from the DHS images stored in herbarium Haussknecht. The proposed approach can detect all leaves and measure their most frequently used morphological traits (e.g., leaf length, width, area, and perimeter). These traits offer relevant information toward improving eco-evolutionary research and biodiversity education.
This paper is a part of the MAMUDS project (https://fusion.cs.uni-jena.de/fusion/projects/managing-multimedia-data-for-science-mamuds/). Its main aim is to develop a multimedia data management platform for the biodiversity domain by extracting the morphological traits from the DHS images. However, we concentrated in this paper on:
- 1.
Releasing a new dataset for the community holding the DHS obtained from the herbarium Haussknecht of Friedrich-Schiller-Universität Jena, Germany.
- 2.
Detecting the objects within the DHS, such as scale-bar, color pallet, specimen label, envelopes, bar-code, and stamp, as they are mostly placed at different locations on the herbarium sheet.
- 3.
Identifying the perfect leaves, the imperfect leaves, and leaves with a missed parts within the herbarium sheet.
- 4.
Automatically measuring the leaf length, width, area, and perimeter of the perfect and imperfect leaf sub-categories.
- 5.
Testing the proposed approach on the DHS containing different species’ leaves and different degrees of adhesion and overlapping.
The rest of this paper is organized as follows. Section 2 shows the related work for object detection and segmentation. The proposed approach for object detection and leaves segmentation is detailed in Section 3. Section 4 describes the experimental results to examine the performance of Deep Leaf. Section 5 presents the result analysis. Finally, Section 6 presents the conclusion and future directions.
Section snippets
Related work
With the progress of machine learning, several supervised and unsupervised methods have been developed and quickly grown in different visual recognition tasks, such as the automatic identification of species from the DHS images. Al-Shakarji et al. [4] have proposed an unsupervised approach for plant leaf detection by adopting a K-means-based mask followed by Expectation-Maximization (EM) algorithm. Also, Unger et al. [5] have designed a new pipeline by employing a Support Vector Machine (SVM)
The proposed approach
Our proposed approach’s main objective is to automatically measure the leaves’ morphological traits by efficiently identifying perfect and imperfect leaves within the DHS images. Deep leaf is composed of several stages (Fig. 3). The details of each stage are explained in the following subsections.
Experimental results
In this section, we perform experiments to validate the reliability and effectiveness of our proposed approach.
Results analysis
The scope of this paper is to examine our proposed approach performance to identify and measure the leaves’ morphological traits from the DHS images. The work’s primary innovation is to use the improved Mask R-CNN algorithm and evaluating its potential for this crucial task [9]. First, the backbone network structure of the deep leaf is enhanced by changing the order of the components on the ResNet50/101 architectures. Furthermore, the Mish activation function was adopted before the weight layer
Conclusion and future work
Our presented work introduces a morphological feature segmentation and measurement scheme based on Mask R-CNN by improving the residual unit of the backbone ResNet-50/101 structure. Based on the observed results, Deep Leaf achieves high precision and robustness when measuring leaves’ morphological traits, where the average relative error of leaf length and leaf width are 4.6%, and 5.7%, respectively.
Future studies would increase the training data’s dimension to add more image variations and
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work was part of the MAMUDS project (Management Multimedia Data for Science), supported by BMBF - Germany (Project No. 01D16009) and MHESR - Tunisia.
References (26)
- et al.
Bshapenet: object detection and instance segmentation with bounding shape masks
Pattern Recognit. Lett.
(2020) - et al.
Attributes based skin lesion detection and recognition: a mask rcnn and transfer learning-based deep learning framework
Pattern Recognit. Lett.
(2021) - et al.
Leafj: an imagej plugin for semi-automated leaf shape measurement
J. Visualiz. Exp.
(2013) - et al.
Easy leaf area: automated digital image analysis for rapid and accurate measurement of leaf area
Appl. Plant Sci.
(2014) - J. Gaikwad, A. Triki, B. Bouaziz, Measuring morphological functional leaf traits from digitized herbarium specimens...
- et al.
Unsupervised learning method for plant and leaf segmentation
IEEE Applied Imagery Pattern Recognition Workshop (AIPR)
(2017) - et al.
Computer vision applied to herbarium specimens of german trees: testing the future utility of the millions of herbarium specimen images for automated identification
BMC Evol. Biol.
(2016) - et al.
Semantic segmentation of herbarium specimens using deep learning techniques
Computational Science and Technology
(2020) - et al.
Objects detection from digitized herbarium specimen based on improvedYOLO V3
Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP, VISAPP
(2020) - et al.
Leafmachine: using machine learning to automate leaf trait extraction from digitized herbarium specimens
Appl. Plant Sci.
(2020)
Mask r-cnn
2017 IEEE International Conference on Computer Vision (ICCV)
The importance of herbaria
Plant Sci. Bull.
The VIA annotation software for images, audio and video
Proceedings of the 27th ACM International Conference on Multimedia
Cited by (26)
Fast Multi-View 3D reconstruction of seedlings based on automatic viewpoint planning
2024, Computers and Electronics in AgricultureEstimation of morphological traits of foliage and effective plant spacing in NFT-based aquaponics system
2023, Artificial Intelligence in AgricultureAn approach for characterization of infected area in tomato leaf disease based on deep learning and object detection technique
2022, Engineering Applications of Artificial IntelligenceApplications of computer vision and machine learning techniques for digitized herbarium specimens: A systematic literature review
2022, Ecological InformaticsCitation Excerpt :Similarly, Davis et al. (2020) trained the same Mask RCNN model to count reproductive structures in digitized herbarium specimens with an accuracy of 87% using Resnet-50 backbone for six reproductive structures. Mask R-CNN model has also been investigated in other studies for a similar purpose, although having different target trait features (Goëau et al., 2022; Love et al., 2021; Triki et al., 2021a; Triki et al., 2021b; Triki et al., 2021c). Despite that the deep learning models have been performing better than crow-sourced data, these models are unable to surpass the expertise of Botanical experts.
Deep Learning Techniques in Leaf Image Segmentation and Leaf Species Classification: A Survey
2023, Wireless Personal CommunicationsReal-time Segmentation and Classification using Feature Extraction for Medicinal Leaves
2023, AIP Conference Proceedings