Elsevier

Pattern Recognition Letters

Volume 150, October 2021, Pages 76-83
Pattern Recognition Letters

Deep leaf: Mask R-CNN based leaf detection and segmentation from digitized herbarium specimen images

https://doi.org/10.1016/j.patrec.2021.07.003Get rights and content

Highlights

  • A modified mask RCNN named Deep Leaf is developed to identify the leaves from the digitized herbarium specimens.

  • Deep Leaf measures automatically the morphological traits of the extracted leaves.

  • Deep features are extracted through an improved ResNet50/101, which is chosen as the backbone network of the feature extraction.

  • We achieved better performance compared with the original mask RCNN algorithm for leaves detection.

Abstract

The generation of morphological traits of plants such as the leaf length, width, perimeter, area, and petiole length are fundamental features of herbarium specimens, thus providing high-quality data to investigate plant responses to ongoing climatic change and plant history evolution. However, the existing measurement methods are primarily associated with manual analysis, which is labor-intensive and inefficient. This paper proposes a deep learning-based approach, called Deep Leaf, for detecting and pixel-wise segmentation of leaves based on the improved state-of-the-art instance segmentation approach, Mask Region Convolutional Neural Network (Mask R-CNN). Deep Leaf can accurately detect each leaf in the herbarium specimen and measure the associated morphological traits. The experimental results indicate that our automated approach can segment the leaves of different families. Compared to manual measurement done by ecologist and botanist experts, the average relative error of leaf length is 4.6%, while the average relative error of leaf width is 5.7%.

Introduction

Plants are crucial elements of global biodiversity, and they play vital roles in protecting human well-being. The careful study of plant history and development processes is imperative. Among the different parts of a plant, leaves are the essential component where their structures are composed of different parts such as blade (lamina), petiole, midrib, stipules, tip, and veins (Fig. 1).

Besides, leaves are depicted in photosynthesis as the main organ whose size and shape vary from area to area within a plant, ranging from an elliptic to a palmate form. Recent studies on plant morphological traits analyses have been commonly used to know how species respond to biotic and abiotic factors and their impact on the environment. Given various ongoing efforts, morphological trait data generally remain available in multiple ecology and earth system sciences formats. Initiatives such as TRY (https://www.try-db.org/TryWeb/Home.php) attempt to fill gaps between the studied and the latest plant traits. Previously, almost all morphological trait measurement methods have been manual, involving advanced equipment and a high level of expertise. Besides, this process is highly labour-intensive, and the calculation is time-consuming. At present, numerous solutions were developed to determine the leaf outlines and categorize leaf shapes from digital images, such as LeafJ [1] and Easy Leaf Area [2]. However, none of the developed solutions can measure the morphological traits from Digitized Herbarium Specimens (DHS) images. Recently, Gaikwad et al. [3] developed the TraitEx tool to measure the morphological traits such as leaf length, width, area, and size from DHS images stored in herbarium Haussknecht of Jena, Germany (http://www.spezbot.uni-jena.de/herbarium). Nevertheless, TraitEx is a semi-automatic tool where the leaves to be measured should be extracted manually from the DHS images. The semi-automatic applications require applying a pre-processing step on the input images to extract features efficiently when utilizing multiple automation degrees. To complement the existing efforts and overcome the semi-automatic applications’ bottleneck, we propose our Deep Leaf approach to automatically measure the leaf morphological traits from the DHS images stored in herbarium Haussknecht. The proposed approach can detect all leaves and measure their most frequently used morphological traits (e.g., leaf length, width, area, and perimeter). These traits offer relevant information toward improving eco-evolutionary research and biodiversity education.

This paper is a part of the MAMUDS project (https://fusion.cs.uni-jena.de/fusion/projects/managing-multimedia-data-for-science-mamuds/). Its main aim is to develop a multimedia data management platform for the biodiversity domain by extracting the morphological traits from the DHS images. However, we concentrated in this paper on:

  • 1.

    Releasing a new dataset for the community holding the DHS obtained from the herbarium Haussknecht of Friedrich-Schiller-Universität Jena, Germany.

  • 2.

    Detecting the objects within the DHS, such as scale-bar, color pallet, specimen label, envelopes, bar-code, and stamp, as they are mostly placed at different locations on the herbarium sheet.

  • 3.

    Identifying the perfect leaves, the imperfect leaves, and leaves with a missed parts within the herbarium sheet.

  • 4.

    Automatically measuring the leaf length, width, area, and perimeter of the perfect and imperfect leaf sub-categories.

  • 5.

    Testing the proposed approach on the DHS containing different species’ leaves and different degrees of adhesion and overlapping.

The rest of this paper is organized as follows. Section 2 shows the related work for object detection and segmentation. The proposed approach for object detection and leaves segmentation is detailed in Section 3. Section 4 describes the experimental results to examine the performance of Deep Leaf. Section 5 presents the result analysis. Finally, Section 6 presents the conclusion and future directions.

Section snippets

Related work

With the progress of machine learning, several supervised and unsupervised methods have been developed and quickly grown in different visual recognition tasks, such as the automatic identification of species from the DHS images. Al-Shakarji et al. [4] have proposed an unsupervised approach for plant leaf detection by adopting a K-means-based mask followed by Expectation-Maximization (EM) algorithm. Also, Unger et al. [5] have designed a new pipeline by employing a Support Vector Machine (SVM)

The proposed approach

Our proposed approach’s main objective is to automatically measure the leaves’ morphological traits by efficiently identifying perfect and imperfect leaves within the DHS images. Deep leaf is composed of several stages (Fig. 3). The details of each stage are explained in the following subsections.

Experimental results

In this section, we perform experiments to validate the reliability and effectiveness of our proposed approach.

Results analysis

The scope of this paper is to examine our proposed approach performance to identify and measure the leaves’ morphological traits from the DHS images. The work’s primary innovation is to use the improved Mask R-CNN algorithm and evaluating its potential for this crucial task [9]. First, the backbone network structure of the deep leaf is enhanced by changing the order of the components on the ResNet50/101 architectures. Furthermore, the Mish activation function was adopted before the weight layer

Conclusion and future work

Our presented work introduces a morphological feature segmentation and measurement scheme based on Mask R-CNN by improving the residual unit of the backbone ResNet-50/101 structure. Based on the observed results, Deep Leaf achieves high precision and robustness when measuring leaves’ morphological traits, where the average relative error of leaf length and leaf width are 4.6%, and 5.7%, respectively.

Future studies would increase the training data’s dimension to add more image variations and

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was part of the MAMUDS project (Management Multimedia Data for Science), supported by BMBF - Germany (Project No. 01D16009) and MHESR - Tunisia.

References (26)

  • B.R. Kang et al.

    Bshapenet: object detection and instance segmentation with bounding shape masks

    Pattern Recognit. Lett.

    (2020)
  • M.A. Khan et al.

    Attributes based skin lesion detection and recognition: a mask rcnn and transfer learning-based deep learning framework

    Pattern Recognit. Lett.

    (2021)
  • K.N. Julin N Maloof et al.

    Leafj: an imagej plugin for semi-automated leaf shape measurement

    J. Visualiz. Exp.

    (2013)
  • H.M. Easlon et al.

    Easy leaf area: automated digital image analysis for rapid and accurate measurement of leaf area

    Appl. Plant Sci.

    (2014)
  • J. Gaikwad, A. Triki, B. Bouaziz, Measuring morphological functional leaf traits from digitized herbarium specimens...
  • N.M. Al-Shakarji et al.

    Unsupervised learning method for plant and leaf segmentation

    IEEE Applied Imagery Pattern Recognition Workshop (AIPR)

    (2017)
  • J. Unger et al.

    Computer vision applied to herbarium specimens of german trees: testing the future utility of the millions of herbarium specimen images for automated identification

    BMC Evol. Biol.

    (2016)
  • B.R. Hussein et al.

    Semantic segmentation of herbarium specimens using deep learning techniques

    Computational Science and Technology

    (2020)
  • A. Triki et al.

    Objects detection from digitized herbarium specimen based on improvedYOLO V3

    Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP, VISAPP

    (2020)
  • W.N. Weaver et al.

    Leafmachine: using machine learning to automate leaf trait extraction from digitized herbarium specimens

    Appl. Plant Sci.

    (2020)
  • K. He et al.

    Mask r-cnn

    2017 IEEE International Conference on Computer Vision (ICCV)

    (2017)
  • V.A. Funk

    The importance of herbaria

    Plant Sci. Bull.

    (2003)
  • D. Abhishek et al.

    The VIA annotation software for images, audio and video

    Proceedings of the 27th ACM International Conference on Multimedia

    (2019)
  • Cited by (26)

    • Applications of computer vision and machine learning techniques for digitized herbarium specimens: A systematic literature review

      2022, Ecological Informatics
      Citation Excerpt :

      Similarly, Davis et al. (2020) trained the same Mask RCNN model to count reproductive structures in digitized herbarium specimens with an accuracy of 87% using Resnet-50 backbone for six reproductive structures. Mask R-CNN model has also been investigated in other studies for a similar purpose, although having different target trait features (Goëau et al., 2022; Love et al., 2021; Triki et al., 2021a; Triki et al., 2021b; Triki et al., 2021c). Despite that the deep learning models have been performing better than crow-sourced data, these models are unable to surpass the expertise of Botanical experts.

    View all citing articles on Scopus
    View full text