GinJinn: An object-detection pipeline for automated feature extraction from herbarium specimens

Abstract

Premise: The generation of morphological data in evolutionary, taxonomic, and ecological studies of plants using herbarium material has traditionally been a labor-intensive task. Recent progress in machine learning using deep artificial neural networks (deep learning) for image classification and object detection has facilitated the establishment of a pipeline for the automatic recognition and extraction of relevant structures in images of herbarium specimens.

Methods and results: We implemented an extendable pipeline based on state-of-the-art deep-learning object-detection methods to collect leaf images from herbarium specimens of two species of the genus Leucanthemum. Using 183 specimens as the training data set, our pipeline extracted one or more intact leaves in 95% of the 61 test images.

Conclusions: We establish GinJinn as a deep-learning object-detection tool for the automatic recognition and extraction of individual leaves or other structures from herbarium specimens. Our pipeline offers greater flexibility and a lower entrance barrier than previous image-processing approaches based on hand-crafted features.

Keywords: TensorFlow; deep learning; herbarium specimens; object detection; visual recognition.

FIGURE 1

Flow diagram of the six GinJinn pipeline steps. A project folder is generated using ginjinn new (1) and the configuration file is modified depending on the user’s needs (1.1). The preparation (2), processing (3), training (4), and export (5) steps are executed sequentially with specific GinJinn commands (setup_dataset, setup_model, train, and export, respectively), or alternatively at once with the single ginjinn auto command. When not using ginjinn auto, the user can modify intermediary TensorFlow configuration files (3.1) for additional control over the model parameters and augmentation options. The trained and exported model can be used for inference of bounding boxes on new data using ginjinn detect. GinJinn commands are indicated by the yellow process boxes. Data inputs and outputs are illustrated with solid and dashed arrows, respectively. After bounding box detection, the extracted structures of interest can be supplied to other tools for downstream analyses.

FIGURE 2

(A) Output type ‘ibb’ (image with bounding boxes) showing class‐wise predicted bounding boxes of leaves with a score of 0.5 or higher drawn on the original image of a herbarium specimen. The score can be interpreted as a probability that the content of the bounding box belongs to a certain object class (in this case, a leaf). (B) Output type ‘ebb’ (extracted bounding boxes with a padding of 25 pixels) for selected true positive examples of the detected leaves shown in A. (C) Output type ‘ebb’ for selected false positive examples of the leaves shown in A.