Original papersGraph-based methods for analyzing orchard tree structure using noisy point cloud data
Introduction
Understanding tree growth is an important consideration for commercial orchard operators. There are many ways to manually measure growth factors, including mobile Leaf Area Index (LAI) measuring devices presented by Confalonieri et al. (2013) and Francone et al. (2014) or ceptometer sensors which Ibell et al. (2015) showed could be used to study tree productivity. However, manual measurements are difficult to automate and can have prohibitive restrictions including time required to take measurements, a requirement to measure in many locations, or weather limitations (such as a need for clear sky). As an alternative, reality capture can be used to get digital models of the trees which can then be analysed. Electromagnetic digitisation methods such as those presented by Arikapudi et al. (2015) are highly accurate but difficult to implement in practice on an orchard scale. Cameras like those applied by Underwood et al. (2016) are cheap, accessible and flexible, but cannot always reconstruct geometric data. LiDAR technology is rapidly improving, and can be a quick and detailed method of reality capture which provides large masses of data and is easy to automate. Wu et al. (2018) measures changes in Leaf Area and Leaf Area Density for various tree crops using terrestrial LiDAR, Westling et al. (2018) presented a method which performed detailed analysis of tree growth factors using low quality LiDAR, and Wu et al. (2020) shows excellent results for mapping structural metrics like crown volume using airborne LiDAR which scales easily. Here, we explore three separate operations which can be performed on low-quality LiDAR scans of orchard trees to enable further analyses, namely trunk location, individual segmentation and matter classification.
Previous works in trunk location in an orchard environment are typically focused on mobile platform localisation and mapping, and involve the use of multiple sensors. Bargoti et al. (2015) locate trunks primarily in the point cloud space using Hough transforms (89.7% accurate), and then reproject the detections into the camera frame to improve the results (95.8%). Shalal et al. (2015) similarly fuse laser scanner and camera data and distinguish between tree and non-tree objects, using the laser scanner to detect edge points and the camera for colour verification (96.64%). Chen et al. (2018) instead fuse camera and ultrasonic data and train an SVM classifier to localise their robot using detected trunks (98.96%). However, all of these methods are working in a limited context, with a platform travelling parallel to rows of trees and processing on a frame-by-frame basis.
Segmentation in this paper is defined as separating individual trees in the data, namely identifying which points belong to which trees. This can allow better insights for end users, since results including tree growth parameters can be mapped to specific trees (Underwood et al. (2016)). McFadyen et al. (2004) showed that yield improves with light interception and tree volume, but only up to a certain point, beyond which orchard crowding reduces yield over time. If individual trees can be discerned, these effects can be better understood than if each row is just a wall of foliage.
Driven by the recent interest in autonomous driving applications, many of the current approaches to point cloud semantic segmentation and classification operate on small point clouds (e.g. Guo et al. (2019), up to 4096 points) as they are designed to run in real-time on single frames. Most modern methods for segmentation in larger point clouds are in specific contexts with simple structures (Poux and Billen (2019)) or work on simplified data from sampled CAD models rather than LiDAR data (Liu et al. (2019)). In agriculture specifically, a variety of methods have been explored. Underwood et al. (2016) use cameras which have many advantages, but demonstrate difficulties in distinguishing overlapping branches, particularly since there is only one vantage point. Guan et al. (2015) uses Euclidian distance clustering to segment trees, but the trees shown are spaced apart with minimal encroachment. Good results were achieved by Li et al. (2012) with aerial LiDAR data using convex hulls, but this was a forestry application where again trees tend to be spaced out enough to make segmentation simple. Reiser et al. (2018) presented good results on ground crops using very sparse point clouds, however their method relied on prior knowledge of crop spacing as well as a known location for each plant. We aim to implement a method which works on very large point clouds with overlapping trees and no prior.
Classification in this paper is defined as assigning pointwise semantic meaning, specifically identifying which points represent leafy versus woody matter. The key insight here is that woody matter (i.e. trunks and branches) are non photosynthetically active, and as explained by Ma et al. (2016) there is benefit in measuring the amount of photosynthetically active material in a tree for growing purposes. One application of this was presented by Westling et al. (2018), who simulate the amount of light absorbed by trees digitized using LiDAR. Identifying woody matter improves the quality of simulation with more accurate light transmission characteristics as well as better estimates of light absorption.
Trunk classification on pure point cloud data can be done in a wide variety of ways. Fritz et al. (2013) and others focused on tall trees with a single primary trunk apply cylinder fitting to detect that trunk, and classify surrounding points as leaves. Su et al. (2019) uses a similar cylinder fitting method without the tall-tree assumption, but relies on high density scans containing minimal clutter points in order to identify cylindrical sections of point cloud. A common approach to point cloud classification presented by several authors (e.g. Lalonde et al., 2006, Ma et al., 2016, Brodu and Lague, 2012) involves using eigenvalue decomposition to describe patches of points into broadly three categories: planar, linear and random. The patches can then be reliably classified as ground, trunk and leaf respectively, though this method is very sensitive to noise and can cause disconnected results due to its patch-based nature. Vicari et al. (2019) presented an eigenvalue method which gets around this limitation by combining graph-based methods to integrate tree structure in the calculation. Livny et al. (2010) similarly use a graph-based approach with optimised model fitting and generalised cylinders to reconstruct the skeletal structure of laser-scanned trees, while Digumarti et al. (2018) achieves good results in extracting the tree skeleton using local feature vectors. Many of these methods rely on high quality data such as that captured by slow tripod-mounted scanners and are less effective on faster mobile data.
Static (tripod) LiDAR such as that used by Vicari et al., 2019, Ma et al., 2016 and others produce excellent results as shown in Fig. 1a. However, use of static LiDAR requires time to set up and calibrate the position at each scan, and requires scanning from multiple positions for good coverage of each object, and the scans must then be combined to form a cohesive point cloud. Due to these factors, they are not practical for scanning large areas like a commercial orchard setting. At the other extreme, aerial LiDAR as used by Windrim and Bryson (2018) can cover acres of land very rapidly, but the resultant data is much less accurate and much of it is occluded. In particular, doing analyses below the top of the canopy becomes difficult. Mobile LiDAR is a good compromise, allowing scanning of multiple acres per day with less occlusion. However, the accuracy can suffer due to the limitations of necessary automated registration. Makkonen et al. (2015) found an RMSE of approximately 15–30 mm using a handheld LiDAR, which is due to a combination of scanner accuracy, operator training and scanning procedure. As shown in Fig. 1, the handheld and mobile options show features like leaves much less distinctly because of this. Despite that, Bauwens et al. (2016) showed that handheld LiDAR produces better coverage doing forest inventory than static LiDAR and Ryding et al. (2015) concluded that handheld sensors are efficient, cost effective and versatile for forest surveying. Furthermore, LiDAR mounted on mobile platforms like that presented by Underwood et al. (2016) enables fully automated capture. When the data can be captured and processed quickly, it can be applied to orchard-scale analysis, or analysis of individual trees over the entire orchard. For these reasons, we are interested in developing methods which are applicable to low-quality point cloud data, and which ideally can be applied to data of variable quality.
Despite the lower quality, Mobile LiDAR has been used in a range of applications and industries, including building modelling in construction (Sepasgozar et al. (2014)), cultural heritage surveying (Chan et al. (2016)) and mining (Dewez et al. (2016)). As mentioned earlier, Reiser et al. (2018) was able to achieve good results doing ground crop plant segmentation with sparse mobile LiDAR data, but relied heavily on priors. Underwood et al. (2016) used LiDAR on a mobile platform for orchard mapping and canopy volume. Westling et al. (2018) presented a light environment simulation method using low-quality point clouds from handheld LiDAR.
Deep learning is an option for processing point clouds, though this presents its own challenges. A review of the state of the art conducted by Guo et al. (2019) found that most current approaches to point cloud object classification operate on point clouds up to 4096 points, which is insufficient for our analyses. Methods using multi-view convolutional neural networks (e.g. Su et al. (2015)) are unlikely to work in our context due to complex occlusions and varied environments. Similarly, volumetric methods like that of Wu et al. (2015) or Maturana and Scherer (2015) are similarly unsuited, since trees are large, varied in size, and highly complex. These methods tend to be limited to voxels of size 32x32x32, which would lose a lot of detail in complex tree crops. Direct point learning methods like PointNet (Qi et al. (2017)) and its derivatives have mostly been used on standard datasets with perfect data sampled from 3D models, which produce far cleaner inputs than data from LiDAR. Guan et al. (2015) was able to use deep learning techniques to identify tree species by LiDAR, but not on the raw point cloud, instead computing the waveform of the data and passing that into a neural net. Windrim and Bryson (2018) and Xi et al. (2018) use fully connected 3D CNNs to perform tree classification to good effect, though were applied to trees which are similar in size and shape and have little overlap. Kumar et al. (2019) was able to identify that objects as trees or non-trees with 90% accuracy, which is an operation on the macro scale and may not be applicable to small-scale features like branches and leaves. Modern machine learning methods rely on extensive labelled datasets that are not readily available in orchard applications. We instead focus on analytical methods rather than deep learning in order to avoid the need for labelled data in new contexts.
We present a system which, like Vicari et al. (2019), uses graph-based methods to perform a range of tasks on point clouds in tree crops, with specific emphasis on handling low-quality and often overlapping data. The method we present relies on the basic geometry of tree-like structures, namely that trees are connected by a network of woody matter, which is invariant to noise, fidelity and occlusion.
Section snippets
Method
In this section, we first describe the methods used to collect or generate data for all experiments. Then, the basic operation implemented is described, namely graph creation and search with an optional feature enrichment edge weighting scheme. Finally, we describe the three operations to which the graph operation was applied.
The operations developed here were primarily using the ACFR Comma and Snark open-source libraries (Australian Centre for Field Robotics (ACFR) (2012)) and have been
Trunk detection
Trunk detection was applied to real data (stands of three avocado trees as well as entire blocks of mango trees) and virtual data, of which qualitative examples are presented in Fig. 11 (virtual trees) and Fig. 12 (high density real trees). The mango data is a mixture of low and medium density (not much overlap) and high density trees (significant overlap). Generally trunk detection works well when trees are well defined, but not as well at the edges of scans where trees and the ground are
Trunk detection
Table 2 shows the results of finding trunks, with an F1 score of 0.774 on real data. A likely cause for false negatives in real data are the high-density (2 m spacing) trees which are visible in Fig. 12. These trees are planted close together and, in the case of trellises, feature canopy close to the ground, which makes identification of a single trunk per tree difficult. Furthermore, the mango data contains fence posts and non-tree items (e.g. vehicles) which would register false positives
Conclusions
We presented a system for processing point cloud data captured using LiDAR at a fruit orchard to detect trunk location with no priors, segment individual trees even in high-density contexts, and classify trunk and leaf matter automatically. The trunk detection method enables automated location of individual trees from low-quality scans, which can provide value in orchard navigation or horticultural applications. Similarly, segmentation of individual trees is difficult in orchard contexts where
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
This work is supported by the Australian Centre for Field Robotics (ACFR) at The University of Sydney. Thanks to Vsevolod (Seva) Vlaskine and the software team for their continuous support, and to Salah Sukkerieh for his continuous direction and support of Agriculture Robotics at the ACFR. For more information about robots and systems for agriculture at the ACFR, please visit http://sydney.edu.au/acfr/agriculture.
References (47)
- et al.
3d terrestrial lidar data classification of complex natural scenes using a multi-scale dimensionality criterion: Applications in geomorphology
ISPRS Journal of Photogrammetry and Remote Sensing
(2012) - et al.
Multi-feature fusion tree trunk detection and orchard mobile robot localization using camera/ultrasonic sensors
Computers and electronics in agriculture
(2018) - et al.
Comparison of leaf area index estimates by ceptometer and PocketLAI smart app in canopies with different structures
Field Crops Research
(2014) - et al.
Iterative individual plant clustering in maize with assembled 2d lidar data
Comput. Ind.
(2018) - et al.
Orchard mapping and mobile robot localisation using on-board camera and laser scanner data fusion–part a: Tree detection
Computers and Electronics in Agriculture
(2015) - et al.
Mapping almond orchard canopy volume, flowers, fruit and yield using lidar and vision sensors
Computers and Electronics in Agriculture
(2016) - et al.
Light interception modelling using unstructured lidar data in avocado orchards
Computers and Electronics in Agriculture
(2018) - R. Arikapudi, S. Vougioukas, and T. Saracoglu. Orchard tree digitization for structural-geometrical modeling. In...
- Australian Centre for Field Robotics (ACFR). Comma and snark: generic c++ libraries and utilities for robotics....
- et al.
A pipeline for trunk detection in trellis structured apple orchards
Journal of field robotics
(2015)
Forest inventory with terrestrial lidar: A comparison of static and hand-held mobile laser scanning
Forests
Survey accuracy analysis of a hand-held mobile lidar device for cultural heritage documentation
Photogrammetrie-Fernerkundung-Geoinformation
Automatic segmentation of tree structure from point cloud data
IEEE Robotics and Automation Letters
Uav-based photogrammetric point clouds–tree stem mapping in open stands in comparison to terrestrial laser scanner point clouds
Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci
Deep learning-based tree classification using mobile lidar data
Remote Sensing Letters
A formal basis for the heuristic determination of minimum cost paths
IEEE transactions on Systems Science and Cybernetics
Preliminary results from the small tree-high productivity initiative
XI International Mango Symposium
Automatic individual tree detection and canopy segmentation from three-dimensional point cloud images obtained from ground-based lidar
Journal of Agricultural Meteorology
Segmentation of bean-plants using clustering algorithms
AGRIS On-line Papers in Economics and Informatics
Development of deep learning architecture for automatic classification of outdoor mobile lidar data
Int. J. Remote Sens.
Cited by (18)
Bridge substructure feature extraction based on the underwater sonar point cloud data
2024, Ocean EngineeringImproving point cloud classification and segmentation via parametric veronese mapping
2023, Pattern RecognitionPhenotyping of individual apple tree in modern orchard with novel smartphone-based heterogeneous binocular vision and YOLOv5s
2023, Computers and Electronics in AgricultureU<sup>2</sup>ESPNet—A lightweight and high-accuracy convolutional neural network for real-time semantic segmentation of visible branches
2023, Computers and Electronics in AgricultureCitation Excerpt :The main methods for branches detection are divided into two main types: LiDAR-based methods and Vision-based methods. The former refers to obtaining 3D point cloud data of branches using LiDAR and detection based on this information (Zhang et al., 2020b; Westling et al., 2021; Fan et al., 2020). LiDAR-based methods, however, are susceptible to the loss of the original input point cloud caused by illumination, resulting in a degradation of the detection effect.
A real-time branch detection and reconstruction mechanism for harvesting robot via convolutional neural network and image segmentation
2022, Computers and Electronics in AgricultureCitation Excerpt :The existing work on branch detection methods can be roughly divided into two categories according to the sensors been utilized, i.e., the Light Detection and Ranging (LiDAR)-based and the camera-based methods. The former is to directly acquire 3D point cloud data of the orchard using LiDAR, and then detect and reconstruct the branches based on this information (Zhang et al., 2020; Westling et al., 2021). For example, Hackenberg et al. developed hierarchical columnar structures to describe the relationships among branches and also extract different tree components (Hackenberg et al., 2015).