Improved crop row detection with deep neural network for early-season maize stand count in UAV imagery

Highlights

• Determine early-season corn stands based on UAV-based remote sensing data.

• Determine early-season corn stands using geometric descriptor information and deep neural networks.

• Recognize and detect crop-rows regardless of the terrain conditions.

Abstract

Stand counts is one of the most common ways farmers assess plant growth conditions and management practices throughout the season. The conventional method for early-season stand count is through manual inspection, which is time-consuming, laborious, and spatially limited in scope. In recent years, Unmanned Aerial Vehicles (UAV) based remote sensing has been widely used in agriculture to provide low-altitude, high spatial resolution imagery to assist decision making. In this project, we designed a system that uses geometric descriptor information with deep neural networks to determine early-season maize stands from relatively low spatial resolution (10 to 25 mm) aerial data, which covers a relatively large area (10 to 25 hectares). Instead of detecting individual crops in a row, we process the entire row at one time, which significantly reduces the requirements for the clarity of the crops. Besides, our new MaxArea Mask Scoring RCNN algorithm could segment crop-rows out in each patch image, regardless of the terrain conditions. The robustness of our scheme was tested on data collected at two different fields in different years. The accuracy of the estimated emergence rate reached up to 95.8%. Due to the high processing speed of the system, it has the potential for real-time applications in the future.

Introduction

Stand counts in crop production refer to counting the number of plants grown in a specific area, in other words, obtaining the population of plants. In-season crop stands are often different than the planting population. Early season crop stands are usually a result of seed quality, planter performance, and seedlings’ response to soil and weather during emergence; while mid to late season crop stands are usually a result of weather, soil, fertilizer, pest pressure and other management practices. Stand counts are one of the most common ways farmers assess plant growth conditions and management practices throughout the season and can be used to make management decisions, such as determining if replanting is required. It is also important for crop insurance industry, planter manufacturers and other agricultural professionals and companies to evaluate products and management practices.

The traditional way of conducting stand counts is to walk in the field, manually sample a few areas and estimate the total number of plants in the field based on the results in sampled areas. This method is time-consuming, laborious, and does not cover the entire field. Efforts have been made over decades to develop automatic ways to replace the labor-intensive manual stand counts. Satellite or manned aircraft based remote sensing systems had been used to cover large area. However, due to the low spatial resolution (3–5 meters) obtained from those remote sensing platforms, plants were not clearly visible in resulting images. Because of this, only spectral reflectance, rather than morphological or texture features, were used to capture the signal of non-uniform stand distributions (Erickson et al., 2004). Ground-based systems had been developed for early and mid-growth stage stand counts when plants are small. Most of them used the machine vision technology to either look downward or from the side between rows (Luck et al., 2008, Thorp et al., 2007, Nakarmi and Tang, 2012, Shi et al., 2013). Although they achieved good counting accuracies with very high-resolution data, currently the throughput is still not high enough to practically apply such technology to large fields.

Stand counts for row crops such as maize are most valuable early in the season when farmers still have time to replant the field if the stand is poor enough. Knowing about a poor stand later in the growing season is not as helpful, as it is too late for action to be taken. Therefore, a system that operated on V4-5 growth stage maize would have a distinct advantage. With the advances in unmanned aerial vehicle (UAV) technology, it becomes possible to have relatively high-resolution data (spatial resolution of 2 to 25 mm) and cover a relatively large area (5 to 25 hectares) almost whenever needed in agriculture (Shi et al., 2016). Though the spatial coverage is still limited by flight endurance, UAV technology nowadays is feasible to provide coverage for an entire field. How to make use of the highly detailed images collected by the UAVs to help agricultural production is the research question to answer. Fortunately, the advancements in the machine/deep learning area provide many possible and powerful solutions. Su et al. (2018) used various vegetation indices derived from a UAV-based five-spectral-band multispectral camera and Random Forest with Bayesian hyperparameter optimization to detect wheat yellow rust. Lin et al. (2017) proposed that region-based convolutional neural networks could improve the accuracy of strawberry flower detection to reliably predict fruit yield. Zou et al. (2019) reported a method using UAV, hyperspectral image sensors, and machine learning based data processing algorithms to accurately measure tree height and diameter at chest height in forests to assess the growth rate of cultivars. Etienne and Saraswat (2019) used unmanned aerial vehicles and image sensors to capture image data of maize and soybean fields to automatically detect weeds in the fields at different stages of the growing season using machine learning techniques. Developing an economical solution for monitoring plant health has always received attention from different aspects. Do et al. (2018) developed a method for monitoring nitrogen and water in plants by analyzing digital images of citrus plants collected from UAV. They evaluated several machine learning techniques such as simple linear regression and convolutional neural networks, and the results were all satisfactory.

For in-season crop stand count specifically, applications that used UAV and machine learning techniques had been proposed as well. Kitano et al. (2019) proposed using deep learning on images captured from UAV for maize stand count. After splitting maize and ground by recognizing each plant in the image using U-net (Ronneberger et al., 2015), they used an opening morphological operator to separate the plants that were close together. Finally, the blob detection (Danker and Rosenfeld, 1981) method was applied for quantifying the number of maize plants. However, their method relied too much on accurately detecting individual plant. If the image resolution is reduced (usually due to the increased flying altitude of the UAV), for example, the width of the plant is reduced from 20 pixels to 5 pixels in the image, this approach would fail due to the small size and ambiguous features of the individual plant. Varela et al. (2018) also proposed a method for maize stand count. Unlike Lin’s approach, they first used the Excess Green (ExG) index (Yaakob, 2017) to separate the vegetation from the background using the difference in color. Then they used the Canny Edge detector and Hough Transform method to detect the rows of plants and the orientation angle, which avoided the interference caused by weeds (they assumed that the weeds were located at inter-row areas). Then, they used 10 geometric descriptors to describe the characteristics of the plant and used a decision tree classifier to identify individual plants and count the number of them in each row. However, similar to Lin’s work, Varela’s scheme relied on detecting each individual plant as well, therefore the performance would also be reduced as the image resolution decreases. In addition, since the directions of crop-rows in the images usually are not absolutely horizontal, the authors first calculated the rotation angle required to rotate to the horizontal level for each row, and then voted to select an appropriate value for a whole image. This method was based on the assumption that all the plant rows in one image were almost parallel. In real cases, rows may not be perfect straight and parallel due to terrain and planting.

Therefore, in this study, a novel early-season maize stand count approach is proposed, which has the following advantages: (1) To better conform to various terrain, each crop-row is considered as an instance, and the stand count algorithm is customized for each row. An innovative instance segmentation algorithm – MaxArea Mask Scoring RCNN is developed for detecting the crop-rows, regardless of their directions and shapes displayed in the image including curved rows that intersect at different angles with numerous gaps. (2) Instead of detecting individual maize plant in a row, a simple but efficient sparse region detection and stand count estimation strategy is proposed, which utilizes the property of connected components in image domain. (3) It is a robust system, which automatically adjusts the parameters of the model according to UAV’s flying altitudes and performs stably. (4) The processing speed of this system is fast, which provides the potential for real-time applications in the future.