Abstract
This paper presents high precision control and deep learning-based corn stand counting algorithms for a low-cost, ultra-compact 3D printed and autonomous field robot for agricultural operations. Currently, plant traits, such as emergence rate, biomass, vigor, and stand counting, are measured manually. This is highly labor-intensive and prone to errors. The robot, termed TerraSentia, is designed to automate the measurement of plant traits for efficient phenotyping as an alternative to manual measurements. In this paper, we formulate a Nonlinear Moving Horizon Estimator that identifies key terrain parameters using onboard robot sensors and a learning-based Nonlinear Model Predictive Control that ensures high precision path tracking in the presence of unknown wheel-terrain interaction. Moreover, we develop a machine vision algorithm designed to enable an ultra-compact ground robot to count corn stands by driving through the fields autonomously. The algorithm leverages a deep network to detect corn plants in images, and a visual tracking model to re-identify detected objects at different time steps. We collected data from 53 corn plots in various fields for corn plants around 14 days after emergence (stage V3 - V4). The robot predictions have agreed well with the ground truth with \(C_{robot}=1.02 \times C_{human}-0.86\) and a correlation coefficient \(R=0.96\). The mean relative error given by the algorithm is \(-3.78\%\), and the standard deviation is \(6.76\%\). These results indicate a first and significant step towards autonomous robot-based real-time phenotyping using low-cost, ultra-compact ground robots for corn and potentially other crops.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abendroth, L.J., Elmore, R.W., Boyer, M.J., & Marlay, S.K. (2011). Corn growth and development. PMR 1009. Iowa State University
Araus, J. L., & Cairns, J. E. (2014). Field high-throughput phenotyping: The new crop breeding frontier. Trends in Plant Science, 19(1), 52–61.
Biber, P., Weiss, U., Dorna, M., & Albert, A. (2012). Navigation system of the autonomous agricultural robot bonirob. In Workshop on Agricultural Robotics: Enabling Safe, Efficient, and Affordable Robots for Food Production (Collocated with IROS 2012), Vilamoura, Portugal
Chen, S. W., Shivakumar, S. S., Dcunha, S., Das, J., Okon, E., Qu, C., et al. (2017). Counting apples and oranges with deep learning: A data-driven approach. IEEE Robotics and Automation Letters, 2(2), 781–788.
Das, J., Cross, G., Qu, C., Makineni, A., Tokekar, P., Mulgaonkar, Y., & Kumar, V. (2015). Devices, systems, and methods for automated monitoring enabling precision agriculture. In 2015 IEEE International Conference on Automation Science and Engineering (CASE), IEEE, (pp. 462–469).
Diehl, M., Bock, H., Schölder, J. P., Findeisen, R., Nagy, Z., & Allgöwer, F. (2002). Real-time optimization and nonlinear model predictive control of processes governed by differential-algebraic equations. Journal of Process Control, 12(4), 577–585.
Ferreau, H., Kraus, T., Vukov, M., Saeys, W., Diehl, M. (2012) High-speed moving horizon estimation based on automatic code generation. In Decision and Control (CDC), 2012 IEEE 51st Annual Conference on (pp. 687–692).
Furbank, R. T., & Tester, M. (2011). Phenomics-technologies to relieve the phenotyping bottleneck. Trends in plant science, 16(12), 635–644.
Halstead, M., McCool, C., Denman, S., Perez, T., & Fookes, C. (2018). Fruit quantity and quality estimation using a robotic vision system. arXiv preprint arXiv:1801.05560
Haseltine, E. L., & Rawlings, J. B. (2005). Critical evaluation of extended kalman filtering and moving-horizon estimation. Industrial and Engineering Chemistry Research, 44(8), 2451–2460. https://doi.org/10.1021/ie034308l.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp 770–778).
He, K., Gkioxari, G., Dollár, P., & Girshick, R. (2017) Mask r-cnn. In Computer Vision (ICCV), 2017 IEEE International Conference on, IEEE (pp. 2980–2988).
Henriques, J. F., Caseiro, R., Martins, P., & Batista, J. (2015). High-speed tracking with kernelized correlation filters. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(3), 583–596.
van Henten, E. J., Goense, D., & Lokhorst, C. (2009). Precision agriculture. The Netherlands: Wageningen Academic Publishers.
Huang, J., Rathod, V., Sun, C., Zhu, M., Korattikara, A., Fathi, A., Fischer, I., Wojna, Z., Song, Y., & Guadarrama, S, et al. (2017) Speed/accuracy trade-offs for modern convolutional object detectors. In IEEE CVPR, vol 4.
Iagnemma, K., Kang, S., Shibly, H., & Dubowsky, S. (2004). Online terrain parameter estimation for wheeled mobile robots with application to planetary rovers. IEEE Transactions on Robotics, 20(5), 921–927. https://doi.org/10.1109/TRO.2004.829462.
Kayacan, E., & Chowdhary, G. (2019). Tracking error learning control for precise mobile robot path tracking in outdoor environment. Journal of Intelligent and Robotic Systems, 95(3–4), 975–986.
Kayacan, E., Kayacan, E., Ramon, H., & Saeys, W. (2013) Modeling and identification of the yaw dynamics of an autonomous tractor. In 2013 9th Asian Control Conference (ASCC) (pp. 1–6).
Kayacan, E., Peschel, J.M., & Kayacan, E. (2016). Centralized, decentralized and distributed nonlinear model predictive control of a tractor-trailer system: A comparative study. In 2016 American Control Conference (ACC) (pp. 4403–4408), https://doi.org/10.1109/ACC.2016.7525615
Kayacan, E., Kayacan, E., Chen, I. M., Ramon, H., & Saeys, W. (2018a). On the Comparison of Model-Based and Model-Free Controllers in Guidance (pp. 49–73). Navigation and Control of Agricultural Vehicles: Springer International Publishing, Cham.
Kayacan, E., Saeys, W., Ramon, H., Belta, C., & Peschel, J. M. (2018b). Experimental validation of linear and nonlinear mpc on an articulated unmanned ground vehicle. IEEE/ASME Transactions on Mechatronics, 23(5), 2023–2030.
Kayacan, E., Young, S. N., Peschel, J. M., & Chowdhary, G. (2018c). High-precision control of tracked field robots in the presence of unknown traction coefficients. Journal of Field Robotics, 35(7), 1050–1062.
Kayacan, E., Zhang, Z., & Chowdhary, G. (2018d). Embedded high precision control and corn stand counting algorithms for an ultra-compact 3d printed field robot. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania, https://doi.org/10.15607/RSS.2018.XIV.036
Krizhevsky, A., Sutskever, I., & Hinton, G. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097–1105).
Lee, S.U., Iagnemma, K. (2016). Robust motion planning methodology for autonomous tracked vehicles in rough environment using online slip estimation. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), (pp. 3589–3594), https://doi.org/10.1109/IROS.2016.7759528
Lee, S.U., Gonzalez, R., Iagnemma, K. (2016). Robust sampling-based motion planning for autonomous tracked vehicles in deformable high slip terrain. In 2016 IEEE International Conference on Robotics and Automation (ICRA), (pp. 2569–2574), https://doi.org/10.1109/ICRA.2016.7487413
Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C.L. (2014). Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740–755). Springer
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., & Berg, A.C. (2015). Ssd: Single shot multibox detector. arXiv preprint arXiv:1512.02325
Liu, X., Chen, S.W., Aditya, S., Sivakumar, N., Dcunha, S., Qu, C., Taylor, C.J., Das, J., Kumar, V. (2018). Robust fruit counting: Combining deep learning, tracking, and structure from motion. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (pp. 1045–1052)
Liu, X., Chen, S. W., Liu, C., Shivakumar, S. S., Das, J., Taylor, C. J., et al. (2019). Monocular camera based fruit counting and mapping with semantic data association. IEEE Robotics and Automation Letters, 4(3), 2296–2303.
Long, J., Shelhamer, E., & Darrell, T. (2015) Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp 3431–3440)
Lu, H., Cao, Z., Xiao, Y., Zhuang, B., & Shen, C. (2017). Tasselnet: Counting maize tassels in the wild via local counts regression network. Plant methods, 13(1), 79.
Mayne, D., Rawlings, J., Rao, C., & Scokaert, P. (2000). Constrained model predictive control: Stability and optimality. Automatica, 36(6), 789–814.
Mehndiratta, M., Kayacan, E., Patel, S., Kayacan, E., & Chowdhary, G. (2019). Learning-Based fast nonlinear model predictive control for custom-made 3D printed ground and aerial robots (pp. 581–605). Cham: Springer.
Mueller-Sim, T., Jenkins, M., Abel, J., & Kantor, G. (2017). The robotanist: A ground-based agricultural robot for high-throughput crop phenotyping. In 2017 IEEE International Conference on Robotics and Automation (ICRA) (pp 3634–3639), https://doi.org/10.1109/ICRA.2017.7989418
Rahnemoonfar, M., & Sheppard, C. (2017). Deep count: Fruit counting based on deep simulated learning. Sensors, 17(4), 905.
Ray, L. E. (2009). Estimation of terrain forces and parameters for rigid-wheeled vehicles. IEEE Transactions on Robotics, 25(3), 717–726. https://doi.org/10.1109/TRO.2009.2018971.
Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems(pp. 91–99)
Shafiekhani, A., Kadam, S., Fritschi, F. B., & DeSouza, G. N. (2017). Vinobot and vinoculer: Two robotic platforms for high-throughput field phenotyping. Sensors, 17(1). https://doi.org/10.3390/s17010214
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Stein, M., & Bargoti, S. (1915). Underwood J (2016) Image based mango fruit detection, localisation and yield estimation using multiple view geometry. Sensors, 16(11). https://doi.org/10.3390/s16111915
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–9)
Young, S. N., Kayacan, E., & Peschel, J. M. (2019). Design and field evaluation of a ground robot for high-throughput phenotyping of energy sorghum. Precision Agriculture, 20(4), 697–722.
Acknowledgements
The information, data, or work presented herein was funded in part by the Advanced Research Projects Agency-Energy (ARPA-E), U.S. Department of Energy, under Award Number DE-AR0000598. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof.
Erkan Kayacan and Zhongzhong Zhang made equal contributions to this work.
Dr. Kayacan was a postdoctoral researcher in Chowdhary’s group when the bulk of this work was undertaken. He is a Lecturer at the University of Queensland since April 2019.
Any questions or comments should be directed to Girish Chowdhary. We thank UIUC-IGB TERRA-MEPP team and EarthSense Inc. for valuable suggestions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This is one of the several papers published in Autonomous Robots comprising the Special Issue on Robotics: Science and Systems.
Rights and permissions
About this article
Cite this article
Zhang, Z., Kayacan, E., Thompson, B. et al. High precision control and deep learning-based corn stand counting algorithms for agricultural robot. Auton Robot 44, 1289–1302 (2020). https://doi.org/10.1007/s10514-020-09915-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10514-020-09915-y