Elsevier

Agricultural Systems

Volume 192, August 2021, 103194
Agricultural Systems

Short Communication
Revisiting linear regression to test agreement in continuous predicted-observed datasets

https://doi.org/10.1016/j.agsy.2021.103194Get rights and content

Highlights

  • This work provides a novel perspective on the use of linear regression to test models' performance.

  • For testing agreement in predicted-observed data, a linear regression model should satisfy symmetry.

  • With unknown uncertainty of predictions and observations, the standardized major axis regression is recommended.

  • With standardized major axis regression, we can obtain error components related to both lack of accuracy and precision.

  • We offer a detailed tutorial assisting users to perform the proposed analysis in R-software.

Abstract

CONTEXT

In agricultural research and related disciplines, using a scatter plot and a regression line to visually and quantitatively assess agreement between model predictions and observed values is an extensively adopted approach, even more within the simulation modeling community. However, linear model fit, use, and interpretation are still controversial in the literature.

OBJECTIVE

The overall goal of this research is to evaluate the usefulness of a symmetric regression line to test agreement on predicted-observed datasets. The specific aims of this study are to: i) discuss the selection of a regression model to fit a line to the predicted-observed scatter, and ii) provide a geometric interpretation of the regression line, decomposing the prediction error into lack of accuracy and lack of precision components, via utilization of illustrative field crop datasets.

METHODS

This study tested and contrasted three alternative linear regression models (Ordinary Least Squares -OLS-, Major Axis -MA-, and Standardized Major Axis -SMA-) in terms of assumptions, loss functions, parameters estimates, and model interpretation for the predicted-observed case.

RESULTS AND CONCLUSIONS

When the uncertainty of predictions and observations are unknown, the SMA represents the most appropriate approach to fit a symmetric-line describing the bivariate predicted-observed scatter. The SMA-line serves as a reference to estimate a weighed difference between predictions and observations. Moreover, this symmetric regression can assist in the decomposition of the square error into additive components related to both lack of accuracy and precision. In summary, the SMA regression tackles the axis orientation problem of the traditional OLS (y vs. x or x or y) and allows to identify error sources that are meaningful to the user.

SIGNIFICANCE

This work offers a novel and simple perspective about the use of linear regression to assess simulation models performance. In order to assist potential users, we also provide a tutorial to compute the proposed assessment of agreement using R-software.

Introduction

Accurate and precise predictions are the ideal outcome of any simulation model. Accuracy refers to the closeness between predicted (P) and observed (O), linked to systematic error or bias. Precision relates to dispersion, or proximity between data points, connected to random variability. Simulations could be both accurate and precise, accurate but imprecise, precise but inaccurate, or inaccurate and imprecise (Fig. 1). The level of agreement is conditional to these two concepts, essential for assessing models' performance (Gauch et al., 2003; Tedeschi, 2006).

A broad set of scoring rules were designed to capture different aspects of agreement (Duveiller et al., 2016; Tedeschi, 2006). Perhaps, the mean square error (MSE) and its square root (RMSE) are the most popular in academia (Gneiting, 2011). The coefficients of correlation (r) and determination (R2) are also widely used for model evaluation, but provide limited information about agreement (Yang et al., 2014). Alternatively, the concordance correlation coefficient (CCC) (Lin, 1989) is another popular normalized metric to evaluate both accuracy and precision at the same time. Although a myriad of additional agreement indices have been developed (Gupta et al., 2009; Moriasi et al., 2007; Willmott et al., 2012, among others), the visual assessment with a scatter plot and a regression line is still widely used in agricultural and related research areas (Piñeiro et al., 2008). A scatter plot presents the advantage of showing data distribution and dispersion patterns (Loague and Green, 1991; Willmott, 1981). Similarly, although there are objections to use linear regression (Harrison, 1990; Kobayashi and Salam, 2000), it is still commonly used to test a null hypothesis of agreement, with the H0: intercept = 0 and slope = 1 (Analla, 1998; Smith and Rose, 1995; Yang et al., 2014).

The ordinary least squares (OLS) is perhaps the most widely adopted model for linear regression. However, for the P-O case, a lack of accord persists related to the scatter's orientation (Piñeiro et al., 2008), dependent and independent variables (Analla, 1998). To the present, there are three prominent positions in the literature. The first supports the PO orientation by considering O as reference (error-free), so using O as the regressor variable and P as the dependent (y) (Willmott, 1981; Yang et al., 2014). A second approach supports the OP orientation, arguing that only O contains natural variability whereas P comes mostly from deterministic models (Mayer and Butler, 1993; Tedeschi, 2006) and considering that PO orientation distorts the interpretation of the relationship (Piñeiro et al., 2008). The third position supports that the orientation does not matter (Mitchell, 1997) since both P and O contain random error, arguing that not acknowledging the uncertainty on predictions (i.e., deterministic) does not imply a null uncertainty (St-Pierre, 2016).

Alternatively, bivariate regression models are characterized by their symmetry, that is, invariant to the axis orientation (Draper and Smith, 1998; Smith, 2009). Representing this group, the major axis (MA) and standardized MA (SMA) regressions are dimension-reduction techniques producing one-dimensional summaries of the scatter (Jolliffe, 2002; Warton and Weber, 2002), broadly used in biology for testing proportionality (isometry vs. allometry) between random variables (Warton et al., 2006). Therefore, a symmetric regression model may represent a suitable alternative for the P-O case.

Disaggregating the prediction error is also a major concern for model evaluation that deserves attention (Gauch et al., 2003; Kobayashi and Salam, 2000; Wallach and Thorburn, 2017). Revisiting the concept of symmetry and the decomposition of the error using bivariate regression models, new methods have been developed to compare satellite images that could be applied for the P-O case (Ji and Gallo, 2006; Duveiller et al., 2016).

The main objective of this study is to discuss the usefulness of a symmetric regression line to test agreement on continuous P-O datasets. Our specific goals are to: i) discuss the selection of the regression model, and ii) propose a geometric interpretation of the square error producing both lack of accuracy and precision. Lastly, we offer a tutorial in R Software (R Core Team, 2020) for this analysis publicly available at:https://doi.org/10.7910/DVN/EJS4M0 (Correndo et al., 2021).

Section snippets

The general error-in-variables model

Either to predict or to observe a quantity (e.g. crop yield) are both techniques producing values with uncertainty, whether it is acknowledged or not (St-Pierre, 2016). To compare how equivalent two techniques are we could define a general model (Francq and Govaerts, 2014), which is referred in the literature as “error-in-variables” model (Moran, 1971), measurement error models (Fuller, 1987), or Model-II regression (Legendre and Legendre, 1998). For illustrative purposes, we first use the

Illustrative dataset

The hypothetical dataset used to illustrate Fig. 2, Fig. 3 has been intentionally set to n = 10; x = 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0; and y = 4.0, 5.5, 2.5, 4.5, 8.0, 5.0, 6.0, 10.0, 7.5, 8.5. At Fig. 2, variables are in-purpose named as x and y, so either of them could correspond to P or O. The OLSv approach (Fig. 2A) results in a regression line ŷOLSv = 2.47 + 0.57×; while using OLSh (Fig. 2B), the regression line is ŷOLSh = −0.52 + 1.02×. Moreover, while the OLSv slope

Discussion

This article provides a novel perspective on using linear regression to test agreement between P and O values. Previous research in ecology has primarily discussed the axis-orientation, however, constrained to the OLS regression (Piñeiro et al., 2008). In this study, we have integrated concepts from methodological research developed at other disciplines including but not limited to: biometry (Jolicoeur, 1990; Warton et al., 2006), astronomy (Isobe et al., 1990), chemistry (Francq and Govaerts,

Conclusions

This manuscript explains the underlying theory, formulae, and illustrative examples to guide the selection of a linear regression model for testing agreement in continuous P-O datasets. We argue the need for a symmetric regression with an interpretation invariant to the axis orientation highlighting the adequacy of the SMA model over other alternatives. Beyond the classical hypothesis testing of the regression-line, our SMA-based approach offers a simple error decomposition producing metrics

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

Authors express their gratitude to the financial support provided by the Fulbright Program (Argentina), Kansas State University, the Kansas Corn Commission, and Corteva Agriscience for sponsoring Mr. Correndo's Ph.D. and Dr. Ciampitti's research program. Authors also give special thanks to the APSIM Initiative for providing the illustrative datasets. This manuscript is contribution No. 21-316-J from the Kansas Agricultural Experiment Station.

References (53)

  • P.L. Mitchell

    Misuse of regression for empirical validation of models

    Agric. Syst.

    (1997)
  • P.A.P. Moran

    Estimating structural and functional relationships

    J. Multivar. Anal.

    (1971)
  • G. Piñeiro et al.

    How to evaluate models: observed vs. predicted or predicted vs. observed?

    Ecol. Model.

    (2008)
  • E.P. Smith et al.

    Model goodness-of-fit analysis using regression and related techniques

    Ecol. Model.

    (1995)
  • N.R. St-Pierre

    Comparison of model predictions with measurements: a novel model-assessment method

    J. Dairy Sci.

    (2016)
  • L.O. Tedeschi

    Assessment of the adequacy of mathematical models

    Agric. Syst.

    (2006)
  • D. Wallach et al.

    Estimating uncertainty in crop models: current situation and future prospects

    Eur. J. Agron.

    (2017)
  • D. Wallach et al.

    Chapter 9: Model evaluation

  • J.M. Yang et al.

    An evaluation of the statistical methods for testing the performance of crop models with observed data

    Agric. Syst.

    (2014)
  • F. Barker et al.

    Properties of the geometric mean functional relationship

    Biometrics

    (1988)
  • R.J. Carroll et al.

    The use and misuse of orthogonal regression in linear errors-in-variables models

    Am. Stat.

    (1996)
  • A.A. Correndo et al.

    R-Code Tutorial: Revisiting linear regression to test agreement in continuous predicted-observed datasets

    Harvard Dataverse

    (2021)
  • N.R. Draper et al.

    Applied Regression analysis

  • G. Duveiller et al.

    Revisiting the concept of a symmetric index of agreement for continuous datasets

    Sci. Rep.

    (2016)
  • W.A. Fuller

    Measurement Error Models

    (1987)
  • H.G. Gauch et al.

    Model evaluation by comparison of model-based predictions and measured values

    Agron. J.

    (2003)
  • Cited by (14)

    • On-farm soybean seed protein and oil prediction using satellite data

      2023, Computers and Electronics in Agriculture
    • Interpretable machine learning methods to explain on-farm yield variability of high productivity wheat in Northwest India

      2022, Field Crops Research
      Citation Excerpt :

      Importantly, many of these approaches differ from those associated with hypothesis-driven, manipulative experimentation, and deductive research (de Mauro et al., 2016). Despite the wealth of studies identifying important variables governing crop yield variability (e.g., Correndo et al., 2021; Park et al., 2018), studies that employ large-scale observations of farmers’ management practices to assess how these affect crop yield directly or in interaction with other practices are less common (e.g., Tseng et al., 2021; Devkota and Yigezu, 2020; Di Mauro et al., 2018). Machine learning lies at the intersection of computer science and statistics, and has been used to unravel patterns in large datasets that are challenging to analyze with more conventional statistical approaches (Tolle et al., 2011).

    View all citing articles on Scopus
    View full text