Short CommunicationRevisiting linear regression to test agreement in continuous predicted-observed datasets
Graphical abstarct
Introduction
Accurate and precise predictions are the ideal outcome of any simulation model. Accuracy refers to the closeness between predicted (P) and observed (O), linked to systematic error or bias. Precision relates to dispersion, or proximity between data points, connected to random variability. Simulations could be both accurate and precise, accurate but imprecise, precise but inaccurate, or inaccurate and imprecise (Fig. 1). The level of agreement is conditional to these two concepts, essential for assessing models' performance (Gauch et al., 2003; Tedeschi, 2006).
A broad set of scoring rules were designed to capture different aspects of agreement (Duveiller et al., 2016; Tedeschi, 2006). Perhaps, the mean square error (MSE) and its square root (RMSE) are the most popular in academia (Gneiting, 2011). The coefficients of correlation (r) and determination (R2) are also widely used for model evaluation, but provide limited information about agreement (Yang et al., 2014). Alternatively, the concordance correlation coefficient (CCC) (Lin, 1989) is another popular normalized metric to evaluate both accuracy and precision at the same time. Although a myriad of additional agreement indices have been developed (Gupta et al., 2009; Moriasi et al., 2007; Willmott et al., 2012, among others), the visual assessment with a scatter plot and a regression line is still widely used in agricultural and related research areas (Piñeiro et al., 2008). A scatter plot presents the advantage of showing data distribution and dispersion patterns (Loague and Green, 1991; Willmott, 1981). Similarly, although there are objections to use linear regression (Harrison, 1990; Kobayashi and Salam, 2000), it is still commonly used to test a null hypothesis of agreement, with the H0: intercept = 0 and slope = 1 (Analla, 1998; Smith and Rose, 1995; Yang et al., 2014).
The ordinary least squares (OLS) is perhaps the most widely adopted model for linear regression. However, for the P-O case, a lack of accord persists related to the scatter's orientation (Piñeiro et al., 2008), dependent and independent variables (Analla, 1998). To the present, there are three prominent positions in the literature. The first supports the PO orientation by considering O as reference (error-free), so using O as the regressor variable and P as the dependent (y) (Willmott, 1981; Yang et al., 2014). A second approach supports the OP orientation, arguing that only O contains natural variability whereas P comes mostly from deterministic models (Mayer and Butler, 1993; Tedeschi, 2006) and considering that PO orientation distorts the interpretation of the relationship (Piñeiro et al., 2008). The third position supports that the orientation does not matter (Mitchell, 1997) since both P and O contain random error, arguing that not acknowledging the uncertainty on predictions (i.e., deterministic) does not imply a null uncertainty (St-Pierre, 2016).
Alternatively, bivariate regression models are characterized by their symmetry, that is, invariant to the axis orientation (Draper and Smith, 1998; Smith, 2009). Representing this group, the major axis (MA) and standardized MA (SMA) regressions are dimension-reduction techniques producing one-dimensional summaries of the scatter (Jolliffe, 2002; Warton and Weber, 2002), broadly used in biology for testing proportionality (isometry vs. allometry) between random variables (Warton et al., 2006). Therefore, a symmetric regression model may represent a suitable alternative for the P-O case.
Disaggregating the prediction error is also a major concern for model evaluation that deserves attention (Gauch et al., 2003; Kobayashi and Salam, 2000; Wallach and Thorburn, 2017). Revisiting the concept of symmetry and the decomposition of the error using bivariate regression models, new methods have been developed to compare satellite images that could be applied for the P-O case (Ji and Gallo, 2006; Duveiller et al., 2016).
The main objective of this study is to discuss the usefulness of a symmetric regression line to test agreement on continuous P-O datasets. Our specific goals are to: i) discuss the selection of the regression model, and ii) propose a geometric interpretation of the square error producing both lack of accuracy and precision. Lastly, we offer a tutorial in R Software (R Core Team, 2020) for this analysis publicly available at:https://doi.org/10.7910/DVN/EJS4M0 (Correndo et al., 2021).
Section snippets
The general error-in-variables model
Either to predict or to observe a quantity (e.g. crop yield) are both techniques producing values with uncertainty, whether it is acknowledged or not (St-Pierre, 2016). To compare how equivalent two techniques are we could define a general model (Francq and Govaerts, 2014), which is referred in the literature as “error-in-variables” model (Moran, 1971), measurement error models (Fuller, 1987), or Model-II regression (Legendre and Legendre, 1998). For illustrative purposes, we first use the
Illustrative dataset
The hypothetical dataset used to illustrate Fig. 2, Fig. 3 has been intentionally set to n = 10; x = 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0; and y = 4.0, 5.5, 2.5, 4.5, 8.0, 5.0, 6.0, 10.0, 7.5, 8.5. At Fig. 2, variables are in-purpose named as x and y, so either of them could correspond to P or O. The OLSv approach (Fig. 2A) results in a regression line = 2.47 + 0.57×; while using OLSh (Fig. 2B), the regression line is = −0.52 + 1.02×. Moreover, while the OLSv slope
Discussion
This article provides a novel perspective on using linear regression to test agreement between P and O values. Previous research in ecology has primarily discussed the axis-orientation, however, constrained to the OLS regression (Piñeiro et al., 2008). In this study, we have integrated concepts from methodological research developed at other disciplines including but not limited to: biometry (Jolicoeur, 1990; Warton et al., 2006), astronomy (Isobe et al., 1990), chemistry (Francq and Govaerts,
Conclusions
This manuscript explains the underlying theory, formulae, and illustrative examples to guide the selection of a linear regression model for testing agreement in continuous P-O datasets. We argue the need for a symmetric regression with an interpretation invariant to the axis orientation highlighting the adequacy of the SMA model over other alternatives. Beyond the classical hypothesis testing of the regression-line, our SMA-based approach offers a simple error decomposition producing metrics
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgements
Authors express their gratitude to the financial support provided by the Fulbright Program (Argentina), Kansas State University, the Kansas Corn Commission, and Corteva Agriscience for sponsoring Mr. Correndo's Ph.D. and Dr. Ciampitti's research program. Authors also give special thanks to the APSIM Initiative for providing the illustrative datasets. This manuscript is contribution No. 21-316-J from the Kansas Agricultural Experiment Station.
References (53)
Model validation through the linear regression fit to actual versus predicted values
Agric. Syst.
(1998)- et al.
Estimation of the concordance correlation coefficient for repeated measures using SAS and R
Comput. Methods Prog. Biomed.
(2013) - et al.
Measurement methods comparison with errors-in-variables regressions. From horizontal to vertical OLS regression, review and new perspectives
Chemom. Intell. Lab. Syst.
(2014) - et al.
Decomposition of the mean squared error and NSE performance criteria: implications for improving hydrological modelling
J. Hydrol.
(2009) Regression of a model on real-system output: an invalid test of model validity
Agric. Syst.
(1990)- et al.
APSIM next generation: overcoming challenges in modernising a farming systems model
Environ. Model. Softw.
(2018) Bivariate allometry: interval estimation of the slopes of the ordinary and standardized normal major axes and structural relationship
J. Theor. Biol.
(1990)- et al.
Statistical and graphical methods for evaluating solute transport models: overview and application
J. Contam. Hydrol.
(1991) A simple bayesian method for adjusting ensemble of crop model outputs to yield observations
Eur. J. Agron.
(2017)- et al.
Statistical validation
Ecol. Model.
(1993)
Misuse of regression for empirical validation of models
Agric. Syst.
Estimating structural and functional relationships
J. Multivar. Anal.
How to evaluate models: observed vs. predicted or predicted vs. observed?
Ecol. Model.
Model goodness-of-fit analysis using regression and related techniques
Ecol. Model.
Comparison of model predictions with measurements: a novel model-assessment method
J. Dairy Sci.
Assessment of the adequacy of mathematical models
Agric. Syst.
Estimating uncertainty in crop models: current situation and future prospects
Eur. J. Agron.
Chapter 9: Model evaluation
An evaluation of the statistical methods for testing the performance of crop models with observed data
Agric. Syst.
Properties of the geometric mean functional relationship
Biometrics
The use and misuse of orthogonal regression in linear errors-in-variables models
Am. Stat.
R-Code Tutorial: Revisiting linear regression to test agreement in continuous predicted-observed datasets
Harvard Dataverse
Applied Regression analysis
Revisiting the concept of a symmetric index of agreement for continuous datasets
Sci. Rep.
Measurement Error Models
Model evaluation by comparison of model-based predictions and measured values
Agron. J.
Cited by (14)
Satellite-based woody canopy cover for Africa: Uncovering bias and recovering best estimates across years
2024, Science of Remote SensingA new spatial estimation model and source apportionment of aliphatic hydrocarbons in coastal surface sediments of the Nayband Bay, Persian Gulf
2023, Science of the Total EnvironmentOn-farm soybean seed protein and oil prediction using satellite data
2023, Computers and Electronics in AgricultureThe urgency for investment on local data for advancing food assessments in Africa: A review case study for APSIM crop modeling
2023, Environmental Modelling and SoftwareInterpretable machine learning methods to explain on-farm yield variability of high productivity wheat in Northwest India
2022, Field Crops ResearchCitation Excerpt :Importantly, many of these approaches differ from those associated with hypothesis-driven, manipulative experimentation, and deductive research (de Mauro et al., 2016). Despite the wealth of studies identifying important variables governing crop yield variability (e.g., Correndo et al., 2021; Park et al., 2018), studies that employ large-scale observations of farmers’ management practices to assess how these affect crop yield directly or in interaction with other practices are less common (e.g., Tseng et al., 2021; Devkota and Yigezu, 2020; Di Mauro et al., 2018). Machine learning lies at the intersection of computer science and statistics, and has been used to unravel patterns in large datasets that are challenging to analyze with more conventional statistical approaches (Tolle et al., 2011).