2.1 Soils and data sets

To establish the Swiss SSL, we obtained soil samples and reference data from two different sources, i.e. (1) the Swiss Soil Monitoring Network (NABO) and (2) the Swiss Biodiversity Monitoring (BDM) program (BAFU, 2014, Fig. 1). NABO currently consists of 108 sites where soils have been continuously measured every 5 years since 1985 for long-term soil monitoring. Out of the 108 sites, we selected 71 sites under agricultural management – comprising arable land (33 sites), permanent grassland (26 sites) and special crops (11 sites; horticulture) – and one protected area. For the mid-IR SSL, we used 596 NABO soil samples from six campaigns conducted between 1985 and 2015.

Figure 1Map of Switzerland with sampling locations of the mid-IR spectral library, including the sites of the Biodiversity Monitoring program (BDM; 6 × 4 km; n=1094) and the National Soil Monitoring Network (NABO; n=71). In total, 71 NABO sites (10 m × 10 m) were sampled with a grid-based stratified design, and 1094 BDM samples were obtained from single sampling events. The NABO sites have been continuously sampled and measured in 5-year intervals since 1985.

The plots at the NABO sites covered 10 m × 10 m each. These were repeatedly sampled for 0–20 cm soil depth. In total, four replicate samples were taken by stratified random sampling and bulking 4 × 25 cores from 100 subareas of 1 m² to account for small-scale soil variability. Desaules et al. (2010) and Gubler et al. (2019) detailed the sample collection and data harmonization process of the measurements. The soils of the BDM were sampled at 0–0.2 m depth from positions on a regular grid of 6 km × 4 km laid over Switzerland (a total of 1094 locations). The points that were not sampled were inaccessible; these were mostly in the alpine regions. Each sampled location included four subsamples that were taken at the intersection of the four cardinal directions from the center point and the circumference of a circle with a radius of 3 to 3.5 m (Meuli et al., 2017). Due to its design which covers all major geographic regions in Switzerland – the Jura Mountains, the Central Plateau and the Alps – the BDM sampling campaign comprises a major part of the biogeochemical diversity of soils and predominant land use types in Switzerland. The wide coverage of soil conditions are an important source of soil chemical variability.

2.2 Chemical reference analysis

Data on chemical and physical soil properties were previously measured and provided by the NABO group. All laboratory soil analyses for the 16 properties were based on the protocols of the Swiss standard method (Agroscope, 1996). Mineral elements were determined by extraction with 1 : 10 ammonium acetate–EDTA solution (AAE10; method following Agroscope, 1996). The measured properties were total C, organic C (OC), total nitrogen (N), pH (CaCl₂), CaCO₃, clay, silt, sand, CEC_pot, P(AAE), K(AAE), Ca(AAE), Mg(AAE), Cu(AAE), Zn(AAE) and Fe(AAE). For samples of BDM and for the more recent NABO sampling campaigns five and six (years 2009–2014), the total C and N measurements were done with dry combustion (LECO TruSpec). For campaigns one through four (years 1985–2014), the OC contents determined with wet oxidation using a modified Walkley–Black method were transformed into dry combustion equivalents, using site-specific robust linear regressions (complementary data of campaigns five and six; Gubler et al., 2018). Carbonates were determined by volumetric calcimetry, using hydrochloric acid (HCl) for digestion. Organic C was obtained by the difference in total C and carbonate C when pH was greater than 6.5. Inorganic (carbonate) C was calculated with 0.12 × CaCO₃. The texture was determined by the pipette method. The pH was measured in CaCl₂, using a 1 : 2 volumetric ratio of soil to water. For CEC_pot, the exchangeable elements were extracted with a 0.05 N–0.025 N HCl-H₂SO₄ solution, which was buffered with triethanolamine for soil samples with pH > 5.9. All soil properties were referenced to dry weight by water correction after drying at 105 ^∘C. All chemical analyses of NABO soils were done on four bulked replicates per site and sampling event. For BDM locations, four spatial replicates were measured each.

2.3 Measuring and processing spectra

All milled soil samples from the NABO and the BDM archive (n=4374; with a particle size below 100 µm) were measured with the VERTEX 70v Fourier transform spectrometer from Bruker (Bruker Optik GmbH, Ettlingen, Germany) at ETH Zurich, using a high-throughput accessory (HTS-XT) and custom 24-well plates tailored to diffuse reflectance measurements. The mid-IR spectrometer was equipped with a KBr beam splitter and a mercury cadmium telluride (MCT) detector, which was permanently cooled with liquid nitrogen during the measurements. The reflectance spectra were acquired between 7500 cm⁻¹ (1333.3 nm) and 600 cm⁻¹ (16 666.7 nm) at an effective resolution of 2 cm⁻¹ and trimmed to the mid-IR range between 3996 and 600 cm⁻¹ before further processing (see below).

Each soil sample was measured twice. The soil surface was flattened evenly and without compression by the thin, round middle part of the spatula. The first measurement position of the 24-well plate contained a gold (Au) reference surface, which produced a single reflectance spectrum for normalizing the reflectance of the 23 following soil measurements. The “atmospheric compensation” routine, implemented in the Bruker OPUS software, was used to eliminate unwanted absorptions of H₂O vapor continuum and CO₂ gas in the measurement chamber, based on the single channel reference spectrum measured once on each plate. All single channel reflectance spectra were obtained by averaging 32 internal measurements.

The resulting reflectance spectra (R; background referenced) were converted to apparent absorbance (A) by $A={\log }_{\mathrm{10}}(\mathrm{1}/R)$. Then, an average spectrum per sample was produced by calculating the mean of all spectral variables for the measured replicates. Finally, the spectrum offset and further scatter effects were reduced, and the features were transformed with a Savitzky–Golay (Savitzky and Golay, 1964) first derivative smoother using a window size of 35 variables (70 cm⁻¹) and third-order polynomial fit. Finally, we selected every eighth spectral variable to reduce redundancy in the spectra (collinearity) and produce more parsimonious spectral estimates of soil properties. This resulted in 209 variables between 634 and 3962 cm⁻¹, which formed the predictors for the subsequent general and local transfer modeling.

2.4 Data processing and statistical computing

All spectral and reference data were processed and modeled with the R software environment for statistical computing and graphics (version 3.6.0; R Core Team, 2019). We used the caret (Kuhn, 2020) R package to streamline the statistical learning process. Basic data transformations, such as data preparation and aggregation, were done using the tidyverse (Wickham, 2019) set of packages and data.table (Dowle and Srinivasan, 2019). The spectral data were handled and processed with the simplerspec (Baumann, 2019) and prospectr (Stevens and Ramirez-Lopez, 2013) packages.

2.5 General soil estimation: rules for the entire SSL

The general soil estimation was done by rules trained with the CUBIST (Quinlan, 1993) learner, separately developed for each analytical soil measure. We chose this algorithm because it has shown excellent performance for modeling soil information and developing SSLs with rather large soil variability and multicollinear spectral variables (Bui et al., 2006; Viscarra Rossel and Webster, 2012; Stevens et al., 2013; Miller et al., 2015; Peng et al., 2015; Viscarra Rossel et al., 2016; Dangal et al., 2019; Padarian et al., 2019b), and because its interpretation is mechanistically more intuitive as it is a form of data partitioning (simple conditions and linear equations). CUBIST first forms model trees, using basic mechanisms of M5 (Quinlan, 1992). CUBIST is a form of a rule-based decision tree with piecewise linear models. Wang and Witten (1996) outlined detailed principles behind the construction of the model trees and derivation of rules, and Viscarra Rossel and Webster (2012) described it for soil spectroscopic modeling.

A CUBIST prediction rule is a unique set of conditions, i.e., “if, then” logical statements, together with the associated ordinary linear regression model. During training, the condensed regression equations are made for samples in the terminal nodes. All preceding split variables are potentially allowed for regression in a final node; however, some of them are pruned or combined in the rules. The smoothed regression equations with the selected variables allows one to predict an individual, new observation. CUBIST features two empirical parameters that can improve predictions, namely committees and neighbors. Committees are ensembles of rules that are created by successive construction of trees, which correct predictions of preceding rules and, thereby, lower predictive errors by averaging. When neighbors are used (maximum nine), a new training sample is predicted, using both unweighted or weighted averages of the measured values of the nearest neighbors, using all features in the training set and the prediction of the new sample using the training rule(s).

2.6 Local soil estimation for plot-level monitoring

We defined a local soil estimation scenario where a new long-term monitoring site was initiated at time zero (t₀). Each one of the 71 NABO sites was assumed to be novel, while the remaining ones were established with spectral and reference data records. We, therefore, conducted 71 separate sample selections from the SSL, each yielding different transfer subsets of the SSL, to test spectral-based soil monitoring using the Swiss mid-IR SSL presented here. We calibrated models at each site using two local samples per given site and a relevant subset of the remaining Swiss SSL (see description below).

Figure 2Conceptual scheme illustrating the transfer of the soil spectral library (SSL) to a long-term monitoring site using the RS-LOCAL approach. The local calibration samples and a subset of the SSL are used to calibrate a partial least squares regression (PLSR) model, which predicts the local validation samples.

Download

The two local samples were chosen from pooled samples at t₀ (first two out of a maximum of four replicates) or in addition at t₁ if there was only one sample in t₀. Figure 2 illustrates the local modeling workflow. All other samples per given site besides the two chosen during calibration (in other words, the successive time series measurements at a monitoring plot) were used as local validation samples (N_site). The respective samples from the remaining SSL included spectra and reference measurements from all BDM samples and NABO samples, excluding the ones from the respective target site. We used only two calibration samples per NABO site to capture the predictive mechanisms at site level because we wanted to avoid overoptimistic local assessment; both local calibration and validation samples were repeated soil measurements, and are otherwise – if not adequately handled in the calibration sampling strategy – at risk of overfitting when soils' composition and relevant properties show constant trends over time.

For each of the 71 sites, the spectral relevant samples from the remaining Swiss SSL were selected using the RS-LOCAL algorithm (see Lobsey et al., 2017). The site-specific samples (m_site) denote local calibration samples from a NABO plot. The recursive reductions of the initial training data, which determined the finally yielded subsets (K_site ), were driven by model performance (RMSE) for the two local calibration samples. For each NABO site, the corresponding K_site set was spiked with the two local calibration samples. On this combined m_site+K_site data set, a final partial least squares regression (PLSR) model, locally adapted for the monitoring plot by optimization on the calibration samples, was developed using 10-fold cross-validation. Finally, the local validation spectra (N_site) were predicted using the most accurate calibration model.

The search algorithm RS-LOCAL has three empirical parameters to control the samples that are selected for the local transfer from the SSL (Lobsey et al., 2017). Parameter k is both the number of samples drawn from the original and reduced library without replacement and the number of samples of the returned SSL subset. Parameter b is the number of times k samples are randomly drawn from the remaining data at iteration i of the performance-driven library reduction. Parameter r is the proportion of samples, which are consistently in weakest models, that are removed at it each reduction step. The configuration of the RS-LOCAL search was optimized for each NABO site. For each site, we ran separate RS-LOCAL runs, testing a full-factorial combination of empirical parameter sets $k=\mathit{\{}\mathrm{30},\mathrm{50},\mathrm{150}\mathit{\}}$, $b=\mathit{\{}\mathrm{10},\mathrm{20},\mathrm{50}\mathit{\}}$ and $r=\mathit{\{}\mathrm{0.05},\mathrm{0.1},\mathrm{0.2}\mathit{\}}$. The RS-LOCAL procedure is based on the PLSR (Wold et al., 1983). For the RS-LOCAL tuning during the subset selection procedure and final calibrations, we tested 1 to 10 PLSR components. The finally selected optimal subset per site yielded the smallest RMSE on the two local calibration samples and was therefore used to predict the local validation samples.

3.1 Summary of reference measurements

The samples from the Swiss Soil Monitoring Network (NABO) exhibited the highest variability across samples for total C and OC (n=592; Table 1). Organic C ranged from 1 to 583 g kg⁻¹. The texture of the soils varied considerably. The pH had values between 3.5 and 7.6, and the soils were slightly acidic overall, with a median of 5.8. Compared to the NABO data set, the soils from the BDM program covered a wider set (n=3723 for total C) and range of measured soil properties. The measured range of total C for BDM (1–583 g kg⁻¹) extended further than that of NABO. The distribution of pH values was similar in the NABO and BDM sets. The BDM data also included the available cations extracted by AAE (see Table 1). The median CEC_pot (potential cation exchange capacity) was almost equivalent to the value of the NABO sites (24 vs. 23 $\mathrm{cmol}(+){\mathrm{kg}}^{-\mathrm{1}}$). Exchangeable Ca showed the largest coefficient of variation (CV = 1.56) among the measured properties of the BDM set. All soil properties, except pH and CEC_pot, were positively or neutrally (sand) skewed for both NABO and BDM data sets, respectively.

Table 1Summary statistics of the measured soil properties of the Swiss soil spectral library derived from the sample archive of the Swiss Soil Monitoring Network (NABO) and the Swiss Biodiversity Monitoring (BDM) program. Total C is total carbon, OC is organic carbon and CEC_pot is potential cation exchange capacity.

Download Print Version | Download XLSX

3.2 General soil estimation with CUBIST modeling

For most of the properties, minimal cross-validated errors were achieved with 100 committees and nine neighbors. The rule-based models explained a large proportion of the variation (R²>0.9) in properties that typically have a strong link to total C (organic C and N; Table 2; Fig. 3). Clay was accurately estimated (RMSE = 47 g kg⁻¹; RPIQ = 3.0; range = 0–602 g kg⁻¹), whereas sand and silt were less accurately estimated. The pH was accurately estimated (RMSE = 0.3; RPIQ = 6.5). Our models discriminated a large proportion in the measured variation of Ca and Mg (ammonium acetate–EDTA) in the mid-IR (R² = 0.97 and 0.79; RPIQ = 2.4 and 1.2). Reference values of potential cation exchange capacity ranged from 0 to 136 $\mathrm{cmol}(+){\mathrm{kg}}^{-\mathrm{1}}$ and were modeled with an RMSE of 7 $\mathrm{cmol}(+){\mathrm{kg}}^{-\mathrm{1}}$ (R²=0.72; RPIQ = 2.0). However, the extractable nutrients P, K, Cu and Zn were insufficiently explained by mid-IR spectral rules (R² = 0.05–0.1; RPIQ = 0.4–0.9). Nonetheless, the rules achieved nearly unbiased property estimates over all measurements. We found marginal local bias at the largest values, mostly for variables with positively skewed distributions, such as total C (Table 2; Fig. 3).

Table 2Cross-validated mid-IR estimates of 18 soil properties derived from rule-based CUBIST models that were developed using all available data. The 10-fold cross-validation procedure was grouped by the site and repeated five times to achieve many test–train data combinations and provide a realistic model assessment with generalization. C and N denote the number of committees and neighbors for CUBIST. Total C is total carbon, OC is organic carbon, N is total nitrogen and CEC_pot is potential cation exchange capacity.

Download Print Version | Download XLSX

Figure 3Agreement between measured and mid-IR predicted values that were obtained from CUBIST models. Models' performance was assessed by site-grouped cross-validation holdouts (five times repeated 10-fold). The lines obtained with local regression (LOESS) smoothing indicate the trends in predictions. Models of soil properties with R²≥0.72 are shown (see Table 2 for more detailed model summaries).

Download

Overall, out of the 16 available soil properties, total C, total N, total CaCO₃, Ca and Mg (ammonium acetate–EDTA), OC, CEC_pot, pH, sand and clay (10) were modeled with relatively good discrimination capacity in the measured ranges (Fig. 3).

Model interpretation and filtering with variable importance

Figure 4 shows that the test RMSE of total C first slightly decreased and then steadily increased from all (209) to less spectral variables using CUBIST and RFE. The lowest error (RMSE_test = 8.10 g kg⁻¹ total C) of spectroscopic estimation was achieved with the spectra with 105 variables. For the subsequent variable reduction steps, model performance steadily dropped until one wavenumber was left (RMSE_test = 18.8 g kg⁻¹ total C).

Figure 4(a) Root mean square error (RMSE) of mid-IR estimates of total C that CUBIST produced at the respective subsets of spectral variables. The performance profile was obtained with a recursive feature elimination (RFE) procedure. The error bars represent the standard deviations of the test RMSE derived with nested cross-validation (n=50). (b) Average importance ranks across the spectrum. Lower rank values indicate higher importance for the estimation of total C. Ranks were determined with RFE. (c) Mid-IR absorbance spectra of the Swiss soil spectral library (n=4295; with corresponding total carbon (C) measurements determined by dry combustion). The unprocessed absorbance spectra are annotated with the 17 most influential spectral variables (wavenumbers) in the CUBIST model (average importance rank <15); these had the highest mean importance ranking determined by the recursive feature elimination procedure.

Download

The spectral feature between 1786 and 1754 cm⁻¹ was the most important one for the estimation of total C with CUBIST (Fig. 4). The 12 spectral variables with the best importance ranks across all RFE iterations and test sets derived from the subset sizes were (starting with the best) 1754 (mean(rank)=1.04), 1786, 1770, 2010, 2506, 1850, 1370, 2522, 1818, 1866, 2058 and 1386 cm⁻¹ (mean(rank)=12.7; Fig. 4).

3.3 Accuracy of the local transfer models compared to the general model

For the example site 65 COR, the best performance of RS-LOCAL was achieved with 55 samples from the SSL (K), 10 sampling events (B) of size K at each iteration and 10 % reduction (r) at each iteration (Fig. 5). Therefore, 55 transfer samples from the SSL were combined with two site calibration samples previously used to supervise the selection from the data source, to form a PLSR calibration model for the estimation of the site validation samples (see Fig. 5a; right). Compared to the target observations from the site (right part of Fig. 5a and b; measured range = 11.9–16.0 g kg⁻¹ C), the selected instances were heterogeneous with regard to their characteristic patterns in raw spectra, their preprocessed feature space and their measurements (range = 8.7–97.7 g kg⁻¹ C). The selected instances covered a significant proportion of the first two components in the feature space of the entire SSL.

Figure 5Illustration of the site-specific transfer modeling of total carbon (C) using RS-LOCAL for the example site 65 COR of the Swiss Soil Monitoring Network (NABO). Panel (a) contains the principal components subspace (PC1 and PC2) of the Savitzky–Golay first derivative mid-IR spectra, and panel (b) outlines the corresponding absorbance spectra (unprocessed for illustration), which are colored by the total C content. The left subplots show the SSL transfer samples (n=55) that were selected from the soil spectral library (n = 4281; excluding all NABO calibration samples). This subset was most accurate when predicting the two calibration samples under the mechanisms RS-LOCAL and their optimal tuning configuration for the site ($\mathit{\{}K=\mathrm{50};B=\mathrm{10};r=\mathrm{0.1}\mathit{\}}$). The right panels shows the time series data for the validation samples of the NABO site called 65 COR.

Download

The RMSE on the site validation samples (${\mathrm{RMSE}}_{N_{\mathrm{site}}}$) at the final subsets varied between 0.01 and 10.73 g kg⁻¹ C and for all tuning parameter combinations and sites and between 0.01 and 3.02 g kg⁻¹ C for the best subsets per site (Fig. A1).

Figure 6Model assessment of the estimated total carbon (C) of 71 NABO sites for the general learning with CUBIST (y axis) vs. local learning transfer with RS-LOCAL (x axis). The four panels depict the root mean square error (RMSE), the mean error (ME), the ratio of performance to interquartile distance (RPIQ) and R². The 1 : 1-line emphasizes the difference between the two approaches.

Download

The local approach reduced the error of the rule-based approach on average by factor 4.4 (Fig. 6; mean(RMSE_RS-LOCAL) = 0.7 g kg⁻¹ C; mean(RMSE_CUBIST) = 3.1 g kg⁻¹ C). The local transfer was more accurate for the majority of NABO sites (69 out of 71 sites). The linear dependency between modeled and measured values was higher for the local transfer compared to the general model (53 out of 71 sites). Moreover, RS-LOCAL produced on average 1.3 times less biased estimates of total C per site for 52 out of 69 sites in terms of absolute values ($\mid \mathrm{ME}\mid$ = 0.1 g kg⁻¹ C vs. 0.5 g kg⁻¹ C). The ratio of performance to interquartile distance (RPIQ) confirmed that local learning in the mid-IR was able to better discriminate developments of total C over time, relative to its measured distribution. Overall, local learning with two local calibration samples and targeted SSL selections allowed for better estimations than the generic CUBIST approach on average (RPIQ = 3.08 vs. 1.00; RPIQ larger for 66 out of 71 sites). Across all validation data points of the NABO set, the RS-LOCAL transfer was 5.6 times more accurate for total C than the general rules in terms of RMSE and RPIQ (RMSE = 0.9g kg⁻¹ C; RPIQ = 31.7)

Predictive mechanisms behind the local transfer

The samples used for the transfer process (RS-LOCAL data) of the example site COR 65 showed high spectral dissimilarity along the first 2 PCs (principal components), explaining 39.8 % of the preprocessed spectral variance (Fig. 5). Compared to the entire SSL with total C measurements available (the source domain prior selection; range of PC1 from −41.4 to 13.0; range of PC2 from −19.0 to 30.0), the selected transfer samples of this site occupied a region of major variation in the PC space (range of PC1 from −15.4 to 11.4; range of PC2 from −10.2 to 10.9). The two local calibration samples and the 12 validation samples in the upper right corner were close to each other in the PC1–PC2 subspace (Fig. 5a; left and right; range of PC1 from 9.2 to 11.0; range of PC2 from 4.9 to 7.5). Not only the absorbance spectra but also the corresponding C reference values were highly variable compared to the exemplary NABO site (Fig. 5b; 7.3–117.8 g kg⁻¹ C for K_RS-LOCAL and 11.9–16.0 g kg⁻¹ C for the plot of this site). This particular target monitoring site indicated that RS-LOCAL selected soils from the SSL with a relatively large spectral diversity and a wide range of total C.

Figure 7Analyzing the mechanisms behind the individual adaptive transfer realized with RS-LOCAL. (a) The left horizontal bars show the root mean square error (RMSE) of mid-IR predictions of the temporal validation set N_site of time series of total carbon (C) for each of the 71 NABO sites, which was calculated without the two respective calibration samples. The blue density plots depict the distribution of the site-specific validation samples, and the brown vertical bars show the measured values of C for the final subsets of SSL used for the transfer (K_site). (b) The distribution of the robust distances from the PCA center of the Savitzky–Golay preprocessed spectra of the entire soil spectral library compared to the subset of instances involved in the individual transfer modeling (K_site) and validation samples (N_site; similarity in site-specific vs. final RS-LOCAL selection) computed with the minimum covariance determinant (MCD) estimator.

Download

The instances selected by RS-LOCAL filled a substantial proportion of the SSL's feature space (Fig. 7), confirming the trend of site 65 COR. We found that RS-LOCAL yielded quite a wide selection of relevant samples from the SSL with reference to both the total C range and a wide coverage of spectral features expressed with robust multivariate locations. The spectral estimations of the site validation sets that resulted from RS-LOCAL-based transfers neither showed trends in the mode or spread for distributions of C measurements nor in the ones from their spectral distances. The measured distributions of K_site SSL subsets and N_site local validation samples for further key soil properties related to the chemical composition (OC, pH, CEC_pot, clay and CaCO₃) were also markedly different, confirming the local transfer of quite heterogeneous soils (Table 3). For example, standard deviations of the 0 %, 25 %, 50 % and 75 % percentile differences between the transfer sets selected the SSL and the samples from the respective NABO site were on average between 18 and 66 g kg⁻¹ for measured C and OC, respectively. Furthermore, the measured clay and CaCO₃ contents were markedly different between the RS-LOCAL selection and the local validation sets (mean absolute median differences of 85 g kg⁻¹ clay and 89 g kg⁻¹ CaCO₃). These findings correspond with the dissimilar selection compared to the local target samples found in the PCA space of preprocessed spectra.

Table 3Standard deviations (SDs) of the absolute differences in percentiles ($P_{\mathrm{0}},P_{\mathrm{25}},P_{\mathrm{50}},P_{\mathrm{75}},$ and P₁₀₀) of final RS-LOCAL subsets (K_site) and corresponding site validation samples (N_site) the across 71 long-term monitoring sites. The aggregated values for six measured soil properties are shown. Total C is total carbon, OC is organic carbon and CEC_pot is potential cation exchange capacity.

Download Print Version | Download XLSX

4.1 General soil estimation with the Swiss SSL

Many of the chemical properties with distinct links to soil organic matter and the key minerals (e.g., clays and quartz) were discriminated well with mid-IR CUBIST models (Table 2; Fig. 3). Specifically, the models estimated total C, OC, N, pH, texture, AAE10-Ca and AAE10-Mg with R²>0.7. This suggests that the majority of developed models are useful for applications that require soil proxies in order to manage land resources. For example, CEC_pot (RMSE = 7.0 $\mathrm{cmol}(+){\mathrm{kg}}^{-\mathrm{1}}$) and pH (RMSE = 0.3) have high ecological importance for nutrient availability in ecosystems. In agriculture, both measures are key factors for soil fertility and nutrient recommendations.

The accuracy of our estimates for the properties that have direct chemical links, through compound-associated absorptions, were mostly comparable to established continental or country-specific mid-IR SSLs. For example, Clairotte et al. (2016) achieved RMSE=2 g kg⁻¹ for OC using mid-IR and the spectrum-based learner for local predictions, while Sila et al. (2016) reported RMSE=4 g kg⁻¹. The accuracy of our general OC estimates was lower (RMSE=9.3 g kg⁻¹; RPIQ = 3.4), which we explain with the relatively large range of measured values and variable mineralogy (Stenberg and Rossel, 2010). We found that total C had more CUBIST rules per committee than OC (Table 2), indicating that total C, which also included inorganic C (mostly CaCO₃), leverages more chemical constituents and latent absorptions for its estimation. In spite of lower parsimony, slightly more accurate estimates of total C were achieved (RMSE=8.4 g kg⁻¹; RPIQ = 4.3).

The majority of soil properties were most accurately estimated with the maximum tested 100 committees and nine neighbors. Instance-based correction with similar data in the training set, (nearest) neighbors, yielded considerably higher accuracy for total C (e.g., RMSE = 8.9 g kg⁻¹ for 20 committees and two neighbors vs. RMSE = 8.1 g kg⁻¹ for 20 committees and nine neighbors; model evaluation across cross-validation folds; results not shown). The number of rules give a first proxy for model complexity and the complementary of spectral features that are involved in prediction. The range in number of rules across the ensembles was widest for total C (6–26), similar for OC (4–24), medium wide for CEC_pot (1–10) and very narrow for CaCO₃ (1–5), to give specific examples. Viscarra Rossel and Webster (2012) report comparably fewer rules (medium – 21; range of all properties – 5–64) for OC and relatively similar number of rules for CEC (15). Nonetheless, such comparisons have to be done with care because the NIR range has a less pronounced representation of functional groups than the mid-IR range and because temperate soils have fundamental differences in chemical composition compared to more weathered tropical soils. For our mid-IR SSL, we were surprised that the rules for OC were complex, similar to the ones for total C; in fact, we also could not find any clear partitioning in the rules with respect to measured ranges and spectral patterns (exploratory analysis not shown), which is in contrast to Viscarra Rossel and Webster (2012). In fact, this is different from the general patterns found by Viscarra Rossel and Webster (2012), where the rules clearly partitioned the data into distinct measured distributions. Last but not least, the diversity in rules for total C and OC of the general estimation approach makes the soil diversity selected from the library and what we found for site-specific local transfer even less exotic (see Sect. 4.2).

The variable importance assessment of the spectroscopic models revealed five major regions of features with particularly high predictive influence for total C, i.e., 2890, 2522, 2010, 1754 and 1370 cm⁻¹ (Fig. 4). We attribute the two absorption peaks near 2890 cm⁻¹ to C−H stretching vibrations of organic matter (Skjemstad and Dalal, 1987), which were also relatively important for estimating C in other studies (e.g., Janik and Skjemstad, 1995; Viscarra Rossel and McBratney, 1998). The important variable at 2522 cm⁻¹ is indicative of C=O absorption due to the carbonyl group present in carbonates (e.g., calcite; Nguyen et al., 1991; Soriano-Disla et al., 2014). The three important absorptions between 2010 and 1786 cm⁻¹ result from three consecutive $\mathrm{Si}-\mathrm{O}-\mathrm{Si}$ (overtone and combination) absorptions, which are indicative of quartz. However, the most important absorptions near 1754 cm⁻¹ showed no distinct peak but an edge feature. This is in accordance with Sila et al. (2016), who identified this region as being most relevant for estimating total C with a (general) random forest model developed from the SSL of the Africa Soil Information Service. This region is close to the C=O stretching vibration of the carboxyl group that occurs around 1725–1720 cm⁻¹ (Madari et al., 2006), which is further confirmed by the high importance of these vibrations found by Janik and Skjemstad (1995). The last relatively important region around 1370 cm⁻¹ was also an edge feature with no distinctly visible peak of chemical group assigned, which, however, might be influenced by the adjacent carboxylate (COO⁻) or −CH absorptions at 1400–1350cm⁻¹ of aliphatic compounds such as humic acids (Madari et al., 2006; Parikh et al., 2014). In summary, the CUBIST–RFE variable importance analysis enabled us to link characteristic absorptions of typically prominent functional groups of soil organic and inorganic C compounds, as well quartz absorptions as indirect correlative features of predictive relevance, with our general model-based estimates of total C.

Because the rule-based models we developed can estimate 10 soil properties reasonably well (R>0.6; RPIQ >2.0; Fig. 3), the Swiss SSL will be useful for new soils when new reference measurements for model adaptation are relatively scarce or not available. Thereby, the Swiss SSL will be cost and time efficient for characterizing soils of similar composition in the near future. The new predictions can further be augmented with straightforward model interpretation, which allows the chemical inference of pedological aspects to provide means of model applicability. Although the combined BDM and NABO set comprises a large soil variability in Switzerland, the diversity of subsoils at depths greater than 20 cm – mostly in terms of the mineral composition – and peat and forest soils are probably not yet represented sufficiently in the SSL. We must therefore continuously update the present SSL with more and deeper soil horizons in the near future.

4.2 Local transfer from the SSL for soil monitoring at plot scale

The local estimates of total C that were derived with RS-LOCAL selection were substantially better on average (RMSE = 0.7 g kg⁻¹ C) than those derived using all of the data and general CUBIST models (RMSE = 3.1 g kg⁻¹ C; Fig. 6). The data-driven estimation at plot scale further considerably reduced bias and increased R² compared to the general CUBIST rules.

Our third goal was to analyze the characteristics of soils that were selected from the SSL and used for establishing locally adaptive models tailored to the respective long-term monitoring sites. Surprisingly, the RS-LOCAL subsets selected from the SSL had rather dissimilar spectra in the robust PCA space (Figs. 5 and 7); their distances to the center had a wide distribution compared to the local samples. The K_site subsets accordingly covered a large proportion of the spectral input space. The likely dissimilar chemical composition of soils was also reflected in the reference measurements of total C. We conducted a broader analysis to interpret the soil context of the selected samples with further soil compositional covariates (OC, pH, CEC_pot, clay and CaCO₃), which also did not resemble the soil characteristics of the local monitoring sites (see Fig. 3). These findings, together with the accurate validation results, clearly indicate that dissimilarity and diversity in soils can also provide the means for fitting locally adaptive models.

Nevertheless, we can yet only speculate about how and why such diverse calibration sets are able to leverage accurate local calibrations. One hypothesis is that, by increasing the range of and variability in spectral variables and measurements, a model can become quite stable in the central range of local reference measurements because a larger range of input variables is considered; thereby, the RS-LOCAL subsets that are selected from the SSL and used for PLSR would stabilize and reduce the errors of the local samples. We imagine that we leverage a similar mechanism as in simple linear regression, where narrowing the range of the independent variable (x) in the training samples would decrease the accuracy of intermediate values of the independent variable. We therefore need to look further into the details of spectral dissimilar learning and, for example, also investigate the relevance of specific spectral features for local spectral transfers. The inherent working principles of RS-LOCAL are in contrast to the spectrum-based learner (SBL) or other forms of memory-based learning that utilize similar samples to infer sample-specific predictions based on existing training data (Lin and Vitter, 1994; Ramirez-Lopez et al., 2013). Our approach could describe a data-driven phenomenon, which implies that spectra can help to estimate a set of unrelated new soils. Another possibility is that there is in fact a pedological explanation that could be elucidated with more soil covariates, such as mineralogy.

Local soil characterization is simpler, quicker and cheaper when a large proportion of properties of new soils are estimated by spectroscopy. Our results suggest the importance of optimizing the transfer of relevant information present in large SSLs to minimize the required amount of conventional laboratory analyses of new soils. Soil chemical and physical heterogeneity can be substantial in large SSLs. Therefore, such data variation can be beneficial for future predictions of the properties of soils. However, the machine learning of a single general model over a heterogeneous training set, and obtaining parameter estimates optimized with a global measure of goodness of fit, can introduce bias and inaccuracy to local (soil) estimation (Hand and Vinciotti, 2003; Ramirez-Lopez et al., 2013; Lobsey et al., 2017). Although the highest estimation accuracy could be achieved only with soils of the target study area (Stenberg and Rossel, 2010; Guerrero et al., 2016), it is impractical and inefficient to derive a single spectral prediction model with those. It requires (1) a large volume of reference measurements for a reasonably accurate multivariate calibration, and (2) it does not utilize already existing soil information.

Currently, the Swiss long-term soil monitoring uses a spatially representative sampling and then bulks the soils into four replicates for reference measurements (Desaules et al., 2010; Gubler et al., 2019). When the long-term monitoring would be augmented with mid-IR spectroscopy, one could make spectral measurements on all subsamples, rather than only on bulked samples, which would deliver spatially explicit information and reduce nuisance factors from different sampling conditions. If not constrained economically (separate drying, sieving and milling of subsamples), a spectral workflow could thus allow one to account for small-scale soil variability and reduce bias in measurements to robustly estimate temporal soil changes. For example, there is currently a relatively large variability in C measurements between the bulked replicate samples at one point in time (Gubler et al., 2019). Our results suggest that unbiased spectral measurements eventually mediate such inconsistencies. Although Gubler et al. (2019) reported only minor changes for the ensemble of permanent cropland or cropland–meadow monitoring sites (30), there were four sites with declining trends and nine sites with increasing trends in OC (−11 % to +16 % relative change per decade, respectively). Here, the trend of spectroscopic predictions could be investigated with respect to specific research questions on agronomic management-induced changes, also with further physicochemical soil characterization (e.g., OC fractions).

Relatively precise and unbiased geographically local estimates of soil properties from diverse and large SSLs can be achieved by a handful of data-driven statistical approaches that are currently popular in the soil science community (Viscarra Rossel and Webster, 2012; Ramirez-Lopez et al., 2013; Guerrero et al., 2014; Lobsey et al., 2017; Tsakiridis et al., 2020). Among the methods, we tested RS-LOCAL Lobsey et al. (2017) in our local soil monitoring scenario. Compared to memory-based learning, such as SBL (Ramirez-Lopez et al., 2013), RS-LOCAL does not precondition the choice of useful subsets based on similarity in the input dimensions, here spectra, when performing the selection of SSL samples. The RS-LOCAL method is applied to exhaustively sample instances from the SSL without replacement, while it preferably selects those that perform well on the local target set, using PLSR. An advantage of the method is that it can deal better with erroneous spectra, and inaccurate and imprecise analytical reference measurements in the SSL, because it filters them as irrelevant instances. Besides chemometric and classical machine learning approaches, convolutional neural networks are being popularized for modeling SSLs with large soil variability (e.g., Liu et al., 2018; Padarian et al., 2019a, b; Tsakiridis et al., 2020). There seems to be a small performance gain of a multi-output CNN with a similarity-based error correction using neighbors compared to the SBL (Tsakiridis et al., 2020; RMSE = 11 g kg⁻¹ vs. 12 g kg⁻¹ for OC). Despite the current development of interpretation methods in deep learning, CUBIST and PLSR modeling employed in both in the SBL and RS-LOCAL offer easier interpretation with comparable accuracy to CNNs.

Transfer learning or local learning introduces a new paradigm to supervised learning, i.e., model building that is governed by the intended model application and thus coupled to it (Hand and Vinciotti, 2003). This contrasts with the general model application, where the inference process is separated from the prediction of new data. Including local samples and their local data characteristics is necessary so that a combined search and learning algorithm has a chance to capture predictive mechanisms. At the same time, the selection process and the partial data dependence within the predictive unit, the site, requires a careful assessment scheme to prevent a potential selection bias in the assessment of the approach. To account for this, we kept the respective site-specific local tuning and calibration set – whose holdout performance directed the iterative search process and the reduction of the SSL – at minimum size of two observations at t₀ or, in addition, t₁ when only one measurement was available from the first sampling (see Fig. 2).

4.3 Future applications and updates of the SSL

We found that data-driven modeling with a selection of spectrally dissimilar soils (see Fig. 7) is accurate for inducing local predictions of total C (Fig. 6). Hence, there is the need to further improve the data-driven selection using RS-LOCAL, i.e., by further optimizing the current version of the algorithm. To address this need, we could use combined memory-based or lazy learning strategies (Stanfill and Waltz, 1986; Lin and Vitter, 1994; Ramirez-Lopez et al., 2013) to optimize with more data-driven transfer methods (Pan and Yang, 2010) in terms of reducing the time needed to evaluate suitable subsets of the SSL for a new application. To give an example, some similarity criteria or clustering before doing calibration sampling could be used as prior information for reducing the SSL size to obtain the final subsets. In principle, the sample reduction could also be done with algorithms that can deal with nonlinear relationships between spectra and soil properties, such as random forest or CUBIST. Another extension is to further filter spectral features and to do data compression to make the local modeling faster and even more adaptive to local conditions.

Our results showed that a transfer of the SSL to individual monitoring sites yielded very low bias and was accurate. This indicates that mid-IR spectroscopy and SSLs have the potential to give quick and relatively precise soil property estimates for soil monitoring. Nevertheless, the sites of the NABO long-term monitoring program has not undergone substantial changes in OC (Gubler et al., 2019). Up to now, although major changes in C content and organic composition should yield a spectral response, spectral changes in OC have mostly been reported along chronosequences (i.e., Awiti et al., 2008) and only rarely for changes within individual plots over time (Deng et al., 2013). Hence, to address this, we propose to further investigate to what extent mid-IR spectroscopy can detect changes of OC, considering small-scale variability and different agronomic management practices. This could, for example, be achieved with a study using soils from a long-term field trial that shows sufficient temporal changes to be detected with spectroscopy.

The current SSL includes soils that contain between 0 and 583 g kg⁻¹ total C and OC (Table 1). Because organic soils can have up to 500 g kg⁻¹ OC, and because more than 98 % of the samples are mineral soils, organic soils are underrepresented in the current Swiss SSL. For this reason, Helfenstein et al. (2021) evaluated the present Swiss SSL for a regional transfer based on new organo–mineral soils from two peatland regions in Switzerland. Although the range of total C measured was large (14–520 g kg⁻¹ C) and the soils were diverse, as few as 5 or 10 site-specific tuning samples were sufficient to estimate the validation samples with reasonable accuracy (RMSE = <30 g kg⁻¹ C; RPIQ >3.4); this was comparable to a local-only calibration with 50 samples. Helfenstein et al. (2021) found considerably lower conditional prediction errors (<10 g kg⁻¹) when considering measurements of <100 g kg⁻¹; this suggests that increasing the amount and compositional complexity of organic soils in the library has potential for more accurately characterizing diverse soil ecoregions with soils that have high organic matter contents.

Our results suggest that the present mid-IR SSL has great potential for applications that require soil data in high temporal and spatial coverage (i.e., for deriving quantitative indicators of soil quality for spatial planning or for soil-related environmental research). Mid-IR spectral modeling was able to estimate many soil properties accurately with a rather large variation in measurements being explained (Fig. 3), making them suitable for agronomic diagnosis and the assessment of soil functions in various landscapes. Currently, fine-grained soil information on the properties and functions across agricultural lands in Switzerland is still scarce and often challenging to harmonize (i.e., measurement methods) because legacy maps are at varying levels of detail and quality (Keller et al., 2018; Grêt-Regamey et al., 2018). For example, only 13 % (127 000 ha) of soil in agricultural land has been mapped with soil attributes of sufficient quality to evaluate its potential for crop production (Rehbein et al., 2020). Soil properties are also insufficiently mapped nationwide from points into space, depth and over time to regionally model soil processes or to evaluate site-specific effects of agricultural practices on soils (i.e., soil C dynamics). Therefore, we suggest coupling infrared spectral estimations with traditional soil surveys and digital soil mapping to speed up the collection of soil information in Switzerland and elsewhere. This will offer the means to test and further extend this SSL so that only minimal amounts of costly and time-consuming traditional laboratory analyses will be needed for characterizing and mapping soils' properties and functions in the next decades.