Fiducial Reference Measurements for validation of Sentinel-2 and Proba-V surface reflectance products

https://doi.org/10.1016/j.rse.2020.111690Get rights and content

Highlights

  • Full diffuser angular and spectral characterisation and traceable calibration

  • Development of simplified atmospheric correction uncertainty using Monte Carlo

  • In situ uncertainty based on instrument, sampling and procedural contributions

  • Consideration of satellite and in situ measurement quantity/condition differences

Abstract

Many derived Earth Observation products share surface reflectance as a common step in their processing chains. This makes the maintenance and improvement of surface reflectance product quality of fundamental importance to ensure information derived from these downstream products can be trusted. Despite this, the literature is relatively light on the implementation of validation methodologies designed for surface reflectance. In response to the need to improve general EO validation methodologies, the concept of fiducial reference measurements (FRM) was created that would produce validation data that is fully characterised and independent, with associated uncertainties, and traceable to SI.

This paper describes a field campaign designed to produce surface reflectance validation data, in line with the FRM ideology, for validating the Sentinel-2 L2A (S2) and Proba-V S1-TOC (Proba) surface reflectance products. The key methodological procedures are outlined in detail to facilitate uptake of FRM methods in future validation studies. These include the calibration and characterisation of field instruments, the field measurement protocol, and uncertainty propagation.

Comparison of S2 with field measurements showed agreement within the stated uncertainties present for all bands and locations at the pixel and area scales. The Proba comparison results demonstrated a general disagreement within the stated uncertainties. However, there is a lack of uncertainty information provided by the product as well as publicly available product uncertainty requirements which mean that it is difficult to assess this fully. The aim of this practical demonstration of FRM-based surface reflectance validation, that utilises metrological practices for uncertainty characterisation, is to encourage the adoption of these procedures in future land surface reflectance validation studies and operational activities.

Introduction

In the terrestrial domain, atmospherically corrected surface reflectance products are the last in a series of processing steps that starts with at-sensor digital counts. These products then form the starting point for numerous application areas such as land cover mapping and the derivation of biophysical essential climate variables (ECV). Therefore, ensuring the quality of surface reflectance products is beneficial to maintaining the integrity of the research coming out of these application areas.

Validation provides a key route to ensuring that these products are performing to their specifications. In this context, an independent data source is used as a reference to which the satellite product is assessed. The prevalence of validation studies and operational validation activities is relatively widespread within the literature for a variety of Earth observation (EO) products (e.g. Liang et al., 2002a, Liu et al., 2009, Guillevic et al., 2012, Camacho et al., 2013) since this provides a means for users to assess the utility of the product for their application. In recognition of the importance of this work, the Committee for Earth Observation Satellites' (CEOS) Land Product Validation (LPV) subgroup was established to coordinate international validation activities across different space agencies and research institutions.

Review of the surface reflectance product validation literature reveals a broad trend towards the assessment of such products against long-term in situ monitoring networks such as the Surface Radiation Budget Network (SURFRAD) (Jin et al., 2003, Salomon et al., 2006, Liu et al., 2009), Aerosol Robotic Network (AERONET) (Vermote et al., 2016) and the AERONET-based Surface Reflectance Validation Network (ASRVN) (Wang et al., 2009, Wang et al., 2010, Sogacheva et al., 2015). These networks are made up of remote ground stations transmitting radiation fluxes (SURFRAD) and multispectral aerosol optical depth (AOD) estimates (AERONET). The ASRVN system provides atmospherically-corrected MODIS surface reflectance for the 50 km2 surrounding the ground station (Wang et al., 2009). The general premise behind this approach is to compare the surface reflectance derived from atmospheric correction using the image derived AOD against the equivalent (or MODIS as in the ASRVN case) using the station derived AOD. This assumes that the surface reflectance derived from the station AOD is the “truth” (Vermote et al., 2016). SURFRAD, on the other hand, derives surface reflectance (in this case albedo) directly from the up and downwelling fluxes measured at the stations. These can then be compared against the product derived surface reflectance.

However, some common themes begin to emerge when analysing the studies using these networks. Firstly, the combined impact of the sensor spatial resolution (MODIS at 500 m in most cases) and the site heterogeneity can cause significant disagreement. These tend to be mitigated in one of two ways: stricter station inclusion requirements (Román et al., 2009, Cescatti et al., 2012) or by using a transfer product with a greater spatial resolution (Liang et al., 2002a, Fan et al., 2014). For example, Jin et al. (2003) found that the increased heterogeneity brought by snow pushed the validation results outside of the product accuracy requirements for the winter months, an effect that was also seen in other studies (Salomon et al., 2006) and attributed this as one of the major issues affecting the validation (Liu et al., 2009). However, with sparse validation datasets often there is little opportunity to be selective. This leads to an alternative approach which attempts to overcome this issue by accounting for that heterogeneity with a finer resolution transfer product. While this is appealing, matching of the product geometries is required (and often not available) and correlation is introduced between the reference and test datasets (i.e. similar processing, atmospheric assumptions, etc.). Furthermore, the transfer product adds its own uncertainty. These three factors make this type of processing challenging and assessment of the results ambiguous.

Secondly, there is little mention or quantitative assessment of the uncertainties associated with the in situ data nor any endeavour to perform the validation in a metrologically robust way. The best attempt appears to be the analysis described in Vermote and Kotchenova (2008) and later used in Vermote et al. (2014) where the authors use the in situ data as a representation of the truth and describe the validation uncertainty as a linear combination of the precision and accuracy results. However, this is a long way off meeting the needs of future applications of satellite-derived data and requires consideration of the reference data uncertainty (Widlowski, 2015). Likewise, reduced ambiguity in the validation results is required.

Thirdly, many of the published activities affecting many sensors (for example: Sentinel-2: Gascon et al. (2017); MODIS: Vermote et al. (2016); and AATSR: Sogacheva et al. (2015)) utilise reference data which cannot be considered independent (e.g. those utilising other satellite products). This is because the main assumption is that the aerosol correction is the dominant error. This may be the case, but by only comparing the impact of the aerosol retrieval, the impact of other errors in the processing chain (radiometric, geolocation, etc.) is lost and the validation is only testing the aerosol retrieval. Validation of satellite surface reflectance data against independent in situ reflectance estimates appears to be more common for assessing the quality of new algorithm developments (e.g. Liang et al., 2002b, Li et al., 2010) rather than operational products.

Given the increasing role that quantitative EO derived products assume in climate and environmental monitoring applications, the quality of this data is coming under increasing levels of scrutiny (Widlowski, 2015, Nightingale et al., 2018, Nightingale et al., 2019). Therefore, detailed assessment of EO data product quality that includes uncertainty characterisation of the retrieval algorithm as well as the in situ data and methods used to validate the algorithm, is essential. Further, provision of the end-to-end quality assessment in a standardised and comprehensible manner is required to help potential data users navigate and understand the nuances between the wealth of similar satellite derived products available to them.

This is particularly important for operational activities, such as the European Union's Copernicus Climate Change Service (C3S) whose mission is to provide authoritative information about the past, present and future climate in Europe and the rest of the world. In a progressive commitment to ensure that all datasets available through the C3S Climate Data Store are traceable, adequately documented and accompanied by quality information so that data users can make informed decisions for their application, C3S has made significant investments in the ongoing development of Evaluation and Quality Control (EQC) functionality (Nightingale et al., 2019). Similarly, the European Space Agency (ESA), in attempting to address some of the issues mentioned above, have created a program of work to address the concept of Fiducial Reference Measurements (FRM). These aim to provide fully characterised and independent in situ data with uncertainties where the measurements are traceable to the International System of Units (SI - from Système international (d’unitès)), or community-agreed standards, to underpin the validation of satellite products. The production of these datasets should follow the Quality Assurance for Earth Observation (QA4EO) guidelines; namely that data provenance is ensured and that uncertainty assessment follows the Guide to the Expression of Uncertainty in Measurement (GUM) approach (JCGM, 2008).

With this in mind, the present paper describes a field campaign designed to produce an FRM dataset for the validation of the Sentinel-2 and Proba-V surface reflectance products, with the aim of developing the concept of an FRM validation. We consider the validation problem as containing three distinct components: the reference sensor, the test sensor, and the measurement conditions. The first two of these are generally considered obvious (and in the context of this study refer to the in situ and satellite sensors respectively), but attention to the latter is rarely formalised in a remote sensing context (i.e. by inclusion into the uncertainty budget). Here we attempt to account for each in turn and formalise the assumptions made throughout, ensuring that the traceability, provenance and uncertainty are at the forefront of the analysis. It is noted that our usage of the terms “uncertainty”, “error” and other related terms in this paper is guided by the GUM (JCGM, 2008) and defined by the international vocabulary of metrology (VIM).

Section snippets

FRM campaign design

Formulating a validation campaign that adheres to FRM principles involves considering the factors influencing the comparison. Table 1 provides a non-exhaustive list of subcategories under each of the distinct validation components.

The goal of collating this information is to determine: which measurements need to be made in the field; the calibration and characterisation experiments that need to take place; the choice of field protocol; and timing of the field data acquisition. Consideration of

Sentinel-2

The high spatial resolution of the S2A instrument facilitates the ability to compare individual pixel values against in situ measurements from a single sample location. Comparison of each location against the corresponding S2A pixel values as well as the site averages compared to the 200 × 200 m S2A average is presented in Fig. 8, Fig. 9. Fig. 8 gives the comparison over the Alfalfa field. Here, all bands and locations show good agreement within the stated uncertainty (k = 2) with respect to

Summary and conclusions

The present paper has demonstrated the fundamental steps required to perform an FRM validation of Sentinel-2 and Proba-V surface reflectance products. Firstly, we use a metrological approach that informs the calibration and characterisation of the reference panel based on the illumination conditions experienced in the field. Secondly, the products are assessed at the pixel level, from which the product requirements are made, for a high and high/medium resolution satellite instrument; as well as

CRediT authorship contribution statement

Niall Origo: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Data curation, Writing - original draft, Writing - review & editing, Visualization, Supervision. Javier Gorroño: Conceptualization, Methodology, Software, Validation, Formal analysis, Data curation, Writing - original draft, Writing - review & editing, Visualization, Supervision. James Ryder: Conceptualization, Methodology, Investigation. Joanne Nightingale: Conceptualization, Writing - review &

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The authors would like to thank Fernando Camacho de Coca, José González Piqueras and Alonso Garrido Lamiñana for their support in arranging the fieldwork; Paul Green for his support in conducting the fieldwork; and, Jadu Dash and Luke Brown for supplying some of the equipment used in the fieldwork. The research leading to these results was partially funded by the EU and European Space Agency through the FRM4Veg project and the National Measurement System of the UK government's Department for

References (43)

  • WilsonR.T.

    Py6S: a Python interface to the 6S radiative transfer model

    Comput. Geosci.

    (2013)
  • AEMet

    Radiation and Ozone

  • BialekA. et al.

    Namibia Field Campaign Technical Note

  • ChunnilallC.J. et al.

    NPL scales for radiance factor and total diffuse reflectance

    Metrologia

    (2003)
  • ClercS. et al.

    S2 MPC. Level 2A Data Quality Report

  • EhleringerJ. et al.

    Solar tracking by plants

    Science

    (1980)
  • FanL. et al.

    Validating MODIS land surface reflectance products using ground-measured reflectance spectra a case study in semi-arid grassland in Inner Mongolia, China

    Int. J. Remote Sens.

    (2014)
  • GasconF. et al.

    Copernicus Sentinel-2A calibration and products validation status

    Remote Sens.

    (2017)
  • GorroñoJ. et al.

    A second version of the radiometric uncertainty tool for the sentinel-2 mission

  • JCGM

    Evaluation of measurement data - guide to the expression of uncertainty in measurement

  • JCGM

    Internationl vocabulary of metrology - basic and general concepts and associated terms (VIM)

  • Cited by (0)

    View full text