2.1 The AVATAR-T campaign

Delayed by more than 1 year due to the COVID-19 pandemic, the AVATAR-T campaign was carried out from 6 September to 28 September 2021 on the island of Sal, Cabo Verde. It further extended the existing datasets of A2D and 2 µm DWL observations under different atmospheric conditions, especially under the influence of the Saharan Air Layer (SAL) dust-laden air masses and tropical wind systems offshore of West Africa. AVATAR-T was DLR's contribution to the international JATAC project, supported by ESA, which combined several airborne participants, including the French SAFIRE Falcon 20 aircraft and the NASA DC-8 (based on the U.S. Virgin Islands), with a number of ground-based instruments located in Mindelo on the island of São Vicente. In the framework of AVATAR-T, a total of 11 research flights were dedicated to satellite underpasses, of which six flights were performed along the ascending orbit in the evening hours. Adding up the lengths of the Aeolus measurement swaths covered by the DLR Falcon aircraft during the underflights, the overall track length for which wind data are available for validation purposes is close to 11 000 km.

Table 1Overview of the Aeolus underflights of the Falcon aircraft in the frame of the AVATAR-T campaign in September 2021 and the wind observations performed with the 2 µm DWL along the Aeolus measurement track.

Download Print Version | Download XLSX

An overview of the 11 underpasses is provided in Table 1, including the geolocations of the start and end points of the respective sampled segment of the Aeolus path and the number of Aeolus and 2 µm DWL observations. While the number of 2 µm DWL observations corresponds to the number of wind profiles with a horizontal averaging length of about 8.8 km (one scanner revolution), one Aeolus observation is spread over a nominal horizontal averaging length of about 87 km. The latter is also referred to as basic repeat cycle (BRC) and can, in principle, contain multiple wind profiles, especially for the Mie channel, where the signals are typically averaged over 10 to 15 km, as a consequence of the grouping algorithm in the L2B processor (Rennie et al., 2020). Over the course of the campaign, the optical alignment of the 2 µm DWL was progressively degraded by the large temperature and humidity fluctuations in the aircraft during and between the flights, resulting in a significant reduction in the data coverage and, ultimately, operational failure toward the end of the campaign. Nevertheless, high-quality wind measurements were obtained from the first five underflights.

The colocated wind data served to assess the quality of the Aeolus wind product under the tropical atmospheric conditions in autumn 2021 with the goal to optimize the operational settings and to refine the retrieval algorithms for the operational phase of Aeolus and beyond. At that stage of the mission, the atmospheric path signal level, as measured with the Rayleigh channel, had decreased by around 50 % with respect to the signal levels detected during the AVATAR-I campaign in September 2019. The degradation in SNR was, however, partly mitigated by lower solar background levels in the tropics compared to the North Atlantic region, which also markedly influences the Rayleigh wind error (Rennie et al., 2021). The same is true for the Rayleigh-clear EE, as will be discussed in the following section.

2.2 The L2B EE and its temporal evolution

The horizontal line-of-sight (HLOS) wind error estimate is included in the Aeolus L2B product and calculated differently for the Rayleigh and Mie channels given the different measurement techniques for deriving the wind from the Doppler frequency shift. The Rayleigh channel relies on the double-edge technique (Chanin et al., 1989; Garnier and Chanin, 1992; Flesia and Korb, 1999; Gentry et al., 2000), where the shift in the broadband Rayleigh backscatter spectrum relative to the emitted spectrum is determined from the signal intensities that are transmitted through two bandpass filters (A and B). The filters are realized by sequential Fabry–Pérot interferometers (FPIs), with adequate spectral width and spacing with respect to each other, for obtaining high spectral sensitivity to small frequency shifts of a few megahertz. The temporal evolution of the filter parameters over the mission also allows insights into the instrument's alignment (Witschas et al., 2022b).

According to the L2B Algorithm Theoretical Basis Document (ATBD; Rennie et al., 2020), the EE for the Rayleigh HLOS wind speed is computed from the uncertainty in the Rayleigh spectrometer response R, which is defined as the contrast between the transmitted signals A and B, as follows:

The total signal that is incident on the Rayleigh detector (A+B), after being corrected for the solar background and the detector noise, is also referred to as the useful signal. The error in the Rayleigh response σ_R is related to the respective SNR values of filters A and B, which are provided in the L1B product and account for the Poisson noise of both the signal and the solar background. The latter is subtracted to obtain the atmospheric signal levels only. The response error is then given as follows (Rennie et al., 2020):

with

being the noise terms for the two FPI filters. The response error is converted to the Rayleigh HLOS wind speed EE σ_HLOS, R by considering the sensitivity of the Rayleigh spectrometer $\partial v_{\mathrm{LOS},\mathrm{R}}/\partial R$ and the projection angle with the horizontal axis θ, i.e. the off-nadir angle of the instrument (θ≈37^∘), as follows:

Assuming, without the loss of generality, that A=B, it can be followed from Eqs. (2) and (3) that

Hence, the Rayleigh EE scales with the reciprocal square root of the useful signal on the detector. Before launch, it was intended to include additional parameters in the HLOS wind EE computation for the Rayleigh channel to account for the (local) sensitivities of the derived LOS wind speed to atmospheric temperature and pressure, which influence the Rayleigh response (Dabas et al., 2008), and to the scattering ratio. However, these additional terms, which are formulated in the ATBD, are not considered in the current algorithm (up to baseline 14) due to the lack of dynamic values as an input for the respective error contributions. Furthermore, the noise that is introduced by the detector and read-out electronics is not accounted for in the EE calculation, although its contribution is not negligible in the case of low atmospheric signal levels, i.e. later in the mission. Consequently, the Rayleigh-clear EE is governed by the Poisson noise of the measured signal and the solar background. Note that it is foreseen that additional noise terms for the EE calculation will be included in future processor versions and, hence, reprocessed datasets.

The wind speed determination in the Mie channel is based on the fringe-imaging technique (McKay, 2002), where a linear interference pattern (fringe) is produced by a Fizeau interferometer and vertically imaged onto a detector. The Doppler frequency shift then manifests as a spatial displacement of the fringe peak position which changes approximately linearly with the frequency of the incident light. The peak position, which is referred to as the Mie response, is currently calculated by a Nelder–Mead downhill simplex algorithm (Nelder and Mead, 1965) to optimize a Lorentzian line shape fit of the signal distribution across the Mie channel detector (Mie Core algorithm; Reitebuch et al., 2018). The Mie channel EE is then related to the precision of the Mie response and is approximated from the solution error covariance of the fit algorithm, as described in the ATBD (Rennie et al., 2020). The covariance matrix is calculated from the partial derivatives of the Lorentzian line shape function with respect to four different fit parameters (peak position, peak height, peak width, and peak offset). An additional correction factor for each detector pixel accounts for the obscuration of the telescope primary mirror by the tripod that bears the secondary mirror (Tan et al., 2008). The error in the Mie response σ_M, i.e. the peak position error, is one element of the covariance matrix and can be converted to the Mie HLOS wind speed EE σ_HLOS, M as follows:

where λ is the laser wavelength, and m describes the Mie response slope, which is determined from a dedicated instrument calibration mode. Thus, the Mie EE is not directly linked to the atmospheric signal levels but rather depends on the signal distribution across the Mie detector and how well the peak from the imaged fringe resembles a Lorentzian line shape. The Mie signal distribution is, however, influenced by the broadband Rayleigh signal that is incident on the Fizeau interferometer and modifies the signal distribution, depending on its strength relative to the Mie peak and the illumination conditions. Moreover, since the fringe is coarsely resolved by only 16 pixels on the Mie detector (European Space Agency, 2008), the fit precision for the determination of its centroid position varies slightly, depending on how the fringe intensity is distributed among adjacent pixels (pixelation effect). The definitions of the L2B EE for the Rayleigh-clear and Mie-cloudy winds suggest that the values in the L2B product do not fully account for all error sources that affect the wind measurement in the two channels. It is also implied that the EE has been varying over the mission lifetime, as the atmospheric signal levels have been declining.

Figure 1Temporal evolution of the reciprocal square root of the of the atmospheric signal level ($\mathrm{1}/\sqrt{\mathrm{ATM}}$) (red), normalized to the value from 26 July 2019, and the L2B Rayleigh-clear (green) and Mie-cloudy EE (blue) during the FM-B period (median per day). The baselines of the L2B product from which the data were extracted are indicated by the horizontal bars on the top of the plot. The lower panels depict the distribution of the Mie (b) and Rayleigh EE (c) for the periods and locations of the two DLR validation campaigns AVATAR-I (blue) and AVATAR-T (orange). The campaign periods are indicated by blue- and orange-shaded stripes in panel (a).

Download

The temporal evolutions of the Rayleigh and Mie EE over the FM-B period are depicted in Fig. 1a. The green and blue curves represent the daily median of the global wind data extracted from the L2B product (baselines 11 to 14). The instrument was switched off for several weeks in March/April 2021 and October 2021, following two so-called Failure Detection Isolation And Recovery (FDIR) events, resulting in data gaps. The red line describes the long-term trend of the reciprocal square root of the atmospheric useful signal ($\mathrm{1}/\sqrt{\mathrm{ATM}}$), as measured with the Rayleigh channel under clear-sky conditions at about 10 km altitude, and normalized to the beginning of the FM-B period. The signal levels dropped by about 70 % between July 2019 and July 2022 ($\mathrm{1}/\sqrt{\mathrm{ATM}}$ increased from 1.0 to $\mathrm{1}/\sqrt{\mathrm{0.3}}\approx \mathrm{1.8}$), while intermittent signal increases can be attributed to instances in which the laser energy was boosted by special operations, e.g. in December 2020 and November 2021. The Rayleigh EE evolution roughly follows the trend of $\mathrm{1}/\sqrt{\mathrm{ATM}}$ in accordance with Eq. (4), which assumes Poisson noise to be the dominant noise source (Reitebuch et al., 2018). The seasonal modulation of the Rayleigh EE is caused by the change in solar background noise, which adds to the shot noise of the signal and which is largest in the Northern Hemisphere summer, with secondary maxima in the Southern Hemisphere summer. The evolution of the Mie-cloudy EE is less affected by the signal trend and rather driven by the data processing algorithms and related settings which were modified several times over the FM-B period. Most notably, it was drastically reduced from around 4 to 3 m s⁻¹ with the release of processor baseline 12 in May 2021. The latter involved a change in the L1B processor to ensure the correct summation of the raw Mie signals after dark current subtraction, without setting negative values to zero, leading to an improved fringe centroid computation and, in turn, a smaller random error and EE of the Mie winds. In early May 2022, the threshold settings of the Mie Core algorithm were relaxed to obtain more valid Mie winds of reasonable quality at the expense of more outliers with high EE, hence increasing the median to around 5 m s⁻¹.

The variability in the EE also becomes obvious when comparing its distribution for two datasets from different periods and geographical locations. Figure 1b shows the histograms of the Mie EE that was extracted from the L2B product for the 10 Aeolus underflights of the AVATAR-I campaign in the North Atlantic in September 2019 (blue bars) and for the 11 underflights of the AVATAR-T campaign in the tropics in September 2021 (orange bars). For the latter dataset, the portion of Mie winds with EE ranging from 3 to 6 m s⁻¹ is significantly larger than for the AVATAR-I dataset, which, however, contains more Mie winds with EE smaller than 3 m s⁻¹ and larger than 6 m s⁻¹. Regarding the Rayleigh channel (Fig. 1c), the EE distribution has shifted towards higher values in accordance with the overall trend shown in Fig. 1a, although the impact of the atmospheric signal decrease between the two campaigns on the EE was partially compensated by the smaller detrimental influence of the solar background in the tropics compared to the higher latitudes and the fact that the range bin thickness was set to be larger (750 compared to 500 m during AVATAR-I).

Given the change in the EE distributions over the FM-B period, the rejection of wind results by using a (fixed) EE threshold as QC would lead to different portions of winds from the respective dataset to be validated and, hence, misleading results when comparing and interpreting the statistics of error distributions for the two campaigns. Therefore, it is crucial to consider the influence of different EE thresholds on the validation results and to find alternative schemes for the QC of the Aeolus data.

3.1 Systematic and random wind error

The statistical results from the model comparison without applying any QC are depicted in Fig. 2 for the Mie-cloudy (Fig. 2a, c) and the Rayleigh-clear wind results (Fig. 2b, d). The scatters in the top plots are colour-coded according to the respective EE value that is assigned to each wind result. Overall, the L2B wind results show good agreement with the model background winds from the ECMWF; however, there are quite a few wind results with large departures from the model, particularly for the Mie channel. Those winds are very often, but not always, associated with a large EE. The bottom plots represent the histograms, or probability density functions (PDFs), of the respective (O-B) wind errors, i.e. L2B minus ECMWF model. The histograms are nearly Gaussian distributed, when excluding the large outliers that are indicated by the red bars and whose identification will be discussed below. The main output of the analysis is the three statistical parameters that are listed in the inset boxes. The systematic wind error, or mean bias, with respect to the ECMWF model background is calculated as follows:

It represents the mean of the wind speed differences between Aeolus and the model, with n being the number of comparable winds after the exclusion of outliers.

The random error is given in terms of the standard deviation, as follows:

Additionally, the scaled median absolute deviation (scaled MAD) is derived as follows:

The scaled MAD is more resilient to outliers in the dataset and thus a more robust measure of the wind error variability than the standard deviation. In case the analysed data are normally distributed, the standard deviation and scaled MAD are identical.

The definitions given here are in line with the those stated in the Aeolus MRD, which assumes that the wind error with respect to other wind observations or the model background can be described by a Gaussian distribution, whose centre and width represent the accuracy and precision of the Aeolus winds. This is justified by the fact that the wind error is dominated by Poisson-distributed photon noise on the detectors, particularly for the Rayleigh channel. For the Mie channel, deficiencies in the signal analysis (Mie Core algorithm) give rise to gross errors which additionally contribute to the error distribution, as will be pointed out in Sect. 3.3. Nevertheless, after sorting out the gross errors, the assumption of a Gaussian distribution is also valid for the Mie channel.

3.2 Modified Z score for outlier detection

Data can be screened for outliers by calculating the modified Z score¹ for each wind observation. It describes its distance from the median, measured in units of the scaled MAD, as follows (Iglewicz and Hoaglin, 1993):

with k denoting the scaled MAD, as introduced in Eq. (7). Since the median and the scaled MAD are more robust to outliers compared to the mean and standard deviation, the modified Z score is a valuable tool for data screening, even if the distribution is far from Gaussian and/or the dataset is small. Iglewicz and Hoaglin (1993) recommended that modified Z scores with an absolute value greater than 3.5 are to be labelled as outliers. However, thresholds of 3 or even smaller are also used in literature, depending on the convention for defining outliers (Tripathy, 2013; Sandbhor and Chaphalkar, 2019).

Table 2Statistical comparison of the Aeolus L2B Mie-cloudy and Aeolus L2B Rayleigh-clear winds against the ECMWF model background winds for all underflights performed during the AVATAR-T campaign. The corresponding scatterplots and histograms are shown in Figs. 2 and 3, respectively. The statistics are derived after the adaptation of the ECMWF model data to the respective L2B measurement grids. The statistical parameters are given without and with the removal of outliers according to the modified Z score (|Z_m|>3.5).

Download Print Version | Download XLSX

Following the recommendation from Iglewicz and Hoaglin (1993), the (O-B) wind errors with respect to the ECMWF model background were filtered based on the modified Z score, i.e. wind results for which |Z_m| is greater than 3.5 were removed from the dataset. For the Mie-cloudy winds, 10.5 % outliers (68 out of 646 wind results) were identified in this manner, while the portion is much smaller for the Rayleigh-clear winds (89 outliers out of 2778 wind results; ≈3.2 %). The outlier removal has a significant impact on the derived statistical parameters (see Table 2), especially for the Mie channel. Since most of the Mie wind outliers are positively biased, the systematic error is as high as 8.3 m s⁻¹ if no QC is applied, but it is reduced to 0.3 m s⁻¹ by the outlier rejection based on the modified Z score. At the same time, the standard deviation is drastically decreased from 29.2 to 5.3 m s⁻¹. The impact on the scaled MAD is much smaller (5.1 to 4.3 m s⁻¹) because it is less prone to outliers, as described above. Regarding the Rayleigh statistics, the biggest effect of the outlier removal is observed for the standard deviation, which is reduced from 15.1 to 7.8 m s⁻¹, while the mean bias (0.4 to 0.1 m s⁻¹) and the scaled MAD (7.2 to 6.9 m s⁻¹) are only slightly affected. The reason is the much more homogeneous distribution of outliers outside of the nearly Gaussian distribution as opposed to the Mie winds (see Fig. 2c and d), so that the rejection of winds with |Z_m|>3.5 mainly leads to a narrowing of the error distribution without changing its centre of mass.

Note that the Z-score filter is sufficiently relaxed to avoid the rejection of wind observations that might be due to extreme wind occurrences. Taking the scaled MAD values for the Mie and Rayleigh winds before applying the filter (5.1 and 7.2 m s⁻¹, respectively) and considering that the median of the wind speed differences is close to zero, only wind results that deviate by more than ≈18 m s⁻¹ (Mie) or ≈25 m s⁻¹ (Rayleigh) from the model are considered to be outliers. This is well above typical model errors that are caused by deficiencies in terms of parametrization and resolution and can, in extreme cases, like in highly convective regions, exceed 10 m s⁻¹ (Rennie et al., 2021).

Figure 3L2B Mie-cloudy (a) and Rayleigh-clear (b) wind error with respect to the ECMWF model background winds vs. the L2B EE for the 11 underflights of the AVATAR-T campaign. The red data points represent outliers, as identified by the modified Z score (|Z_m|>3.5). Note that there are many more Mie outliers with errors greater than ±50 m s⁻¹.

Download

The differences in the outlier distributions also become obvious when plotting the (O-B) wind error against the EE of the respective wind product (Fig. 3). At higher EE, the number of winds with larger error with respect to the model increases, as expected. However, while the larger Rayleigh wind errors are almost evenly distributed, there is a preponderance of Mie outliers with a positive bias relative to the model background. Note that the Mie data include many more outliers with errors greater than ±50 m s⁻¹, which are not displayed in Fig. 3a for the sake of readability. One of these extreme outliers, with a wind error of 146 m s⁻¹, has an EE of only 5.6 m s⁻¹. This outlier would spoil the Mie statistics at a moderate EE threshold of 5.6 m s⁻¹, unless an additional QC scheme, such as the modified Z score, is applied to the dataset. Conversely, there are several winds with a small error, with respect to the model, but high EE. Some of these low-error winds are, however, non-physical results and merely part of the random wind distribution around the reference model wind, especially for the Rayleigh channel (see also Fig. 2a and b).

3.3 Mie gross errors

Outliers that are not represented by a Gaussian distribution can be referred to as gross errors and are usually caused by data transmission or instrument failure. In the case of the Aeolus Mie-cloudy winds, gross errors generally originate from the false detection of a Mie fringe by the Mie Core algorithm, especially at weak particulate backscatter signals when the Rayleigh background or other noise sources may produce a small peak on the Mie detector, which is then erroneously registered as a fringe. This results in non-physical wind speeds with large departures from the actual wind, depending on the position of the peak, which is also determined by the illumination conditions of the Fizeau interferometer and thus by the telescope obscuration along the orbit. As shown in Fig. 3a, such gross errors are not reliably assigned an adequately large EE to be filtered out accordingly by common QC methods.

Experience with conventional observation systems and associated instrument and retrieval QC settings in operational meteorological analysis suggest that the rate, or probability, of gross errors present in the analysis (after QC) should not exceed a few percent to avoid deterioration of the NWP skill (Lorenc and Hammon, 1988). This is also important to prevent random wind estimates that are, by coincidence, close to the true wind from influencing the analysis. Consequently, in addition to specifying the required wind random error for the Mie and Rayleigh winds in different altitude ranges, the Aeolus MRD contains a gross error requirement, which states that the probability of gross errors shall be less than 5 % within a wind speed range of 6 times the random error requirement (6σ), while no gross errors shall be present outside of this wind speed range (European Space Agency, 2016). The gross error requirement is only applicable to the Mie channel, whereas it is already covered by the random error requirement for the Rayleigh channel, where the noise sources are assumed to be purely Gaussian.

Figure 4Histograms of the L2B Mie-cloudy wind error against ECMWF model background data from the period between 20 and 25 March 2022 for the Northern/Southern hemispheres and ascending/descending orbits. The dashed and dotted lines represent Gaussian fits without and with offset, respectively.

Download

Before launch, it was assumed that the Mie gross errors are uniformly distributed so that, in principle, the increase in random error caused by the gross errors within the 6σ range is less than 5 % (European Space Agency, 2016). However, analysis of the Mie-cloudy wind data after launch revealed a strongly asymmetric distribution of gross errors, with a predominance of positively biased winds, as shown in the previous section. This fact becomes even more obvious when regarding larger datasets. Figure 4 depicts histograms of the (O-B) Mie wind error (in logarithmic scale) from 5 d of data (from 20 through 25 March 2022), subdivided into four groups depending on the geographical location (Northern and Southern hemispheres, NH/SH) and the orbital node of the satellite (ascending and descending). Gaussian fits without and with an offset term are represented as dashed and dotted lines for each data subset, respectively. Due to the complex gross error distribution, which differs among the subsets and features multiple local maxima along the wind error range, the contribution of non-Gaussian errors cannot be approximated by a simple analytic function, e.g. by a constant offset, as assumed before launch, or by a linear relationship. Therefore, the use of a Gaussian fit with offset is not considered useful to describe the Mie wind error distribution including gross errors, as opposed to the pre-launch assumptions. Instead, QC should be performed such that the resulting wind error distribution is as close to a Gaussian distribution (without offset) as possible. However, when using this approach, one accepts the inclusion of non-physical wind observations that, by chance, fall within the Gaussian distribution. A convenient method to assess to what extent a given dataset follows a Gaussian distribution is provided by normal quantile plots which are introduced in the next section.

3.4 Normal quantile plots

Normal quantile plots are a graphical tool for evaluating whether or not a dataset is approximately normally distributed (Chambers et al., 1983). They represent a special case of quantile–quantile plots (QQ plots), where the probability distribution of observed (empirical) data are compared to a specified theoretical distribution by plotting their quantiles against each other. A linear pattern of the scattered points suggests that the measured distribution is reasonably described by the theoretical, e.g. Gaussian, distribution, while departures from a straight line indicate deviations from normality. Usually, a reference line is added to the plot which passes through the first (Q25) and third quartiles (Q75) of the distributions, denoting the values that cut off the first and last quarter of the data when it is sorted in ascending order. The residuals from this reference line are then a measure of the non-normality. The residuals can be measured either in quantiles or in units of the quantity plotted by considering the mean and standard deviation of the sample distribution. A normal quantile plot can be interpreted less ambiguously than a histogram, and the shape of the curve allows conclusions to be drawn about the skewness and kurtosis of the distribution.

Figure 5Normality check of the Rayleigh-clear wind error distribution from the ECMWF model comparison based on the AVATAR-T dataset for two different QC approaches. (a, b) Histograms of the Rayleigh wind error with respect to the ECMWF model background (a) after QC based on an EE threshold of 10 m s⁻¹ and after QC based on the modified Z score (|Z_m|>3.5). The vertical dashed lines indicate the first and third quartiles of the given (sample) distribution (Q25_s; Q75_s) and the theoretical normal distribution used for comparison (Q25_n; Q75_n). The corresponding parameters are also visualized as horizontal and vertical dashed lines (quartile lines) in the normal quantile plots for the two QC approaches plotted in panels (c) and (d), respectively. The quartile reference line (red), which is defined by the intersection points of the quartile lines, is used to calculate the residual quantiles which are depicted in panels (e) and (f).

Download

The use of a normal quantile plot is demonstrated using the example from Sect. 3.1, i.e. the validation of L2B Rayleigh-clear winds against ECMWF model background data from the AVATAR-T campaign (see Fig. 2b and d). Two different QC schemes were applied to study their influence on the normality of the resulting wind error distribution. The first QC corresponds to the widely used approach where L2B winds for which the EE exceeds a certain value are filtered out. Here, a threshold of 10 m s⁻¹ was chosen, which is slightly larger than the most commonly used value of 8 m s⁻¹, but is regarded as an appropriate threshold given the increased noise levels during the campaign compared to the start of the FM-B and elevated EE of the Rayleigh winds accordingly (see Fig. 1). The second QC is based on the modified Z score, discarding those winds for which |Z_m|>3.5. The corresponding histograms are displayed in Fig. 5a and b, with the red bars indicating the data that are removed by the respective QC scheme. While the EE approach rejects 5.5 % of the wind results across the entire wind error range, the QC that relies on the modified Z score (by design) only removes data from the far edges of the distribution. As a consequence, the normal quantile plot for the second approach, which is depicted in Fig. 5d, shows smaller departures from the reference line compared to the EE approach (Fig. 5c). The residuals from the reference line are plotted in the bottom row of Fig. 5, revealing large deviations from normality for the first QC scheme, particularly on the edges. The flipped S-shape of the curve suggests a so-called heavy-tailed distribution of the wind errors, i.e. the existence of large outliers. In other words, the QC scheme based on the EE threshold does not completely remove all the Rayleigh gross errors, so that the resulting (O-B) wind error distribution is not Gaussian, as it is required for a proper assessment of the L2B wind error according to the MRD. In contrast, when using the modified Z score for QC, the wind error distribution (O-B) shows much higher normality with residuals smaller than 1σ (see the grey-shaded area in Fig. 5f), while retaining a higher fraction of valid winds (close to the median) in the dataset.

Although the modified Z score is a useful technique for increasing the normality of a given dataset, it should be noted that the choice of the Z score threshold is crucial. If the chosen limit is too small, e.g.  |Z_m|>2, then a significant portion of the wings of the error distribution is cut off, resulting in a light-tailed distribution which manifests as an S-shaped curve in the normal quantile plot and, hence, decreased normality. For the model comparison of Rayleigh-clear winds, a Z score limit ranging from 3.0 to 3.5 was found to yield a high degree of normality, which is in accordance with the recommendation by Iglewicz and Hoaglin (1993). Concerning the Mie-cloudy winds, the situation is more complicated due to the pronounced asymmetry in the gross error distribution which even affects the outlier-robust median and scaled MAD and, hence, the modified Z score. Therefore, the combination of an EE threshold and a modified Z-score filter is suggested as a reasonable QC approach, as will be elaborated on in Sect. 4. Prior to that, the influence of various QC settings on the statistical results of the model comparison is presented in the following section.

3.5 Influence of the QC on the validation results

The evaluation of the Aeolus wind data quality is predicated on the determination of the systematic (accuracy) and random error (precision) of the wind results in the L2B data product. These key parameters are described by the mean bias μ (Eq. 6) and standard deviation σ (Eq. 7) with respect to the reference instrument or model data. The scaled MAD k (Eq. 8) is sometimes calculated as an additional measure of the wind precision, although it is not compliant with the random error definition that is formulated in the MRD. Note that, in case of a normally distributed (O-B) wind error, the scaled MAD is identical to the standard deviation, and a distinction between the two parameters is obsolete. In this sense, the degree of deviation between k and σ is another measure of non-normality.

Figure 6Results from the statistical comparison of L2B Mie-cloudy (a) and Rayleigh-clear winds (b) against ECMWF model background wind data from the AVATAR-T campaign, depending on the EE threshold without and with outlier removal based on the modified Z score. The bar plots depict the portion of filtered winds after QC (|Z_m|<3.5; blue and green bars) and gross errors (|Z_m|>3.5; red bars) from all wind results that are flagged valid in the L2B product and pass the specified EE threshold. The portion of gross errors is indicated above the bars. The dashed lines and open symbols refer to the statistical results (mean bias μ, standard deviation σ, and scaled MAD k) without removing the gross errors from the datasets, while the solid lines represent the statistical parameters after QC based on the modified Z score.

Download

The statistical parameters are strongly influenced by the applied QC approach and related threshold settings, which also determine the fraction of wind results (with validity flag = 1) that are discarded from the original dataset. For the purpose of summarizing the impact of the chosen QC settings on the validation results, a combined bar and line graph was developed, as depicted in Fig. 6, for the error assessment of the Mie-cloudy and Rayleigh-clear winds against the ECMWF model background data from the AVATAR-T campaign. The diagram intends to give an overview of the most relevant statistical parameters (μ, σ, k, and fraction of valid winds that is considered in the statistics) dependent on the chosen EE threshold. The x axis denotes the EE threshold up to which winds results are retained in the dataset. The left y axis refers to the plotted lines which describe the mean bias (circles), standard deviation (squares), and scaled MAD (diamonds). The statistical parameters are calculated for two cases, i.e. without and with the additional application of a modified Z-score filter (|Z_m|<3.5), represented by the dashed and solid lines, respectively. The bar chart, referring to the right y axis, indicates the percentage of valid data that are included in the statistical analysis, whereby the red columns specify the portion of wind results that are rejected by the modified Z-score filter. Note that the L2B product also contains Rayleigh-clear winds with EE >15 m s⁻¹ (≈3 %), reaching even EE values beyond 100 m s⁻¹, so that the columns in Fig. 6b do not reach 100 %, as opposed to the Mie-cloudy winds depicted in Fig. 6a, where the maximum EE is 14.7 m s⁻¹.

The plot in Fig. 6a demonstrates that the QC based on the EE threshold rejects almost 40 % of the valid Mie winds at a threshold of 4 m s⁻¹ (which is often used in validation studies), while this portion is reduced to less than 20 % when the threshold is relaxed to 6 m s⁻¹. At the same time, multiple large outliers are added to the analysed dataset, including the one with 146 m s⁻¹ mentioned before, and strongly affect the mean bias (from 0.1 to 1.4 m s⁻¹) and standard deviation (from 4.5 to 12.9 m s⁻¹), while the scaled MAD is only slightly increased from 3.7 to 4.0 m s⁻¹. Consequently, solely using the EE as a QC parameter implies that either a large portion of data is discarded or several Mie gross errors are included in the dataset that distort the statistics. In both cases, the validation results become less significant, especially if the original dataset is already small, e.g. in ground-based campaigns with only a few Aeolus overpasses to be analysed.

A solution to this problem is offered by the modified Z-score filter, which sorts out the gross errors and hence permits higher EE thresholds. Using a threshold of 6 m s⁻¹ and removing the aforementioned outliers by a subsequent QC step based on the modified Z score (|Z_m|<3.5) retains 80 % of all valid winds in the dataset, while ensuring robust statistical parameters (μ=0.2 m s⁻¹, σ=4.6 m s⁻¹, and k=3.9 m s⁻¹). The latter are changed to some extent when the EE threshold is further increased up to a point where all valid Mie winds pass the first QC step and only the modified Z-score filter takes effect, rejecting ≈10 % of the data which are identified as gross errors. The resulting values (μ=0.3 m s⁻¹, σ=5.3 m s⁻¹, and k=4.3 m s⁻¹) represent the statistics shown in Fig. 2c and Table 2.

Regarding the Rayleigh-clear wind results, the impact of an additional Z-score filter is less striking, owing to the much more homogeneous distribution of the outliers, as explained in the previous sections. At a typical EE threshold of 8 m s⁻¹ (Witschas et al., 2020), about 90 % of all valid winds pass the first QC step, including less than 1 % of outliers. The latter hardly affect the statistical parameters. The systematic error changes from 0.10 to 0.03 m s⁻¹, while the standard deviation and the scaled MAD decrease from 7.5 to 7.0 m s⁻¹ and from 6.7 to 6.6 m s⁻¹ upon the application of the modified Z-score filter, respectively. When the EE threshold is more and more relaxed, eventually switching off the EE-based QC, the influence of 3 % of outliers in the dataset becomes more pronounced, so that the modified Z-score filter is also useful for the Rayleigh winds, reducing the mean bias from 0.43 to 0.07 m s⁻¹, the standard deviation from 15.1 to 7.8 m s⁻¹, and the scaled MAD from 7.2 to 6.9 m s⁻¹ (see also Table 2).

To conclude, the modified Z score is an effective method to discard gross errors from the L2B data that are detrimental to the validation results, thus enabling higher EE thresholds and, in turn, larger portions of valid winds to be included in the statistical analysis. This is particularly true for the Mie-cloudy winds, where extreme (mainly positively biased) outliers with small EE may occur in the dataset, which, if not properly rejected, lead to a strongly skewed wind error distribution. The choice of the Z score limit, however, remains arbitrary and may have a non-negligible influence on the statistical results, depending on the actual dataset that is used for the validation of the L2B winds. For the model comparison discussed above, the statistical parameters change by less than 7 % if the Z score limit is reduced from 3.5 to 3.0. The largest influence is found for the Rayleigh standard deviation, which decreases to 7.3 m s⁻¹ (compared to 7.8 m s⁻¹ for a Z score limit of 3.5), as the portion of outliers accounts for 4.5 % (compared to 3.2 %).