Estimating Changes in the Observed Relationship Between Humidity and Temperature Using Noncrossing Quantile Smoothing Splines

Abstract

The impacts of warm season heat extremes are dependent on both temperature and humidity, so it is critical to properly model their relationship, including how it may be changing. This presents statistical challenges because the bivariate temperature–humidity (measured here by dew point) distribution is complex and spatially variable. Here, we develop a flexible, semiparametric model based on quantile smoothing splines to summarize the distributional dependence of dew point on temperature, including how the dependence is changing with increasing global mean temperature. Noncrossing constraints enforce both the validity of the modeled distributions and the physical constraint that dew point cannot exceed temperature. The proposed method is first demonstrated with four synthetic, representative case studies. We then apply it to data from 2416 weather stations spanning the globe, with a focus on analyzing dew point trends during hot days. In general, dew point is increasing on both hot, humid and hot, dry days in the tropics and high latitudes, but decreasing in the subtropics, especially on hot, dry days. These changes appear to be mostly explained by changes in the temperature–dew point relationship, rather than by increases in temperature with a fixed temperature–dew point relationship.

Matrix form of Noncrossing Quantile Smoothing Splines

In order to solve for the parameters in our regularized quantile smoothing splines model with noncrossing constraints, we write it in standard form as follows:

(5)

and describe each term in this optimization problem below.

Throughout this appendix, bold face implies a two-dimensional matrix, whereas normal script indicates a vector or, in the case of $\tau$ and $\lambda$, a constant. Note that this stands in contrast to the main text, where vectors are bolded, in order to more clearly indicate the relevant matrix operations.

The parameter vector ${z}$ and cost vector ${c}$ are,

$$\begin{aligned} \begin{aligned} {z}&= \begin{bmatrix} {\theta }^+&{\theta }^-&{\epsilon }^+&{\epsilon }^-&{u}^+&{u}^-&{v}^+&{v}^- \end{bmatrix}\\ {c}&= \begin{bmatrix} {0}_{m}&{0}_{m}&\tau {1}_{n}&-\tau {1}_{n}&\lambda {1}_{n-1}&\lambda {1}_{n-1}&\lambda {1}_{n-1}&{\lambda }{1}_{n-1} \end{bmatrix}, \end{aligned} \end{aligned}$$

(6)

where the m model parameters are contained in ${\theta }$ and the remaining components of ${z}$ are the values of the residuals ($\epsilon$’s) and the second derivatives of the ${\alpha }_0$ (u’s) and ${\alpha }_1$ values (v’s), which are the fitted splines at each knot. The superscripts $+$ and − denote the magnitude of the positive and negative components, respectively, such that, e.g., ${\theta } = {\theta }^+ - {\theta }^-$. The cost vector contains the minimization of Eq. (3), with no penalty on the values of ${\theta }$, a quantile-weighted penalty on the residuals, and the $\lambda$-weighted penalty on the second derivatives of the splines.

The equality term, $\mathbf {A}{z}={b}$, is written as,

$$\begin{aligned} \begin{aligned} \mathbf {A}&= \begin{bmatrix} \mathbf {X} &{} -\mathbf {X} &{} \mathbf {1}_{n\times n} &{} -\mathbf {1}_{n\times n} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} \\ \mathbf {D_0} &{} -\mathbf {D_0} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times n} &{} -\mathbf {1}_{n\times (n-1)} &{} \mathbf {1}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} \\ \mathbf {D_1} &{} -\mathbf {D_1} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} -\mathbf {1}_{n\times (n-1)} &{} \mathbf {1}_{n\times (n-1)} \end{bmatrix}\\ {b}&= \begin{bmatrix} {y}&{0}_{n-1}&{0}_{n-1} \end{bmatrix}, ^\intercal \end{aligned} \end{aligned}$$

(7)

where the matrix $\mathbf {X}$ is the design matrix for Eq. (2), the $\mathbf {D}_k$ are the second derivative operators for the spline coefficients, ${\alpha }_k$, and the length (dimensions) of the vectors (matrices) of zeros and ones are indicated by the subscripts. Thus, the first row of $\mathbf {A}$ contains the constraint that the data minus the conditional quantile fit equals the residuals. The second and third rows define that the second derivate of the spline coefficients equal ${u}$ and ${v}$, respectively.

Finally, all noncrossing constraints, as well as the standard constraint that all are in the inequality term, $\mathbf {G}{z}\le {h}$. Define $c=2m+2n+4\times (n-1)$ as the length of z. Then, for the median quantile, $\tau = 0.5$,

$$\begin{aligned} \begin{aligned} \mathbf {G}&= \begin{bmatrix} &{}&{}&{}&{}-\mathbf {1}_{c\times c}&{}&{}&{} \\ \mathbf {X} &{} -\mathbf {X} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} \end{bmatrix}\\ {h}&= \begin{bmatrix} {0}_{c}&{T} \end{bmatrix}, \end{aligned} \end{aligned}$$

(8)

where ${T}$ is the vector of sorted local temperatures. The first row of $\mathbf {G}$ constrains all parameter values in ${z}$ to be positive, whereas the second row ensures that the conditional quantile of dew point does not exceed temperature.

For quantile levels above the median, $\tau _{i} = 0.5 + \delta _i$, where $\delta _i$ is positive and increasing, define ${q}$ as the best fit conditional quantile for the $\tau _{i-1}$ quantile. Then the inequality constraint is slightly modified to:

$$\begin{aligned} \begin{aligned} \mathbf {G}&= \begin{bmatrix} &{}&{}&{}&{}-\mathbf {1}_{c\times c}&{}&{}&{} \\ \mathbf {X} &{} -\mathbf {X} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} \\ -\mathbf {X} &{} \mathbf {X} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} \end{bmatrix}\\ {h}&= \begin{bmatrix} {0}_{c}&{T}&-{q} \end{bmatrix}, \end{aligned} \end{aligned}$$

(9)

where the last row in $\mathbf {G}$ is the noncrossing constraint for consecutive quantiles.

Finally, for quantile levels below the median, $\tau _{i} = 0.5 + \delta _i$, where $\delta _i$ is negative and decreasing, define ${q}$ as the best fit conditional quantile for the $\tau _{i-1}$ quantile. In this case, we can remove the constraint that the conditional dew point quantile is less than the local temperature because it will already be contained within the noncrossing constraint. Thus, the inequality constraint is:

$$\begin{aligned} \begin{aligned} \mathbf {G}&= \begin{bmatrix} &{}&{}&{}&{}-\mathbf {1}_{c\times c}&{}&{}&{} \\ \mathbf {X} &{} -\mathbf {X} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times n} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} &{} \mathbf {0}_{n\times (n-1)} \end{bmatrix}\\ {h}&= \begin{bmatrix} {0}_{c}&{q} \end{bmatrix} \end{aligned} \end{aligned}$$

(10)