Online Aggregation of Probabilistic Forecasts Based on the Continuous Ranked Probability Score

Abstract—Methods for generating predictions online and in the form of probability distributions of future outcomes are considered. The difference between the probabilistic forecast (probability distribution) and the numerical outcome is measured using the loss function (scoring rule). In practical statistics, the continuous ranked probability score (CRPS) is often used to estimate the discrepancy between probabilistic forecasts and (quantitative) outcomes. The paper considers the case when several competing methods (experts) give their online predictions as distribution functions. An algorithm is proposed for online aggregation of these distribution functions. The performance bounds of the proposed algorithm are obtained in the form of a comparison of the cumulative loss of the algorithm and the loss of expert hypotheses. Unlike existing estimates, the proposed estimates do not depend on time. The results of numerical experiments illustrating the proposed methods are presented.

SUBSTITUTION FUNCTION

For arbitrary loss function $\lambda (\gamma ,\omega )$ (γ ∈ [0, 1] and ω ∈ {0, 1}), we consider a parametric curve on the plane

$$\left( {{{e}^{{ - \eta \lambda (\gamma ,0)}}},{{e}^{{ - \eta \lambda (\gamma ,1)}}}} \right).$$

(A1)

We consider the scenario in which the curve is concave. The concavity condition is written as

$$x{\kern 1pt} '(\gamma )y{\kern 1pt} ^{"}(\gamma ) - x{\kern 1pt} ^{"}(\gamma )y{\kern 1pt} '(\gamma ) \geqslant 0,$$

(A2)

for all γ, where x(γ) = ${{e}^{{ - \eta \lambda (\gamma ,0)}}}$ and y(γ) = ${{e}^{{ - \eta \lambda (\gamma ,1)}}}$.

In particular, for quadratic loss function $\lambda (\gamma ,\omega )$ = ${{(\gamma - \omega )}^{2}}$, we have x(γ) = ${{e}^{{ - \eta {{\gamma }^{2}}}}}$ and y(γ) = ${{e}^{{ - \eta {{{\left( {\gamma - 1} \right)}}^{2}}}}}$.

After simple transformations, inequality (A2) is equivalent to inequality

$$\eta \gamma (1 - \gamma ) \leqslant \frac{1}{2}.$$

Quantity $\gamma (1 - \gamma )$ takes on a maximum value of 1/4 at 0 ≤ γ ≤ 1, so that the concavity condition is satisfied for any γ at 0 < η ≤ 2.

The η-mixability condition means that, for any distribution w = $({{\omega }_{1}},...,{{\omega }_{N}})$ on a set of N experts and any forecasts f = $({{f}_{1}},...,{{f}_{N}})$, we can find γ* for which inequalities

$${{e}^{{ - \eta (\lambda (\gamma ^{ *},\omega ))}}} \geqslant \sum\limits_{i = 1}^N {{{w}_{i}}{{e}^{{ - \eta \lambda ({{f}_{i}},\omega )}}}} ,$$

(A3)

are satisfied at ω = 0, 1. Points $\left( {{{e}^{{ - \eta \lambda ({{f}_{i}},0)}}},{{e}^{{ - \eta \lambda ({{f}_{i}},1)}}}} \right)$ (i = 1, …, N) belong to curve (A1), and their convex combination (point M) lies inside the convex region bounded by such a curve. Condition (A3) means that the abscissa and ordinate of point N = $({{e}^{{ - \eta \lambda (\gamma ^{ *},0)}}},{{e}^{{ - \eta \lambda (\gamma ^{ *},1)}}})$ are no less than the abscissa and ordinate of point M. We search for point N. The line passing through point M marks point N = $({{e}^{{ - \eta \lambda (\gamma ^{ *},0)}}},{{e}^{{ - \eta \lambda (\gamma ^{ *},1)}}})$ on curve (A1) (see Fig. 1). Forecast γ* is calculated from the condition

$$\frac{{{{e}^{{ - \eta \lambda (\gamma ^{ *},1)}}}}}{{{{e}^{{ - \eta \lambda (\gamma ^{ *},0)}}}}} = \frac{{\sum\limits_{i = 1}^N {{{w}_{i}}{{e}^{{ - \eta \lambda ({{f}_{i}},1)}}}} }}{{\sum\limits_{i = 1}^N {{{w}_{i}}{{e}^{{ - \eta \lambda ({{f}_{i}},0)}}}} }}.$$

(A4)

For a quadratic loss function, equality (A4) yields the following expression for γ*:

$$\gamma ^{ *} = {\text{Subst}}({\mathbf{f}},{\mathbf{w}}) = \frac{1}{2} - \frac{1}{{2\eta }}\ln \frac{{\sum\limits_{i = 1}^N {{{w}_{i}}{{e}^{{ - \eta f_{i}^{2}}}}} }}{{\sum\limits_{i = 1}^N {{{w}_{i}}{{e}^{{ - \eta {{{({{f}_{i}} - 1)}}^{2}}}}}} }}.$$

The mixability condition for the quadratic loss function makes it possible to use η = 2.

It can be easily shown that, for any ω ∈ [0, 1], function f(γ) = ${{e}^{{ - \eta {{{(\gamma - \omega )}}^{2}}}}}$ is concave with respect to γ ∈ [0, 1] for 0 < η < 1/2. In this case, quantity γ* = $\sum\nolimits_{i = 1}^N {{{w}_{i}}{{f}_{i}}}$ satisfies inequality (A3) at any 0 < η < 1/2 in accordance with the definition of concavity.