Optimizing Expected Shortfall under an ℓ₁ Constraint—An Analytic Approach

Abstract

Expected Shortfall (ES), the average loss above a high quantile, is the current financial regulatory market risk measure. Its estimation and optimization are highly unstable against sample fluctuations and become impossible above a critical ratio $r=N/T$, where N is the number of different assets in the portfolio, and T is the length of the available time series. The critical ratio depends on the confidence level $\alpha$, which means we have a line of critical points on the $\alpha -r$ plane. The large fluctuations in the estimation of ES can be attenuated by the application of regularizers. In this paper, we calculate ES analytically under an ${\ell }_1$ regularizer by the method of replicas borrowed from the statistical physics of random systems. The ban on short selling, i.e., a constraint rendering all the portfolio weights non-negative, is a special case of an asymmetric ${\ell }_1$ regularizer. Results are presented for the out-of-sample and the in-sample estimator of the regularized ES, the estimation error, the distribution of the optimal portfolio weights, and the density of the assets eliminated from the portfolio by the regularizer. It is shown that the no-short constraint acts as a high volatility cutoff, in the sense that it sets the weights of the high volatility elements to zero with higher probability than those of the low volatility items. This cutoff renormalizes the aspect ratio $r=N/T$, thereby extending the range of the feasibility of optimization. We find that there is a nontrivial mapping between the regularized and unregularized problems, corresponding to a renormalization of the order parameters.

1. Introduction

A risk measure is a functional on the probability distribution of the fluctuating returns of a security or a portfolio. Since it is impossible to condense all the information in a probability distribution into a single number, there is no unique way to choose the “best” risk measure. In Markowitz’s ground breaking portfolio selection theory [1], with the assumption of Gaussian distributed returns, variance offered itself as the natural risk measure. The crises of the late eighties and early nineties led both the industry and regulators to realize that the most dangerous risk lurked in the asymptotically far tail of the return distribution. To grasp this risk, a high quantile of the profit and loss distribution called Value at Risk (VaR) was introduced by J.P. Morgan [2]. For a certain period, VaR became a kind of industry standard, and it was embraced by international financial regulation as the official risk measure in 1996 [3]. Value at Risk is a threshold which losses only exceed with a small probability (such as, e.g., 0.05 or 0.01), corresponding to a confidence level of $\alpha =0.95$, resp. $0.99$. (In this context, it is customary to regard losses as positive and profits as negative). As a quantile, VaR is not sensitive to the distribution of losses above the confidence level and is not subadditive when two portfolios are combined. This triggered a search for alternatives and led Artzner et al. [4] to formulate a set of axioms that any coherent risk measure should satisfy. The simplest and most intuitive of these coherent measures is the Expected Shortfall (ES) [5,6]. ES is essentially the expected loss above a high quantile that can be chosen to be the VaR itself. After a long debate about the relative merits and drawbacks of ES, whose details are not pertinent to our present study, regulators adopted ES as the current official market risk measure to be used to assess the financial health of banks and determine the capital charge they are required to hold against their risks. The regulators and the industry settled on a confidence level of $\alpha =0.975$ [7].

ES is mainly designed to be a diagnostic tool. At the same time, it is also a constraint that banks have to respect when considering the composition of their portfolios. It is then in their best interest to optimize ES, in order to keep their capital charge as low as possible. However, the optimization of ES is fraught with problems of estimation error, which is quite natural if one considers that the number of different items N in a bank’s portfolio can be very large, whereas the number of observations (the length of the available time series T) is always limited. In addition, at the regulatory confidence level, one has to throw away 97.5% of the data. Moreover, the estimation error increases with the ratio $r=N/T$ and at a critical value of r, it actually diverges, growing beyond any limit. As shown in [8], the instability of the optimization of ES (as well as all the coherent risk measures) follows directly from the coherence axioms [4].

The divergence of ES is the signature of a phase transition. The critical r for ES is smaller or equal to $1/2$, its value depending on the confidence level $\alpha$. For ES, there is then a line of critical points, a phase diagram, on the $r-\alpha$ plane. A part of this phase diagram has been traced out by numerical simulations in [9], while the full phase diagram has been determined by analytical calculations by Ciliberti et al. [10]. Going beyond merely determining the phase diagram, a detailed study of the estimation error and other relevant quantities has been performed inside the whole feasibility region in [11,12], and it was shown that, due to the nontrivial behavior of the contour lines of constant estimation error, especially in the vicinity of $\alpha =1$, the number of data necessary to have a reasonably low estimation error was way above any T available in practice.

Because of the large sample fluctuations of ES, its optimization constitutes a problem in high dimensional statistics [13]. A standard tool to tame these large fluctuations is to introduce regularizers, which penalize large excursions. Although the introduction of these penalties may seem an arbitrary statistical trick coming from outside of finance, it was shown in [14] that these regularizers express liquidity considerations, and take into account, already at the construction of the portfolio, the expected market impact of a future liquidation. The regularizers are usually chosen to be some constraints on the norm of the portfolio weights. In [15], we studied the effect of an ${\ell }_2$ regularizer on ES and found that ${\ell }_2$ obviously suppresses the instability and, for sufficiently small r and with a strong enough regularizer, it extends the range where the estimation error is reasonably small by a factor of about 4.

It is interesting to see how an ${\ell }_1$ regularizer works with ES. (The importance of studying the effect of various regularizers in combination with the different risk measures was emphasized by [16]). The regularizer ${\ell }_1$ is known to produce sparse solutions, which means that in order to rein in large fluctuations, it eliminates some of the securities from the portfolio. This obviously contradicts the principle of diversification, but considerations of transaction costs or the technical difficulties of managing large portfolios may make it desirable to remove the most volatile items from the portfolio, and this is precisely what a no-short constraint tends to do.

It has been known for 20 years now that the optimization of ES can be translated into a linear programming problem [17]. Accordingly, as it has been realized in [18], the piece-wise linear ${\ell }_1$ with an infinite slope corresponding to an infinite penalty on short selling can prevent the instability of ES. The purpose of this paper is to determine the effect of ${\ell }_1$-regularization on the phase diagram and also on the behavior of the various quantities of interest inside the region where the optimization of ES is feasible and meaningful. (We will see that as a result of regularization new characteristic lines appear on the $r-\alpha$ plane, beyond which the optimization of ES is still mathematically feasible, but the results become meaningless, as they correspond to negative risk.) In [12], a detailed analytical investigation of the behavior of the estimation error, the in-sample cost, the sensitivity to small changes in the composition of the portfolio, and the distribution of optimal weights were carried out in the non-regularized case. Here, we derive the same quantities for an ${\ell }_1$-regularized ES, including the special case where short selling is banned, that is when the portfolio weights are constrained to be non-negative. The density of the items eliminated from the portfolio, to be referred to as the “condensate” in the following, is also determined. The most striking result of the present study is that the regularized solution can be mapped back onto the unregularized one. We are not aware of a similarly tight relationship between a regularized and an unregularized problem, not only in a finance context, but neither in the general context of machine learning.

2. Method and Preliminaries

If the true probability distribution of returns were known, it would be easy to calculate the true value of Expected Shortfall and the optimal portfolio weights. However, the true distribution of returns is unknown, therefore one has to rely on finite samples of empirical data. This means one observes N time series of length T and estimates the optimal weights and ES on the basis of this information. It is clear that the weights and ES so obtained will deviate from their “true” values. (The latter would be obtained in an infinitely long stationary sample.) The deviation of the estimated values will be the stronger the shorter the length T and the larger the dimension N. Performing this measurement on different samples one would obtain different estimates: there is a distribution of ES and of the optimal weights over the samples. In a real market, one cannot repeat such an experiment multiple times. Instead, one has to squeeze out as much information as possible from a single sample of limited size. There are well-known numerical methods for this, like cross-validation or bootstrap [19]. In contrast, in the present work we aim to obtain analytic results. In order to mimic empirical sampling, we choose a simple data generating process, such as a multivariate Gaussian. The true value of ES is easy to obtain for this case, which provides a standard to measure finite sample deviations from. Then we determine ES for a large number of random samples of length T drawn from this underlying distribution, average it over the random samples and finally compare this average to its true value. This procedure will give us an idea about how large the estimation error is for a given dimension N, sample size T, and confidence level $\alpha$, under the idealized conditions of stationarity and Gaussian fluctuations, and how much it will be reduced when we apply an ${\ell }_1$ regularizer of a given strength. It is reasonable to assume that the estimation error obtained under these idealized circumstances will be a lower bound to the estimation error for real-life processes.

Now we wish to implement this program via analytic calculations. The averaging over the random samples just described is analogous to the averaging over the random realization of disorder in the statistical physics of random systems, which enables us to borrow methods from that field, in particular the replica method [20]. It assumes that both N and T are large, with their ratio $r=N/T$ kept finite (thermodynamic or Kolmogorov limit). A small value of r corresponds to the classical setup in statistics where one has a large number of observations relative to the dimension. Estimates in this case are sharp and close to their true values. In contrast, when r is of order unity, or larger, we are in the high dimensional limit where fluctuations are large. It is here that the regularizer becomes important.

In the usual application of ${\ell }_1$ in finite dimensional numerical studies, the regularizer eliminates the dimensions one by one, in a stepwise manner, as the strength of the regularizer is increasing. In our present work, the large $N,T$ limit and the averaging over infinitely many samples result in a continuous dependence of the “condensate” density (the relative number $N_0/N$ of the dimensions eliminated by ${\ell }_1$) on the aspect ratio r, the confidence level $\alpha$, and the strength of ${\ell }_1$. In a study of ${\ell }_1$-regularized variance [21], we found that the stepwise increase of the density of eliminated weights in a numerical experiment nicely follows the continuous curve obtained analytically. It is obvious that the situation is similar in the case of ES, but we have also confirmed this by numerical simulations.

For the sake of simplicity, we will also assume that the returns are independent, that is the true covariance matrix is diagonal. This is not an innocent assumption: it will be seen, for example, that the maximum degree of sparsity that ${\ell }_1$ can achieve in this scheme is one half of the total number of dimensions, whereas for correlated returns the maximum sparsity can be either larger or smaller than $1/2$, according to whether correlations are predominantly positive or negative. Combining ${\ell }_1$ with a non-diagonal covariance matrix poses additional technical difficulties that we wish to avoid in the present account. However, we do allow the diagonal elements ${\sigma }_i$ of the covariance matrix to be different from each other.

As a further simplification, we do not impose any other constraint on the optimization of ES beside the budget constraint and the ${\ell }_1$ regularizer. In particular, we do not set a constraint on the expected return, and seek the global minimum of the regularized ES. This is in line with a number of studies, [22,23,24] among others, which focus on the global minimum in the problem of variance optimization, because of the extremely noisy estimates of the expected return. Furthermore, the global minimum is precisely what one needs in minimizing tracking-errors, that is, when trying to follow, say, a market index as closely as possible [23].

The replica method used below have already been applied with minor variations to various portfolio optimization problems in a number of papers [10,11,12,14,18,21,25,26,27,28], where the replica derivation of the main formulae were repeatedly explained, so we do not need to go through that exercise again here. Then the natural starting point for our present work is the detailed study of the behavior of ES without regularization in [12]. The argument there leads to a relationship between ES and an effective cost or free energy per asset f as follows:

$$\mathrm{ES}=\frac{fr}{1-\alpha }.$$

(1)

The free energy f itself is given by the minimum of a functional depending on six order parameters

$$\begin{array}{ccc}\hfill f(\lambda ,ϵ,q_0,\Delta ,{\widehat{q}}_0,\widehat{\Delta })& =& \lambda +\frac{1}{r}(1-\alpha )ϵ-\Delta {\widehat{q}}_0-\widehat{\Delta }{q}_0\hfill \\ \hfill & +& {\langle {\min }_w\left[V(w,z,\sigma )\right]\rangle }_{\sigma ,z}+\frac{\Delta }{2r\sqrt{\pi }}\underset{-\infty }{\overset{\infty }{\int }}\mathrm{d}s{e}^{-s^2}g\left(\frac{ϵ}{\Delta }+s\sqrt{\frac{2q_0}{{\Delta }^2}}\right),\hfill \end{array}$$

(2)

where

$$V(w,z,\sigma )=\widehat{\Delta }{\sigma }^2{w}^2-\lambda w-zw\sigma \sqrt{-2{\widehat{q}}_0}+{\eta }^{+}\theta \left(w\right)w-{\eta }^{-}\theta (-w)w$$

(3)

and the double average ${\langle \cdots \rangle }_{\sigma ,z}$ means

$$\underset{0}{\overset{\infty }{\int }}\mathrm{d}\sigma \frac{1}{N}\sum _i\delta (\sigma -{\sigma }_i)\underset{-\infty }{\overset{\infty }{\int }}\frac{\mathrm{d}z}{\sqrt{2\pi }}{e}^{-z^2/2}\dots$$

(4)

Finally, the function g in the integral in (2) is defined as

$$g\left(x\right)=\left\{\begin{array}{cc}0,& x\ge 0\\ x^2,& -1\le x\le 0\\ -2x-1,& x<-1\end{array}\right..$$

(5)

The differences with respect to the setup in [12] are the following: a trivial change of notation ($\tau$ there is $1/r$ here); the variable $\sigma$ has been introduced in (3), which together with the recipe (4) allows us to consider assets with different volatilities ${\sigma }_i$; and the regularizer has been built into the effective potential (3). Note that the ${\ell }_1$ in (3) is asymmetric in order to allow us to penalize long and short positions separately. The usual ${\ell }_1$ corresponds to ${\eta }^{+}={\eta }^{-}$, the ban on short selling to ${\eta }^{-}\to \infty$. We will also use the arrangement where there is a finite penalty ${\eta }^{-}$ on short positions and none on long ones ${\eta }^{+}=0$.

A note on signs: for consistency, the order parameters $\lambda$, $\Delta$, $q_0$, and $\widehat{\Delta }$ must be positive, ${\widehat{q}}_0$ negative, and $ϵ$ can be of either sign. Furthermore, $\lambda$ must be larger or equal to the right slope of the regularizer: $\lambda \ge {\eta }^{+}$.

Before setting out to derive the stationarity conditions that determine the optimal value of the free energy and thence of ES, we spell out the meaning of the order parameters. The first of these is the Lagrange multiplier $\lambda$ that enforces the budget constraint:

$$\sum _{i=1}^N{w}_i=N.$$

(6)

Note that the sum of portfolio weights is set to N here, instead of the usual 1. This is to keep the weights of order unity in the large N limit.

Because of the relationship between $\lambda$ and the budget constraint, $\lambda$ can be thought of as a kind of chemical potential. It is an important quantity, because, as we shall see later, its value at the stationary point is equal to the free energy, hence directly related to the optimal value of ES. In [12], we argued that this optimal value of ES is, in fact, the in-sample estimate of Expected Shortfall. According to (1), ES is proportional to the product $fr$, which means f, and hence $\lambda$ too, must be inversely proportional to r when $r=N/T\to 0$, because ES is certainly finite in this limit: a finite N and $T\to \infty$ corresponds to the case of having complete information. This spurious divergence of f and $\lambda$ is an artifact, due to our having absorbed a factor $1/r$ in their definition. This is explained purely by convenience: we wish to keep as close to the convention in [12] as possible. The opposite limit, when $\lambda -{\eta }^{+}$ vanishes, is another important point: it signals the instability of the portfolio, and the onset of the phase transition.

The next order parameter, $ϵ$, was suggested by [17] as a proxy for Value at Risk. Indeed, in the limit $r\to 0$ where we know the true distribution of returns, $ϵ$ will be seen to be equal to the known value of VaR for a Gaussian.

The third order parameter, $q_0$, is of central importance: According to [12], the ratio of the out-of-sample estimate ${\mathrm{ES}}_{out}$ and its true value ${\mathrm{ES}}^{\left(0\right)}$ is given by the square root of $q_0$. For the case of different ${\sigma }_i$s considered here, $q_0$ has to be amended by a factor depending on the structure of the portfolio [21] as

$${\tilde{q}}_0=q_0\frac{1}{N}\sum _i\frac{1}{{\sigma }_i^2}.$$

(7)

Then the ratio of the estimated and true ES will be

$$\frac{{\mathrm{ES}}_{out}}{{\mathrm{ES}}^{\left(0\right)}}=\sqrt{{\tilde{q}}_0}$$

(8)

that is the relative estimation error is $\sqrt{{\tilde{q}}_0}-1$.

The fourth order parameter, $\Delta$, measures the sensitivity to a small shift in the returns.

The remaining two order parameters, ${\widehat{q}}_0$ and $\widehat{\Delta }$, are auxiliary variables that do not have an obvious meaning, they enter the picture through the replica formalism, and can be eliminated once the stationarity conditions have been established. The stationarity or saddle point conditions are derived by taking the derivative of the free energy with respect to the order parameters and setting them to zero. They will be written up in the next Section.

3. Results

First, we are going to spell out the saddle point conditions in full detail and reduce them to special cases later.

Let us bring the integral in (2) to a more convenient form by integrating by parts:

$$I=\frac{1}{\sqrt{\pi }}\underset{-\infty }{\overset{\infty }{\int }}\mathrm{d}s{e}^{-s^2}g\left(\frac{ϵ}{\Delta }+s\sqrt{\frac{2q_0}{{\Delta }^2}}\right)=\frac{2q_0}{{\Delta }^2}\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]-1-2\frac{ϵ}{\Delta }.$$

(9)

With this identity, the free energy becomes

$$f=\lambda -\frac{\alpha ϵ}{r}-\Delta {\widehat{q}}_0-\widehat{\Delta }{q}_0-\frac{\Delta }{2r}+\frac{q_0}{r\Delta }\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]+{\langle \min V\rangle }_{\sigma ,z}.$$

(10)

The function W in the above formulae, together with two related functions $\mathsf{\Phi }$ and $\mathsf{\Psi }$, will frequently appear in the following; they are integrals of the Gaussian $\frac{1}{\sqrt{2\pi }}{e}^{-x^2/2}$:

$$\begin{array}{ccc}\hfill \mathsf{\Phi }\left(x\right)& =& {\int }_{-\infty }^x\mathrm{d}t\frac{1}{\sqrt{2\pi }}{e}^{-t^2/2}\hfill \end{array}$$

(11)

$$\begin{array}{ccc}\hfill \mathsf{\Psi }\left(x\right)& =& {\int }_{-\infty }^x\mathrm{d}t\mathsf{\Phi }\left(t\right)\hfill \end{array}$$

(12)

$$\begin{array}{ccc}\hfill W\left(x\right)& =& {\int }_{-\infty }^x\mathrm{d}t\mathsf{\Psi }\left(t\right).\hfill \end{array}$$

(13)

Now we evaluate the minimum of V in (3) and denote the “representative weight” where this minimum is located by $w^{*}$. It works out to be

$$w^{*}=\frac{\lambda +\sigma z\sqrt{-2{\widehat{q}}_0}-{\eta }^{+}\mathsf{\Theta }\left(w^{*}\right)+{\eta }^{-}\mathsf{\Theta }(-w^{*})}{2{\sigma }^2\widehat{\Delta }},$$

(14)

or

$$w^{*}=\left\{\begin{array}{c}\frac{\lambda +\sigma z\sqrt{-2{\widehat{q}}_0}-{\eta }^{+}}{2{\sigma }^2\widehat{\Delta }},\mathrm{if}z\ge \frac{{\eta }^{+}-\lambda }{\sigma \sqrt{-2{\widehat{q}}_0}}\hfill \\ \hfill \\ 0,\mathrm{if}-\frac{\lambda +{\eta }^{-}}{\sigma \sqrt{-2{\widehat{q}}_0}}<z<\frac{{\eta }^{+}-\lambda }{\sigma \sqrt{-2{\widehat{q}}_0}}\hfill \\ \hfill \\ \frac{\lambda +\sigma z\sqrt{-2{\widehat{q}}_0}+{\eta }^{-}}{2{\sigma }^2\widehat{\Delta }},\mathrm{if}z\le -\frac{\lambda +{\eta }^{-}}{\sigma \sqrt{-2{\widehat{q}}_0}}.\hfill \end{array}\right.$$

(15)

With this and (4), one can calculate $V^{*}$, the value of V at the minimum, and perform the double averaging to obtain

$${\langle V^{*}\rangle }_{\sigma ,z}=\frac{{\widehat{q}}_0}{\widehat{\Delta }}\frac{1}{N}\sum _i\left[W\left(\frac{\lambda -{\eta }^{+}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)+W\left(-\frac{\lambda +{\eta }^{-}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)\right].$$

(16)

Then, the fully explicit form of the free energy becomes

$$\begin{array}{ccc}\hfill f& =& \lambda -\frac{\alpha ϵ}{r}-\Delta {\widehat{q}}_0-\widehat{\Delta }{q}_0-\frac{\Delta }{2r}+\frac{q_0}{r\Delta }\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]\hfill \\ \hfill & +& \frac{{\widehat{q}}_0}{\widehat{\Delta }}\frac{1}{N}\sum _i\left[W\left(\frac{\lambda -{\eta }^{+}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)+W\left(-\frac{\lambda +{\eta }^{-}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)\right].\hfill \end{array}$$

(17)

It is now straightforward to take the derivatives of f with respect to the order parameters and derive the stationary conditions.

From $\partial f/\partial \lambda =0$, it follows that

$$1=\frac{\sqrt{-2{\widehat{q}}_0}}{2\widehat{\Delta }}\frac{1}{N}\sum _i\frac{1}{{\sigma }_i}\left[\mathsf{\Psi }\left(\frac{\lambda -{\eta }^{+}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)-\mathsf{\Psi }\left(-\frac{\lambda +{\eta }^{-}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)\right].$$

(18)

The derivative with respect to ${\widehat{q}}_0$ yields

$$2\Delta \widehat{\Delta }=\frac{1}{N}\sum _i\left[\mathsf{\Phi }\left(\frac{\lambda -{\eta }^{+}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)+\mathsf{\Phi }\left(-\frac{\lambda +{\eta }^{-}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)\right].$$

(19)

From the derivative with respect to $\widehat{\Delta }$, we get

$$q_0=-\frac{{\widehat{q}}_0}{{\widehat{\Delta }}^2}\frac{1}{N}\sum _i\left[W\left(\frac{\lambda -{\eta }^{+}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)+W\left(-\frac{\lambda +{\eta }^{-}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)\right].$$

(20)

As mentioned before, $q_0$ determines the out-of-sample estimate for ES and the estimation error.

The derivative with respect to $q_0$ leads to

$$2r\Delta \widehat{\Delta }=\mathsf{\Phi }\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-\mathsf{\Phi }\left(\frac{ϵ}{\sqrt{q_0}}\right),$$

(21)

where use has been made of the identity

$$W\left(x\right)=\frac{1}{2}x\mathsf{\Psi }\left(x\right)+\frac{1}{2}\mathsf{\Phi }\left(x\right).$$

(22)

The condition for the derivative with respect to $ϵ$ to vanish is

$$\alpha =\frac{\sqrt{q_0}}{\Delta }\left[\mathsf{\Psi }\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-\mathsf{\Psi }\left(\frac{ϵ}{\sqrt{q_0}}\right)\right].$$

(23)

The derivation of the last equation takes a little more effort. Let us go back to the free energy in (2) and take the derivative with respect to $\Delta$. Noticing that ${\langle V\rangle }_{\sigma ,z}$ does not depend on $\Delta$, and using the integral given in (9), we have

$$\frac{\partial f}{\partial \Delta }=-{\widehat{q}}_0+\frac{1}{2r}I+\frac{\Delta }{2r}\frac{\partial I}{\partial \Delta }=0$$

(24)

valid at the stationary point. From here we find

$$\frac{1}{2r}{I}_{st}={\widehat{q}}_0+\frac{2q_0}{r{\Delta }^2}\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]-\frac{ϵ}{r\Delta }-\frac{\sqrt{q_0}}{r\Delta }\mathsf{\Psi }\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right),$$

(25)

where (9) was used again and we denoted by $I_{st}$ the integral I evaluated at the stationary point. Now we apply the identity (22) and the stationary conditions (23), (21) to arrive at

$$\frac{1}{2r}{I}_{st}={\widehat{q}}_0+\frac{2q_0\widehat{\Delta }}{\Delta }-(1-\alpha )\frac{ϵ}{r\Delta },$$

(26)

which, combined with (9), finally leads to

$${\widehat{q}}_0+\frac{2q_0\widehat{\Delta }}{\Delta }+\alpha \frac{ϵ}{r\Delta }+\frac{1}{2r}-\frac{q_0}{r{\Delta }^2}\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]=0.$$

(27)

The Equations (18)–(23) and (27) constitute the system of equations for the six order parameters. These equations are valid both for the regularized and (setting ${\eta }^{+}={\eta }^{-}=0$) for the unregularized cases.

Let us now work out the relationship between the free energy and the chemical potential. Comparing (16) and (20), we see that ${\langle V^{*}\rangle }_{\sigma ,z}=-q_0\widehat{\Delta }$, which with (10) and (27), results in the simple formula

$$f=\lambda$$

(28)

at the stationary point, as we anticipated before. In [12], we argued that the stationary value of f determines the in-sample estimate of ES through (1).

The last object to determine is the distribution of weights:

$$p\left(w\right)={\langle \delta (w-w^{*})\rangle }_{\sigma ,z}.$$

(29)

With (14), we find

$$\begin{array}{ccc}\hfill p\left(w\right)& =& n_0\delta \left(w\right)+\frac{1}{N}\sum _i\frac{1}{{\sigma }_w^{\left(i\right)}\sqrt{2\pi }}\exp \left(-\frac{1}{2}{\left(\frac{w-w_i^{+}}{{\sigma }_w^{\left(i\right)}}\right)}^2\right)\theta \left(w\right)\hfill \end{array}$$

(30)

$$\begin{array}{ccc}& +& \frac{1}{N}\sum _i\frac{1}{{\sigma }_w^{\left(i\right)}\sqrt{2\pi }}\exp \left(-\frac{1}{2}{\left(\frac{w-w_i^{-}}{{\sigma }_w^{\left(i\right)}}\right)}^2\right)\theta (-w),\hfill \end{array}$$

(31)

where $\delta \left(w\right)$ is the Dirac delta,

$${\sigma }_w^i=\frac{\sqrt{-2{\widehat{q}}_0}}{2\widehat{\Delta }{\sigma }_i}$$

(32)

is the (estimated) variance of the ith return,

$$w_i^{+}=\frac{\lambda -{\eta }^{+}}{2{\sigma }_i^2\widehat{\Delta }}$$

(33)

is the center of the Gaussian distribution of the (estimated) positive weight i,

$$w_i^{-}=\frac{\lambda +{\eta }^{-}}{2{\sigma }_i^2\widehat{\Delta }}$$

(34)

is the same for negative weight i, and finally,

$$n_0=\frac{1}{N}\sum _i\left[\mathsf{\Phi }\left(\frac{\lambda +{\eta }^{-}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)-\mathsf{\Phi }\left(\frac{\lambda -{\eta }^{+}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)\right]$$

(35)

is the density of the assets whose weights are set to zero by the regularizer.

We wish to make an important remark here: the right hand side of (19) is just $1-n_0$. This will prove to be the key to the mapping between the regularized and unregularized cases.

Let us record the condensate density $n_0$ also for the special case when short positions are excluded (${\eta }^{-}\to \infty$), but long positions are not penalized (${\eta }^{+}=0$):

$$n_0=\frac{1}{N}\sum _i\left[1-\mathsf{\Phi }\left(\frac{\lambda }{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)\right].$$

(36)

From (36), we can see that, since $\mathsf{\Phi }\left(x\right)$ is monotonic increasing and, for $x\ge 0$, concave, the contribution to $n_0$ from assets with larger ${\sigma }_i$s is larger than that from smaller ${\sigma }_i$s. This means that in the no-short limit, the regularizer ${\ell }_1$ eliminates more volatile assets with larger probability than the less volatile ones. Thus, we can think of the no-short constraint as a smooth upper cutoff in volatility. This is not true in the generic case (35), where the contributions of the small and large volatility items depend on the order parameters and the regularizer’s slopes ${\eta }^{+}$ and ${\eta }^{-}$ in a complicated manner: the probability of an asset with volatility ${\sigma }_i$ to be removed is given by the difference of the two term in (35) under the sum. We do not wish to analyze this situation in detail, apart from the remark that a sufficiently large ${\eta }^{-}$ generally favors the elimination of large volatility items.

The integral of $p\left(w\right)$ is, of course, 1. Its first moment, ${\langle w^{*}\rangle }_{\sigma ,z}$, works out to be the same as (18):

$${\langle w^{*}\rangle }_{\sigma ,z}=1.$$

(37)

The second moment of the weight distribution is readily obtained as

$${\langle {\left(w^{*}\right)}^2\rangle }_{\sigma ,z}=-\frac{{\widehat{q}}_0}{{\widehat{\Delta }}^2}\frac{1}{N}\sum _i\frac{1}{{\sigma }_i^2}\left[W\left(\frac{\lambda -{\eta }^{+}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)+W\left(-\frac{\lambda +{\eta }^{-}}{{\sigma }_i\sqrt{-2{\widehat{q}}_0}}\right)\right].$$

(38)

The variance of the weight distribution is then

$${\langle {\left(w^{*}\right)}^2\rangle }_{\sigma ,z}-{\left({\langle w^{*}\rangle }_{\sigma ,z}\right)}^2,$$

(39)

which is equal to $q_0-1$, when the variances of the assets are all equal to 1. For a portfolio with different ${\sigma }_i$’s, however, the relevant quantity that determines the out-of-sample estimate of ES is not the second moment of the weight distribution, but the true variance of the ith asset multiplied by the estimated portfolio weights squared and summed over the different assets, that is

$${\langle {{\sigma }^2\left(w^{*}\right)}^2\rangle }_{\sigma ,z},$$

(40)

which is precisely $q_0$ as given in (20), and this is the quantity (multiplied by the correction as in (7)) that enters the formula for the out-of-sample estimate of ES in (8). For a not too inhomogeneous portfolio, the difference between the second moment of the weight distribution and $q_0$ is not significant, so we can think of $q_0$ as a measure of the variance of the portfolio.

Now we are ready to consider various special cases.

3.1. The Limit of Complete Information

When we have many observations (very long time series, $T\to \infty$) relative to the dimension N of the portfolio, we are in the $r=N/T\to 0$ limit. As we have already mentioned, this also corresponds to the “chemical potential” $\lambda$ going to infinity. Obviously, in this limit, the regularizer plays no role.

We need the asymptotic behavior of the functions appearing in our stationary conditions: for $x\to \infty$, $\mathsf{\Phi }\left(x\right)\to 1$, $\mathsf{\Psi }\left(x\right)\sim x$, and $W\left(x\right)\sim x^2/2$, while for $x\to -\infty$, all three vanish exponentially.

Then from (18) we have

$$1=\frac{\lambda }{2\widehat{\Delta }}\frac{1}{N}\sum _i\frac{1}{{\sigma }_i^2}.$$

(41)

From (19)

$$2\Delta \widehat{\Delta }=1.$$

(42)

Combining the two:

$$1=\lambda \Delta \frac{1}{N}\sum _i\frac{1}{{\sigma }_i^2}.$$

(43)

We know from (1) and (28) that $\lambda$ must be inversely proportional to r when $r\to 0$. It follows that $\Delta \sim r$ for small r.

Then, from (20) we find

$$q_0={\Delta }^2{\lambda }^2\frac{1}{N}\sum _i\frac{1}{{\sigma }_i^2}.$$

(44)

Combined with the previous equation, this gives

$$q_0=\frac{1}{\frac{1}{N}{\sum }_i\frac{1}{{\sigma }_i^2}}.$$

(45)

The “true” ($r\to 0$) value of the order parameter $q_0$ is thus determined by the structural constant $\frac{1}{N}{\sum }_i\frac{1}{{\sigma }_i^2}$, which is given by the variances of the returns ${\sigma }_i^2$. This is in accord with the corresponding result found in the case of the ${\ell }_1$-regularized variance risk measure [21,29]. The above result for $q_0$ also means that the quantity ${\tilde{q}}_0$ introduced in (7) is equal to 1, and according to (8) the out-of-sample estimate of ES is equal to its true value ${\mathrm{ES}}^{\left(0\right)}$, the estimation error is zero—an obvious result for the case of complete information.

From (23) with $\Delta \to 0$ we obtain $\alpha =\mathsf{\Phi }(ϵ/\sqrt{q_0})$, or

$$ϵ={\mathsf{\Phi }}^{-1}\left(\alpha \right)\sqrt{q_0}.$$

(46)

Now from (21) we get $r={\mathsf{\Phi }}^{\prime }\left(\frac{ϵ}{\sqrt{q_0}}\right)\frac{\Delta }{\sqrt{q_0}}$, or

$$\Delta =r\sqrt{q_0}\frac{1}{\frac{1}{\sqrt{2\pi }}{e}^{-{ϵ}^2/2q_0}}.$$

(47)

However, then we have found

$$\lambda =\frac{q_0}{\Delta }=\frac{1}{r}\frac{1}{\sqrt{2\pi }}{e}^{-{ϵ}^2/2q_0}\sqrt{q_0}=\frac{1}{r}\frac{1}{\sqrt{2\pi }}{e}^{-{\left({\mathsf{\Phi }}^{-1}\left(\alpha \right)\right)}^2/2}\sqrt{q_0}.$$

(48)

Since $\lambda =f$ and $\mathrm{ES}=fr/(1-\alpha )$, we have the $r\to 0$ limit (the true value) of ES:

$${\mathrm{ES}}^{\left(0\right)}=\frac{1}{1-\alpha }\frac{1}{\sqrt{2\pi }}{e}^{-{\left({\mathsf{\Phi }}^{-1}\left(\alpha \right)\right)}^2/2}\sqrt{q_0}.$$

(49)

We record the $r\to 0$ limits of the two auxiliary variables, $\widehat{\Delta }$ and $\widehat{q}$, for completeness:

$$\widehat{\Delta }=\frac{1}{2r\sqrt{q_0}}\frac{1}{\sqrt{2\pi }}{e}^{-{ϵ}^2/2q_0}$$

(50)

and

$${\widehat{q}}_0\sim -\frac{1}{r},$$

(51)

with a coefficient that will not be needed in the following.

Let us turn to the distribution of weights now.

In the $r\to 0$ limit, the widths of the Gaussians in (30) all vanish, so the Gaussians become delta functions:

$$p=\frac{1}{N}\sum _i\delta (w-w_i^{+})\theta \left(w\right)+\frac{1}{N}\sum _i\delta (w-w_i^{-})\theta (-w).$$

(52)

In the $r\to 0$ limit, the weights are all positive, so the second sum disappears.

For the weights, $w_i^{+}$ we find

$$w_i^{+}\simeq \frac{\lambda }{2{\sigma }_i^2\widehat{\Delta }}=\frac{\lambda \Delta }{{\sigma }_i^2}=\frac{1}{{\sigma }_i^2}\frac{1}{\frac{1}{N}{\sum }_k\frac{1}{{\sigma }_k^2}}.$$

(53)

They sum to N, as stipulated.

The variance of a linear combination of independent random variables with averages $w_i^{+}$ and variances ${\sigma }_i^2$ is

$${\sigma }_p^2=\sum _i{\left(w_i^{+}\right)}^2{\sigma }_i^2=\frac{N}{\frac{1}{N}{\sum }_k\frac{1}{{\sigma }_k^2}}.$$

(54)

Now we recognize the meaning of the (true value of the) order parameter $q_0$: it is the normalized (to $\mathcal{O}\left(1\right)$) variance of the portfolio. This also explains the correction factor appearing in (7). We also see that (46) and (49) are the standard expressions for Value at Risk and Expected Shortfall indeed.

We emphasize again that all the results presented in this subsection are only valid in the $r\to 0$ limit when we are dealing with a finite dimension N and infinitely long time series T.

For finite r, the sample fluctuations start to broaden the delta spikes in the distribution of weights, the condensation of zero weights begins, $\lambda$ decreases, and all the formulae above become considerably more complicated. We turn to this situation in the next subsections.

By now, we have learned everything that was to be learned from keeping the variances ${\sigma }_i$ different, in particular the tendency of the elimination of the most volatile assets by the regularizer in the case of restriction of short selling. In order to simplify the presentation and avoid the appearance of very large and hardly transparent formulae, henceforth we set all the ${\sigma }_i$’s equal to 1. We stress, however, that the main message of this paper, namely the existence of a mapping between the regularized and unregularized cases, depends only on the structure of the equations, and works also with different $\sigma$’s.

3.2. Without Regularization

In this subsection, we set ${\eta }^{+}={\eta }^{-}=0$, that is we consider our problem without regularization, and according to what has just been said, put ${\sigma }_i=1$. We will make use of the identities

$$\begin{array}{ccc}\hfill \mathsf{\Phi }\left(x\right)+\mathsf{\Phi }(-x)& =& 1\hfill \end{array}$$

(55)

$$\begin{array}{ccc}\hfill \mathsf{\Psi }\left(x\right)+\mathsf{\Psi }(-x)& =& x\hfill \end{array}$$

(56)

$$\begin{array}{ccc}\hfill W\left(x\right)+W(-x)& =& \frac{1}{2}(x^2+1).\hfill \end{array}$$

(57)

The free energy (17) becomes

$$\begin{array}{ccc}\hfill f& =& \lambda -\frac{\alpha ϵ}{r}-\Delta {\widehat{q}}_0-\widehat{\Delta }{q}_0-\frac{\Delta }{2r}+\frac{q_0}{r\Delta }\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]-\frac{{\lambda }^2}{4\widehat{\Delta }}+\frac{{\widehat{q}}_0}{2\widehat{\Delta }}.\hfill \end{array}$$

(58)

For the saddle point equations, we find:

$$1=\frac{\lambda }{2\widehat{\Delta }},$$

(59)

$$2\Delta \widehat{\Delta }=1,$$

(60)

$$q_0=\frac{{\lambda }^2}{4{\widehat{\Delta }}^2}-\frac{{\widehat{q}}_0}{2{\widehat{\Delta }}^2},$$

(61)

$$2r\Delta \widehat{\Delta }=r=\mathsf{\Phi }\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-\mathsf{\Phi }\left(\frac{ϵ}{\sqrt{q_0}}\right),$$

(62)

$$\alpha =\frac{\sqrt{q_0}}{\Delta }\left[\mathsf{\Psi }\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-\mathsf{\Psi }\left(\frac{ϵ}{\sqrt{q_0}}\right)\right],$$

(63)

$${\widehat{q}}_0+\frac{2q_0\widehat{\Delta }}{\Delta }+\frac{\alpha ϵ}{r\Delta }+\frac{1}{2r}-\frac{q_0}{r{\Delta }^2}\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]=0.$$

(64)

These equations are rather similar to their counterparts in the previous subsection, but of course $r\to 0$ is not assumed here. As for their solutions, they were discussed and illustrated in several figures in [12], therefore we will not dwell upon them here. (Some results will be given in Section 3.6.) Instead, we write up the corresponding equations in the case where no short positions are allowed and make a term-by-term comparison between the two sets of equations.

3.3. No Short Selling

Short positions will be excluded by imposing infinite penalty on them by letting ${\eta }^{-}$ go to infinity. The functions $\mathsf{\Phi }\left(x\right)$, $\mathsf{\Psi }\left(x\right)$, and $W\left(x\right)$ all vanish when $x\to -\infty$. Long positions will not be penalized, so we set ${\eta }^{+}=0$.

The free energy becomes

$$\begin{array}{ccc}f\hfill & =& \lambda -\frac{\alpha ϵ}{r}-\Delta {\widehat{q}}_0-\widehat{\Delta }{q}_0-\frac{\Delta }{2r}+\frac{q_0}{r\Delta }\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]\hfill \end{array}$$

(65)

$$\begin{array}{ccc}& +\hfill & \frac{{\widehat{q}}_0}{\widehat{\Delta }}W\left(\frac{\lambda }{\sqrt{-2{\widehat{q}}_0}}\right).\hfill \end{array}$$

(66)

The stationary conditions now read as:

$$1=\frac{\sqrt{-2{\widehat{q}}_0}}{2\widehat{\Delta }}\mathsf{\Psi }\left(\frac{\lambda }{\sqrt{-2{\widehat{q}}_0}}\right),$$

(67)

$$2\Delta \widehat{\Delta }=\mathsf{\Phi }\left(\frac{\lambda }{\sqrt{-2{\widehat{q}}_0}}\right),$$

(68)

$$q_0=-\frac{{\widehat{q}}_0}{{\widehat{\Delta }}^2}W\left(\frac{\lambda }{\sqrt{-2{\widehat{q}}_0}}\right),$$

(69)

$$2r\Delta \widehat{\Delta }=\mathsf{\Phi }\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-\mathsf{\Phi }\left(\frac{ϵ}{\sqrt{q_0}}\right),$$

(70)

$$\alpha =\frac{\sqrt{q_0}}{\Delta }\left[\mathsf{\Psi }\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-\mathsf{\Psi }\left(\frac{ϵ}{\sqrt{q_0}}\right)\right],$$

(71)

$$r\left({\widehat{q}}_0+\frac{2q_0\widehat{\Delta }}{\Delta }\right)+\frac{\alpha ϵ}{\Delta }+\frac{1}{2}-\frac{q_0}{{\Delta }^2}\left[W\left(\frac{\Delta +ϵ}{\sqrt{q_0}}\right)-W\left(\frac{ϵ}{\sqrt{q_0}}\right)\right]=0,$$

(72)

the last equation being the same as (64), just multiplied by r.

In the distribution of weights in (30), the second sum of Gaussians will disappear, because for ${\eta }^{-}\to \infty$, all the weights (34) go to infinity. The weights (33) become

$$w_i^{+}=\frac{\lambda }{2\widehat{\Delta }},$$

(73)

while the density of zero weights is now

$$n_0=1-\mathsf{\Phi }\left(\frac{\lambda }{\sqrt{-2{\widehat{q}}_0}}\right),$$

(74)

which with (68) leads to

$$1-n_0=2\Delta \widehat{\Delta }.$$

(75)

From (74), we see that $n_0=0$ for $r=0$ and increases as $\lambda$ decreases, until it reaches its maximal value $1/2$ when $\lambda$ vanishes. Mathematically, there is nothing to prevent us from continuing to increase r and driving $\lambda$ to negative values, which would allow $n_0$ to grow beyond $1/2$, up to $n_0=1$, but a negative $\lambda$ would cause the free energy and thus also ES to change sign—an extreme case of “in-sample optimism”, entirely due to the lack of sufficient information. We consider such a situation “unphysical”, and never go beyond the point where $\lambda$ (or $\lambda -{\eta }^{+}$ if ${\eta }^{+}>0$) vanishes anywhere in this paper.

3.4. No-Short Mapping

We are now ready to spell out the mapping between the no-short case and the unregularized one.

The first point to notice is that the only difference between Equation (62) valid in the unregularized case and its counterpart (70) in the no-short case (combined with (75)) appears on their left hand side: the terms r and $(1-n_0)r$, respectively. This suggests to introduce an effective r:

$$r_{\mathrm{eff}}=(1-n_0)r.$$

(76)

Now $r=N/T$, and $n_0$ is the density of the assets removed by the regularizer, thus $(1-n_0)r=\frac{N-N_0}{T}$ is the number of surviving assets divided by the length of the time series. As $r_{\mathrm{eff}}$ increases from zero to $1/2$, r will increase between zero and 1.

Inspired by the connection between r and $r_{\mathrm{eff}}$, we compare the two sets of equations and recognize that, in fact, the whole system of saddle point equations can be mapped from the regularized case to the unregularized one. A variable that appears in all the subsequent equations is

$$z=\frac{\lambda }{\sqrt{-2{\widehat{q}}_0}},$$

(77)

where the variables $\lambda$ and ${\widehat{q}}_0$ are those that appear in the no-short equations.

Then the connection between the order parameters belonging to the two cases is the following:

$$q_0=q_0^{\mathrm{eff}}\frac{z}{\mathsf{\Psi }\left(z\right)},$$

(78)

$$\Delta ={\Delta }_{\mathrm{eff}}\sqrt{\frac{z}{\mathsf{\Psi }\left(z\right)}},$$

(79)

$$ϵ={ϵ}_{\mathrm{eff}}\sqrt{\frac{z}{\mathsf{\Psi }\left(z\right)}},$$

(80)

$$\lambda ={\lambda }_{\mathrm{eff}}\sqrt{\frac{z}{\mathsf{\Psi }\left(z\right)}}\mathsf{\Phi }\left(z\right),$$

(81)

$${\widehat{q}}_0={\widehat{q}}_0^{\mathrm{eff}}\mathsf{\Phi }\left(z\right),$$

(82)

$$\widehat{\Delta }={\widehat{\Delta }}_{\mathrm{eff}}\sqrt{\frac{\mathsf{\Psi }\left(z\right)}{z}}\mathsf{\Phi }\left(z\right).$$

(83)

A direct substitution shows that if the order parameters on the left hand sides of the above equations satisfy the no-short equations, then the effective variables satisfy the unregularized ones, provided we also replace r with $r_{\mathrm{eff}}$. In particular, the contour maps of the unregularized order parameters presented in [12] can be taken over and simply blown up by a factor $\frac{1}{1-n_0}$ to obtain the contour maps of the no-short variables. Given the relation between $q_0$ and the estimation error, we see that the mapping also means that a given error belongs to a larger r in the no-short case than in the unregularized one, in other words, the no-short constrained problem demands $(1-n_0)$ times less data (shorter time series) than the unregularized one.

One may wonder whether this mapping expresses some symmetry of the problem, that is whether the free energy functional is invariant under this mapping. The answer is no: the mapping works only in the saddle point equations, it is a property of the stationary point.

It is important to learn the range of this transformation. In the limit $r\to 0$, the transformation is the identity, but this is trivial: when we have complete information, the regularizer does not play any role. It is more interesting to consider the vicinity of the phase transition in the unregularized case, where $q_0^{\mathrm{eff}}$ and ${\Delta }_{\mathrm{eff}}$ diverge. These divergences are removed by the mapping, no singularity is found in the no-short case. This is in accord with [18]: the infinite penalty on short positions precludes the phase transition and no singularity shows up in $q_0$, $\Delta$, or $ϵ$. Mathematically, we can continue the unregularized solutions into the non-feasible region beyond the phase boundary, but they make no sense there (for example, $q_0$ changes sign, $\Delta$ and $ϵ$ become imaginary, etc.), while their mapped counterparts continue to behave reasonably. According to (76), when $r_{\mathrm{eff}}$ reaches the critical point $r_c\left(\alpha \right)$, the corresponding value of r in the no-short problem will be twice as large, so the whole phase diagram is multiplied by a factor 2. Beyond the mapped phase boundary the regularized solutions still survive, but their meaning becomes questionable, because the free energy, hence also ES change sign. As noted in the previous Subsection, we refrain from the discussion of this unphysical region.

3.5. Mapping for Generic ${\ell }_1$ Constraint

The mapping between the generic ${\ell }_1$-constrained ES optimization and the unregularized one is a straightforward generalization of the results in the previous Subsection. The mapping is made more complicated because of the sums and differences of the $\mathsf{\Psi }$, $\mathsf{\Phi }$, and W functions appearing on the right hand side of Equations (18)–(20). We introduce the following notation for these combinations:

$$A_{\mathsf{\Psi }}=\mathsf{\Psi }\left(\frac{\lambda -{\eta }^{+}}{\sqrt{-2{\widehat{q}}_0}}\right)-\mathsf{\Psi }\left(-\frac{\lambda +{\eta }^{-}}{\sqrt{-2{\widehat{q}}_0}}\right),$$

(84)

$$A_{\mathsf{\Phi }}=\mathsf{\Phi }\left(\frac{\lambda -{\eta }^{+}}{\sqrt{-2{\widehat{q}}_0}}\right)+\mathsf{\Phi }\left(-\frac{\lambda +{\eta }^{-}}{\sqrt{-2{\widehat{q}}_0}}\right),$$

(85)

and

$$A_W=W\left(\frac{\lambda -{\eta }^{+}}{\sqrt{-2{\widehat{q}}_0}}\right)+W\left(-\frac{\lambda +{\eta }^{-}}{\sqrt{-2{\widehat{q}}_0}}\right),$$

(86)

where we have set all the ${\sigma }_i=1$.

In terms of these quantities the generic map reads as

$$q_0=q_0^{\mathrm{eff}}\frac{2A_W-A_{\mathsf{\Phi }}}{{\left(A_{\mathsf{\Psi }}\right)}^2},$$

(87)

$$\Delta ={\Delta }_{\mathrm{eff}}\frac{\sqrt{2A_W-A_{\mathsf{\Phi }}}}{A_{\mathsf{\Psi }}},$$

(88)

$$ϵ={ϵ}_{\mathrm{eff}}\frac{\sqrt{2A_W-A_{\mathsf{\Phi }}}}{A_{\mathsf{\Psi }}},$$

(89)

$$\lambda ={\lambda }_{\mathrm{eff}}\frac{z{A}_{\mathsf{\Phi }}}{\sqrt{2A_W-A_{\mathsf{\Phi }}}},$$

(90)

$${\widehat{q}}_0={\widehat{q}}_0^{\mathrm{eff}}{A}_{\mathsf{\Phi }},$$

(91)

$$\widehat{\Delta }={\widehat{\Delta }}_{\mathrm{eff}}\frac{A_{\mathsf{\Phi }}{A}_{\mathsf{\Psi }}}{\sqrt{2A_W-A_{\mathsf{\Phi }}}}.$$

(92)

For the condensate density $n_0$, we have

$$1-n_0=A_{\mathsf{\Phi }},$$

(93)

and for the effective aspect ratio

$$r_{\mathrm{eff}}=2r\Delta \widehat{\Delta }=r{A}_{\mathsf{\Phi }}=(1-n_0)r.$$

(94)

As before, if the order parameters satisfy the regularized stationarity conditions (18)–(27) (with ${\sigma }_i=1$), then the effective parameters will satisfy the unregularized Equations (59)–(64), and vice versa.

Note that the above equations remain invariant if we redefine $\lambda$ as $\lambda -{\eta }^{+}$ and ${\eta }^{-}$ as ${\eta }^{-}+{\eta }^{+}$. So we can set ${\eta }^{+}=0$ and ${\eta }^{-}+{\eta }^{+}=\eta$ without loss of generality. We will use this setup in the following, in order to reduce the number of parameters when solving the stationarity equations.

3.6. Solutions for the Order Parameters

Except for a few exceptional points, it is impossible to obtain the solutions of the stationarity equations in closed, analytical form, but it is perfectly possible to get them numerically, by a computer. (The case of $\alpha =1$ is exceptional in several respects and will not be considered here.) In the following, the solutions will be presented in graphical form.

Figure 1 exhibits three special lines, belonging to three different cases: the unregularized case, the one with a finite regularizer, and the one with a no-short constraint.

The blue line is the upper boundary of the region where the optimization of unregularized ES is feasible. This line was first determined in [10]. It is a phase boundary, along which a phase transition takes place: $q_0$, $\Delta$, and $ϵ$ diverge here, while $\lambda$ becomes zero. The unregularized equations can be solved also above this line, up to the horizontal line at $r=1$ (not shown in the Figure), but the solutions are meaningless: $q_0$ is negative, while $\lambda$, $\Delta$, and $ϵ$ become imaginary. The unregularized equations do not have any solution above $r=1$.

The green line is the image of the unregularized phase boundary under the mapping described in the previous Subsection, and corresponds to a one-sided regularizer with ${\eta }^{-}=0.05$, ${\eta }^{+}=0$. There is no phase transition when we cross this line, the order parameters remain smooth, finite quantities, but $\lambda$ (along with the free energy and the in-sample estimate of ES) changes sign, rendering the solution in the region above the green line “unphysical”. Nevertheless, if we keep following the solutions beyond the green line we can go up to the image of the $r=1$ line (mapped into $r\to \infty$), where $q_0$ and $\Delta$ will ultimately diverge. The region between the green line and the image of the $r=1$ line has an intricate structure, but because it corresponds to negative risk, it is of no interest for us in the present context.

In the no-short case, there is always a solution with the order parameters remaining finite all the way up to infinity, which is the image of the $r=1$ line under the no-short map. However, as we cross the orange line, $\lambda$ changes sign, and the region beyond it is meaningless again. The orange line is the unregularized phase boundary (blue line) blown up by a factor $\frac{1}{1-n_0}=2$. All this is in accord with the picture described in [18] in that the no-short constraint eliminates the critical line. The solutions becoming unphysical beyond a certain r-range could not be foreseen on the basis of the analysis in [18].

Figure 2 shows the $\eta$-dependence of $q_0$ and the density of the zero weights $n_0$ at criticality, and that of the value of the critical r. In the unregularized case ($\eta \to 0$), $q_0\to \infty$, while in the no-short case ($\eta \to \infty$) $q_0\to \pi$. At $\alpha =0.975$, the value of the critical $r_c$ increases from $r_c\approx 1/2$ in the unregularized case to $\approx 1$ for the no-short case. The proportion of the assets eliminated from the portfolio (the condensate density) goes from zero for $\eta =0$ to $1/2$ for large $\eta$.

In Figure 3, we display the r-dependence of $q_0$, $\Delta$, and $\lambda$ for the three cases: unregularized, regularized, and no-short. Without regularization, $q_0$ and $\Delta$ increase with r and diverge at an $r_c$ slightly less than $\frac{1}{2}$; while $\lambda$ decreases from infinity at $r=0$ to zero at $r_c$. (The confidence limit $\alpha$ is set at its regulatory value 0.975 in these figures.) Under the regularizer ${\eta }^{-}=0.05$, ${\eta }^{+}=0$, $q_0$, and $\Delta$ increases up to the r where $\lambda$ vanishes. The situation is similar for an infinitely strong (no-short) regularizer, with the limiting value of $q_0=\pi$ and $\lambda =0$ at $r\approx 1$.

The left panel in Figure 4 shows the relative out-of-sample estimation error, which is related to the out-of-sample estimate of ES by (8) (${\tilde{q}}_0=q_0$ now, as we have set all the ${\sigma }_i$ =1). These curves are similar to the curves of $q_0$ in the previous Figure. It can be seen that the curves of the relative estimation error run very close to each other for small values of r: there is no substantial reduction of the error in this range. Where they fan out and the effect of regularization starts to be felt (say around $r=0.1$), the relative error is already about 20%.

The middle panel in Figure 4 shows the behavior of the density of zero weights as function of r for the finite $\eta$-regularized and the no-short cases. In the no-short case, $n_0$ reaches its maximal value $\frac{1}{2}$ at $r\approx 1$ (for $\alpha =0.975$) where $\lambda$ vanishes. For a regularizer of finite strength, it always remains below $\frac{1}{2}$.

The right panel in Figure 4 displays the behavior of the in-sample estimate of ES for the three cases. This quantity is directly related to $\lambda$ through (1) and (28). The monotonic and fast decay of these curves demonstrates what is called in-sample optimism, a strong underestimation of risk.

4. Discussion

In the preceding Section we compared the behavior of the order parameters in the three instances considered in this paper: the case of the unregularized, the ${\ell }_1$-regularized, and the no-short constrained Expected Shortfall optimization. We have seen that without regularization, there is a phase transition as we cross the phase boundary $r_c\left(\alpha \right)$ shown in Figure 1 with $\Delta$, $q_0$, and $ϵ$ diverging here, as known since the paper [10]. In contrast, the infinite penalty on short positions suppresses this phase transition, while an ${\ell }_1$ regularizer with finite slopes only shifts the phase boundary. These facts were also known from earlier work [14,18]. However, the picture has turned out to be more complicated than envisaged in [18]. The numerical solution for the order parameters performed in this paper has revealed that new characteristic lines emerge both in the case of finite regularization and the no-short constraint, along which the order parameter $\lambda$ and, consequently, the free energy and the in-sample estimate of Expected Shortfall change sign. We have determined the position of these new characteristic lines: in the no-short case the new line is the curve $2r_c\left(\alpha \right)$, for a finite regularizer it is $\frac{r_c\left(\alpha \right)}{1-n_0}$, where $n_0\le \frac{1}{2}$. We have omitted the detailed analysis of the regions above these lines, where the estimated risk becomes negative. Instead, we confined ourselves to merely pointing out that the critical line for the no-short constraint is projected out to infinity, so the phase transition is removed indeed, while for a finite slope regularizer the critical line is shifted into the unphysical, negative risk region, where for some values of the regularizer’s strength $\eta$, it even develops two branches.

We have also found the behavior of the various order parameters, most notably that of $q_0$ that determines the out-of-sample estimation error of ES, the free energy that gives the in-sample estimator, and the susceptibility-like quantity $\Delta$, and displayed their behavior for the three cases studied here. It is satisfactory to see that $q_0$ and $\Delta$ remain finite up to the new characteristic lines, that is, the regularizer acts as expected: it suppresses the divergent sample fluctuations in the optimization of ES. Unfortunately, this suppression is not strong enough to bring down the estimation error to acceptable values, except for the range of small $r=\frac{N}{T}$ ratios where it demands far too long time series for any realistic N, and where r is small already without any regularization.

What is the meaning of this phase transition? As analyzed in [8,26] it follows from the coherence axioms that coherent risk measures, including ES, are unstable in the sense that whenever an asset or a combination of assets in the portfolio stochastically dominates the others in a given sample, the investor can take an extremely large long position in the dominant asset and compensate this with an appropriately large short position, without violating the budget constraint. This means that the weight of the dominant asset runs away practically to infinity, resulting in an arbitrarily large negative value of the risk measure. This is a mirage of an arbitrage, which can disappear in the next sample, or change into another arbitrage with a different weight running away to infinity. In practice, there are always constraints that prevent such a divergence from taking place. The ban on short selling is just this sort of constraint. The runaway solutions try to escape, but get arrested at the walls constituted by the constraint, in the case of a no-short ban, at the coordinate planes. This is how the condensate of zero weights builds up. This mechanism is the stronger the larger the ratio $r=N/T$.

There is nothing surprising about solutions sitting on the constraint-walls or at corners in a linearly programmable problem, such as the optimization of ES. In the usual applications of linear programming, the constraints typically express some physical limitation like a finite amount of resources, material or labor, etc. In the present finance problem, such a finite resource would be the limited budget, but if short selling is not constrained, the budget in itself cannot prevent runaway solutions. The ban on short positions corresponds to an infinitely strong ${\ell }_1$ regularizer, which, combined with the budget constraint, is already sufficient to take care of the runaway solutions. So, with a no-short ban on, we can increase r (that is the dimension, or decrease the amount of data) without any mathematical contradiction showing up; neither $q_0$ nor $\Delta$ will diverge. It is clear, however, that the solution based on less and less information becomes increasingly meaningless. In these circumstances, the optimization will not tell us anything useful about the structure of the market, it will be determined more and more by the constraint.

What we regard as the most intriguing result of this paper is the existence of a mapping between the regularized and the unregularized problems.