16.3 Backtesting in Action

The data used in this section is a bond portfolio of a German bank from 1994 to 1995. The portfolio is not adjusted so that the exposure vector $ w_t=w$ is time dependent. We assume that (15.7) and (15.8) hold. The VaR forecast is based on both prediction rules introduced in Section 15.1 that are used to estimate the parameters $ \sigma_t$ of the forecast distribution in RMA and EMA given $ \gamma = 0.94$. In light of the bond crisis in 1994 it is interesting how both techniques respond to this stress factor.

Fig.: The dots show the observed changes $ L_t$ in the value of the portfolio. The dashed line represents the forecasted VaRs based on RMA (99% and 1%). The solid line represents the same for EMA. 28050 SFEVaRbank.xpl 28053 SFEVaRtimeplot.xpl
\includegraphics[width=1.2\defpicwidth]{plot1.ps}

The significance level under consideration is $ \alpha = 1\%$ for large losses and $ \alpha = 99\%$ for large profits. To investigate we include plots of time series from the realized P/L (i.e., profit-loss) data $ L_t$ as compared to the respective VaR estimator $ \widehat{VaR}_t$ calculated with (15.12). If the model and the estimation of the parameter $ \sigma_t$ based on forecast distribution are adequate, then approximately $ 1\%$ of the data should lie below the $ 1\%$ and above the $ 99\%$ VaR Estimators. In addition in Figure 15.1 the crossings for the case where VaR is estimated with EMA are marked. We recognize that in 1994 (1995) there were a total of 10 (9) crossings determined for the EMA method. This strongly contrasts the 17 (3) observed values for the RMA Method. It is clear that the RMA technique leads to, above all during the bond crisis in 1994, too many crossings for the $ 1\%$ VaR estimator, which means that the probability of larger losses is underestimated. This tendency to underestimate the risk is produced from the observation width of $ 250$ days, when the market is moving towards a more volatile phase. The opposite is true when moving in the other direction; RMA overestimates risk. The EMA adapts more quickly to market phases since data in the past has less of influence on the estimator due to the exponentially deteriorating weights. With 28056 SFEVaRtimeplot.xpl we have calculated the estimated VaRs for another bank using the EMA and RMA respectively.

The poor forecast quality of the RMA, in particular for the left side of the distribution, can also be seen in that for a particular day the VaR was exceeded by $ 400 \%$. If the model (15.7) - (15.8) is correct, then the variable (15.19) must have a standard deviation of about $ 0.41$. īThe empirical standard deviation calculated from the data is about 0.62. According to the volatility scale of the RMA the risk is underestimated on average by $ \frac{0.62 - 0.41}{0.41} \approx 50
\%$. The EMA plot in Figure 15.1 shows a better calibration. The empirical standard deviation of (15.19) is in this case around $ 0.5$, which corresponds to an underestimation of risk by approximately $ 25\%$.

All other diagnostic measurements are entered into the QQ plot of the variable

$\displaystyle \frac{L_{t+1}}{\widehat{VaR}_t} = \frac{L_{t+1}}{2.33 \hat{\sigma}_t},$ (16.19)

see Figure 15.2 and Figure 15.3.

Fig.: QQ plot of $ L_{t+1}/\widehat{VaR}_t$ for RMA in 1994. 28060 SFEVaRqqplot.xpl
\includegraphics[width=0.9\defpicwidth]{plot2.ps}

If the VaR forecast $ \widehat{VaR}_t$ was perfect, the QQ plot would produce a straight line and fill out the area in $ [-1,1]$.

Fig.: QQ plot of $ L_{t+1}/\widehat{VaR}_t$ for EMA in 1994. 28064 SFEVaRqqplot.xpl
\includegraphics[width=0.9\defpicwidth]{plot3.ps}

A comparison of the graphs in Figure 15.2 and Figure 15.3 show that the EMA method is calibrated better than the RMA method. The RMA method clearly shows outliers at both ends. The interval boundaries of $ [-1,1]$ are in both cases clearly exceeded. This indicates a possible inadequacy of an assumed normal distribution. QQ plots for the year 1995 are not shown, which also clearly show the dominance of EMA over RMA.

Another important assumption of our model is the independence of the re-scaled random variable $ Z_t$.

Fig.: Time diagram of the exceedances at the 80% significance level from VaR for RMA (left) and EMA. The superiority of EMA is obvious. 28068 SFEVaRtimeplot2.xpl
\includegraphics[width=1.2\defpicwidth]{plot4.ps}

Figure 15.4 shows the outliers of another bank

$\displaystyle \{ t, { \boldsymbol{1}}(L_{t+1} > \widehat{VaR}_t ) \}, \ t=1, ..., 750,$ (16.20)

as a function of $ t$. The contradictory temporal non-uniform distribution of the outliers from the independence of $ Z_t$ is clearer to see by the RMA method than by the EMA method.

The exploratory analysis clearly shows the differences between RMA and EMA. As a supplement we now compare both estimation techniques with an appropriate test within the framework of the model (15.7) - (15.8). We again consider the sample residuals $ \hat{Z}_{t+1}$ from (15.16) and set the threshold value in (15.14) to $ u=0.8416$, i.e., to the $ 80\%$ quantile of the distribution of $ Z_{t+1} = \frac{L_{t+1}}{\sigma_t}$. From this we obtain $ \vartheta = 1.4$ according to (15.14). Due to the asymptotic distribution (15.18) we can check the significance of the hypothesis

$\displaystyle H_0 \ : \vartheta \stackrel{(<)}{=} 1.4.$ (16.21)

A better approximation than the standard normal distribution for the sample is the Student $ t(20)$ distribution, if we generalize the degrees of freedom.

$\displaystyle {\cal L} (\hat{Z}_{t+1}) = {\cal L} \big(\frac{L_{t+1}}{\hat{\sigma}_t}\big) \approx t(20).$ (16.22)

The value of $ \vartheta$ obtained differs from the value given above by $ 5 \%$, the corresponding variances $ \varsigma^2$ by $ 18
\%$. Therefore, we also consider the hypothesis

$\displaystyle H_0 \ : \vartheta \stackrel{(<)}{=} 1.47.$ (16.23)

The following Table 15.2 to Table 15.5 summarizes our results.


Table: $ H_0 \ : \vartheta \stackrel{(<)}{=} 1.4 $
Method $ \vartheta = 1.4$ $ \varsigma = 0.46 $ $ \sqrt{N(u)} \frac{\hat{\vartheta} - \vartheta}{\hat{\varsigma}} $ significance $ N(u)$
EMA $ \hat{\vartheta} = 1.72$ $ \hat{\varsigma} = 1.01 $ 2.44 0.75% 61
RMA $ \hat{\vartheta} = 1.94$ $ \hat{\varsigma} = 1.3 $ 3.42 0.03% 68



Table: $ H_0 \ : \vartheta \stackrel{(<)}{=} 1.47 $
Method $ \vartheta = 1.47 $ $ \varsigma = 0.546 $ $ \sqrt{N(u)} \frac{\hat{\vartheta} - \vartheta}{\hat{\varsigma}} $ significance $ N(u)$
EMA $ \hat{\vartheta} = 1.72$ $ \hat{\varsigma} = 1.01 $ 2.01 2.3% 61
RMA $ \hat{\vartheta} = 1.94$ $ \hat{\varsigma} = 1.3 $ 3.04 0.14% 68


From Table 15.2 and Table 15.3 it is obvious that the observed outliers for EMA are calibrated better than for the RMA method. For a random sample of $ 260$ values we expect $ 52$ outliers (standard deviation $ 6.45$). For EMA we observe $ 61\, (61-52 \approx 1.5 \cdot$   standard deviation$ )$ outliers and for RMA $ 68 \, (68 - 52 \approx 2.5 \cdot$    standard deviation$ )$. Naturally the outliers influence the test considerably. Therefore, we repeat the analysis excluding the outliers and obtain (15.4) and (15.5).


Table: $ H_0 \ : \vartheta \stackrel{(<)}{=} 1.4 $ largest outlier excluded
Method $ \vartheta = 1.4$ $ \varsigma = 0.46 $ $ \sqrt{N(u)} \frac{\hat{\vartheta} - \vartheta}{\hat{\varsigma}} $ significance $ N(u)$
EMA $ \hat{\vartheta} = 1.645$ $ \hat{\varsigma} = 0.82 $ 2.31 1% 60
RMA $ \hat{\vartheta} = 1.83$ $ \hat{\varsigma} = 0.93 $ 3.78 0.00% 67



Table: $ H_0 \ : \vartheta \stackrel{(<)}{=} 1.47 $ largest outlier excluded
Method $ \vartheta = 1.47 $ $ \varsigma = 0.546 $ $ \sqrt{N(u)} \frac{\hat{\vartheta} - \vartheta}{\hat{\varsigma}} $ significance $ N(u)$
EMA $ \hat{\vartheta} = 1.645$ $ \hat{\varsigma} = 0.82 $ 1.65 5% 60
RMA $ \hat{\vartheta} = 1.83$ $ \hat{\varsigma} = 0.93 $ 3.1 0.15% 67


Concluding we can say that the EMA method gives better calibrated results than the RMA method. Both methods are extremely sensitive to outliers and should both be considered. Even the EMA method suffers from the assumptions (15.7) - (15.8), which are based on the Delta-Normal Model, can only be approximately fulfilled. The residuals $ Z_t$ are neither normally distributed nor independent, although the EMA method is not strongly effected by the independence assumption due to its exponentially decreasing memory.