Bank-Wide Backtesting

Table 2 sets out the Committee’s agreed boundaries for these zones and the presumptive SAMA response for each backtesting outcome, based on a sample of 250 observations. For other sample sizes, the boundaries should be deduced by calculating the binomial probabilities associated with true coverage of 99%, as in Table 1. The backtesting amber zone begins at the point such that the probability of obtaining that number or fewer exceptions equals or exceeds 95%. Table 2 reports these cumulative probabilities for each number of exceptions. For 250 observations, it can be seen that five or fewer exceptions will be obtained 95.88% of the time when the true level of coverage is 99%. Thus, the backtesting amber zone begins at five exceptions. Similarly, the beginning of the backtesting red zone is defined as the point such that the probability of obtaining that number or fewer exceptions equals or exceeds 99.99%. Table 2 shows that for a sample of 250 observations and a true coverage level of 99%, this occurs with 10 exceptions. Backtesting zone boundaries Table 2 Backtesting zone Number of exceptions Backtesting-dependent multiplier (to be added to any qualitative add- on per [MAR 33.44]) Cumulative probability Green 0 1.50 8.11% 1 1.50 28.58% 2 1.50 54.32% 3 1.50 75.81% 4 1.50 89.22% Amber 5 1.70 95.88% 6 1.76 98.63% 7 1.83 99.60% 8 1.88 99.89% 9 1.92 99.97% Red 10 or more 2.00 99.99% Notes to Table 2: The table defines the backtesting green, amber and red zones that SAMA will use to assess backtesting results in conjunction with the internal models approach to market risk capital requirements. The boundaries shown in the table are based on a sample of 250 observations. For other sample sizes, the amber zone begins at the point where the cumulative probability equals or exceeds 95%, and the red zone begins at the point where the cumulative probability equals or exceeds 99.99%. The cumulative probability is simply the probability of obtaining a given number or fewer exceptions in a sample of 250 observations when the true coverage level is 99%. For example, the cumulative probability shown for four exceptions is the probability of obtaining between zero and four exceptions. Note that these cumulative probabilities and the type 1 error probabilities reported in Table 1 do not sum to one because the cumulative probability for a given number of exceptions includes the possibility of obtaining exactly that number of exceptions, as does the type 1 error probability. Thus, the sum of these two probabilities exceeds one by the amount of the probability of obtaining exactly that number of exceptions. 16.18 The backtesting green zone needs little explanation. Since a model that truly provides 99% coverage would be quite likely to produce as many as four exceptions in a sample of 250 outcomes, there is little reason for concern raised by backtesting results that fall in this range. This is reinforced by the results in Table 1, which indicate that accepting outcomes in this range leads to only a small chance of erroneously accepting an inaccurate model. 16.19 The range from five to nine exceptions constitutes the backtesting amber zone. Outcomes in this range are plausible for both accurate and inaccurate models, although Table 1 suggests that they are generally more likely for inaccurate models than for accurate models. Moreover, the results in Table 1 indicate that the presumption that the model is inaccurate should grow as the number of exceptions increases in the range from five to nine. 16.20 Table 2 sets out the Committee’s agreed guidelines for increases in the multiplication factor applicable to the internal models capital requirement, resulting from backtesting results in the backtesting amber zone.