Bank-Wide Backtesting

Table 1 reports the probabilities of obtaining a particular number of exceptions from a sample of 250 independent observations under several assumptions about the actual percentage of outcomes that the model captures (ie these are binomial probabilities). For example, the left- hand portion of Table 1 sets out probabilities associated with an accurate model (that is, a true coverage level of 99%). Under these assumptions, the column labelled “exact” reports that exactly five exceptions can be expected in 6.7% of the samples. Probabilities of exceptions from 250 independent observations Table 1 Model is accurate Model is inaccurate: possible alternative levels of coverage Coverage = 99% Coverage = 98% Coverage = 97% Coverage = 96% Coverage = 95% Exact Type 1 Exact Type 2 Exact Type 2 Exact Type 2 Exact Type 2 0 8.1% 100.0% 0.6% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 1 20.5% 91.9% 3.3% 0.6% 0.4% 0.0% 0.0% 0.0% 0.0% 0.0% 2 25.7% 71.4% 8.3% 3.9% 1.5% 0.4% 0.2% 0.0% 0.0% 0.0% 3 21.5% 45.7% 14.0% 12.2% 3.8% 1.9% 0.7% 0.2% 0.1% 0.0% 4 13.4% 24.2% 17.7% 26.2% 7.2% 5.7% 1.8% 0.9% 0.3% 0.1% 5 6.7% 10.8% 17.7% 43.9% 10.9% 12.8% 3.6% 2.7% 0.9% 0.5% 6 2.7% 4.1% 14.8% 61.6% 13.8% 23.7% 6.2% 6.3% 1.8% 1.3% 7 1.0% 1.4% 10.5% 76.4% 14.9% 37.5% 9.0% 12.5% 3.4% 3.1% 8 0.3% 0.4% 6.5% 86.9% 14.0% 52.4% 11.3% 21.5% 5.4% 6.5% 9 0.1% 0.1% 3.6% 93.4% 11.6% 66.3% 12.7% 32.8% 7.6% 11.9% 10 0.0% 0.0% 1.8% 97.0% 8.6% 77.9% 12.8% 45.5% 9.6% 19.5% 11 0.0% 0.0% 0.8% 98.7% 5.8% 86.6% 11.6% 58.3% 11.1% 29.1% 12 0.0% 0.0% 0.3% 99.5% 3.6% 92.4% 9.6% 69.9% 11.6% 40.2% 13 0.0% 0.0% 0.1% 99.8% 2.0% 96.0% 7.3% 79.5% 11.2% 51.8% 14 0.0% 0.0% 0.0% 99.9% 1.1% 98.0% 5.2% 86.9% 10.0% 62.9% 15 0.0% 0.0% 0.0% 100.0% 0.5% 99.1% 3.4% 92.1% 8.2% 72.9% Notes to Table 1: The table reports both exact probabilities of obtaining a certain number of exceptions from a sample of 250 independent observations under several assumptions about the true level of coverage, as well as type 1 or type 2 error probabilities derived from these exact probabilities. The left-hand portion of the table pertains to the case where the model is accurate and its true level of coverage is 99%. Thus, the probability of any given observation being an exception is 1% (100% – 99% = 1%). The column labelled "exact" reports the probability of obtaining exactly the number of exceptions shown under this assumption in a sample of 250 independent observations. The column labelled "type 1" reports the probability that using a given number of exceptions as the cut-off for rejecting a model will imply erroneous rejection of an accurate model using a sample of 250 independent observations. For example, if the cut-off level is set at five or more exceptions, the type 1 column reports the probability of falsely rejecting an accurate model with 250 independent observations is 10.8%. The right-hand portion of the table pertains to models that are inaccurate. In particular, the table concentrates of four specific inaccurate models, namely models whose true levels of coverage are 98%, 97%, 96% and 95% respectively. For each inaccurate model, the exact column reports the probability of obtaining exactly the number of exceptions shown under this assumption in a sample of 250 independent observations. The type 2 columns report the probability that using a given number of exceptions as the cut-off for rejecting a model will imply erroneous acceptance of an inaccurate model with the assumed level of coverage using a sample of 250 independent observations. For example, if the cut-off level is set at five or more exceptions, the type 2 column for an assumed coverage level of 97% reports the probability of falsely accepting a model with only 97% coverage with 250 independent observations is 12.8%.