Comparing the performance of medium-range forecast models

B. Geerts and E. Linacre

3/'02

Davis (1) verified the prognoses provided by four different agencies for 24 hours hence, 48h, 72h, 96h and 120h (i.e. five days). He compared the sea level pressure and 500 hPa patterns in the prognoses to the analyses (i.e. the observations). This was repeated for three consecutive days during a cold outbreak over Australia during 11 - 13th April 1994. In every case the comparison was in terms of nine criteria important to operational forecasters. Conclusions which can be drawn from Appendix 1 in the original paper are these -

  1. On the whole, the best 24h, 48h and 72h prognostics over those three days came from the European Centre for Medium Range Forecasting (average mark 7.3), followed closely by the Medium Range Forecast model of the US National Weather Service (7.2), and then the Australian Global Assimilation and Prediction System (7.1), with the output of the UK Meteorological Office the least satisfactory on this occasion (5.7).
  2. With all four medium-range forecast models, the accuracy of the prognoses was greatest on the first and least on the fifth day. With rare exceptions, the accuracy of prognosis declines with the longer lead-times, as would be expected. For instance, the average mark for the ECMWF prognoses during the comparison period was 8.0 at 24h, 7.3 at 48h, 6.4 at 72h, 5.7 at 96h, and 4.0 at 120h.
  3. The accuracy does not decrease linearly with time. Compared to a linear decay, the 48h, 72h, and 96h forecasts generally are better-than-expected, and the 120h one worse-than-expected. In other words, the quality of the forecasts decays rapidly after 4 days.
  4. The day-to-day variability of the accuracy of the forecasts is high. Presumably, the models are poorly initialised on some days, either because of inadequate data quality and/or quality, or else because an intense storm is poorly sampled.

 

References

(1) Davis, C.J., 1997: An analysis of the Global Computer Models during the cold outbreak situation of 11-13 April 1994 over southeast Australia. Meteor. Note 211 (Australian Bureau of Meteorology), 23pp.