Evaluating Budget Forecasts

George Megalogenis made the following observation in today's Australian.

The long-run budget forecasts the Treasurer issued at the depths of the global financial crisis - which warned that the surplus would not be restored until 2015-16 - have turned out to be closer to the truth than his more recent upward revisions that promised a surplus three years earlier, in the present financial year. Mr Swan insists Labor will secure that surplus with extra spending cuts to be announced and implemented before the next federal election.

So let's have a look. The following charts present the actual Federal Budget outcomes against various government forecasts. The forecasts labeled with a B are from the May Budget. The forecasts labeled with an M are from the Mid-Year Economic and Fiscal Outlook (MYEFO) which typically comes out in November. The forecast labeled with a U comes from the Updated Economic and Fiscal Outlook (UEFO) in February 2009 (which accompanied the announcement of the second stimulus package). I have charted the underlying cash balance and the fiscal balance against the various projections.

With MYEFO in 2008 (which included the first stimulus package) and UEFO in 2009 we see the first two of three downward estimates. The May 2009 forward estimates are the most pessimistic, and have proved the most accurate to date. Since MYEFO 2009, the government has consistently overestimated its capacity to return the budget to surplus.

One trick statisticians use to measure the accuracy of a forecast is to look at the out-of-sample mean square error of the forecast compared with the actuals when they come in. The formula follows.

Unfortunately not all of the actuals are in for the budget forecasts from 2009, but for those that are in, the next two tables gives us an idea how the government performed with each of its forward estimates.

Fiscal Balance:

	Series	MSE	Count Forecast Observations Checked	Percent Forecast Checked
1	Budget07	2145.04	5.00	100.00
2	MYEFO07	2954.19	4.00	100.00
3	Budget08	3734.83	5.00	100.00
4	MYEFO08	2830.80	4.00	100.00
5	UEFO09	323.28	4.00	100.00
6	Budget09	9.52	4.00	80.00
7	MYEFO09	96.92	3.00	75.00
8	Budget10	397.16	3.00	60.00
9	MYEFO10	609.98	2.00	50.00
10	Budget11	309.22	2.00	40.00
11	MYEFO11	145.95	1.00	25.00
12	Budget12	6.16	1.00	20.00

Underlying cash balance:

	Series	MSE	Count Forecast Observations Checked	Percent Forecast Checked
1	Budget07	1998.95	5.00	100.00
2	MYEFO07	2735.79	4.00	100.00
3	Budget08	3261.88	5.00	100.00
4	MYEFO08	2384.63	4.00	100.00
5	UEFO09	224.44	4.00	100.00
6	Budget09	30.35	4.00	80.00
7	MYEFO09	55.76	3.00	75.00
8	Budget10	332.91	3.00	60.00
9	MYEFO10	513.74	2.00	50.00
10	Budget11	224.82	2.00	40.00
11	MYEFO11	44.09	1.00	25.00
12	Budget12	0.44	1.00	20.00

As can be seen, the Budget 09-10 forecast (which is 80 per cent evaluated) is only beaten by the Budget 12-13 forecast (which is only 20 per cent evaluated - and that evaluation was of a single forecast of a result two months hence from the date of the forecast). The MSE can only stay the same or increase as future actuals become available and the estimate series is evaluated against them.