R Cheat Sheet - Lists and Vectors

I have completed a second cheat sheet, this time on R lists and vectors. I grapple with the wonderful world of list indexing. I love the way the R indexing idiom changes depending on whether the object is a vector, list or data.frame (not). As much as I like R, it is ugly and semantically inconsistent under the hood.

Enjoy.

R Cheat Sheet - Data Frames

I have been working with R for a few years now. It is a powerful yet utterly frustrating computer language to learn. It feels like a maze of twisty little passages which are all different yet similar.

For someone from the curly-bracket family of languages (C, C++, Java, Objective-C), R is both hauntingly familiar and completely alien. There is often ten different ways to do much the same thing. Sometimes there will be no difference. Sometimes there are subtle and deep differences which don't matter most of the time. And sometimes these differences raise their head to get you at the most inconvenient time.

One of the ways I learn a language it to put together cheat sheets as I go along. These normally cover the basic syntax of the language, the core functions, and some of the idiom of the language. With R, I have found that I need a lot more cheat sheets than usual.

In this process, I have one cheat sheet that I am relatively happy with when it comes to sharing with others: my R Data Frames Cheat Sheet. Data frames are the work horse of R.

If you can see any errors or if you know of better ways to manipulate data frames: Let me know.

Credit Aggregates for August 2012

The Reserve Bank of Australia has released the credit aggregates for August 2012.

Credit growth appears to be slowing.

It is slowing in all sectors except investor housing, and even that is pretty flat.

Monthly growth and trend monthly growth charts below highlight the significant slowing in business credit growth in July and August 2012. 

The actual aggregates follow.

Government Finances II - Grovelling Apology

Yesterday I posted on the Government Finances for June. I included the raw monthly data, a 12 month rolling total and a 25-term Henderson moving average. My objective with the latter two was to get an understanding of the trends that might lurk behind a very noisy series.

I suspect I failed. And that I failed badly!

It looks like the noise in the data (and the extraordinary expenses in June) overwhelmed my attempts to discern a trend.

We will start my grovelling apology with the Henderson Moving Average for Company Tax. The first chart is with the data to May 2012. The second chart draws on the data to June 2012 (as published yesterday).

What we are looking at here is all artifact. Sorry about that. Let's look at the same series, but this time using a seasonal decomposition (again with May first then June).

There is still some movement in the trend line between May and June, but this time at least it is not a completely different story. Still, we are not out of the woods. Look at the noise in the seasonally adjusted series.

Even these trends from the seasonal decomposition can be misleading. The next two charts - the first to May, the second to June - compare the trends for total expenses and total revenue.

Which is it? Are we on the verge of a Budget surplus or are the horns of our dilemma growing further apart? What ever the outcome ends up being, it is highly unlikely that the June data will be seen in retrospect as proof of a complete turnaround when it came to balancing the Budget.

As they say in the classics, if you torture the data for long enough, it will tell you anything.

Dear reader, I apologise for subjecting this data to excessive torture.

Household finances

Today the ABS released the Financial Accounts. The table I find most interesting is the state of household finances. In financial terms it looks like the household sector was a little poorer in Q2 2012. And certainly still poorer than prior to the GFC.

Government finances

For eleven months of the year, the Department of Finance produces monthly financial statements. They come out somewhere between one month and three months after the conclusion of the relevant month. The last set issued was for May 2012.

Within a couple of weeks of their publication, the Reserve Bank of Australia takes these statements and compiles them into its historical series: table E1. The RBA has a monthly and annual series.

The one month of the year that is a bit tricky is June. The RBA Table E1 row for June 2012 can be reconstructed from the Final Budget Outcome document, but it takes a little bit of work. Well quite a bit of work, which I have now completed.

Because the data is pretty noisy, I have used both a twelve month rolling average and a 25-term Henderson Moving Average to peer into the underlying trend. I have also provided the raw monthly data as bar charts.

Let's start with PAYG tax, which appears to be holding up reasonably well.

Company tax, on the other hand ... looks a little unhealthy in June. The July reading will be important to get a handle on how this revenue stream is holding up. I suspect the Henderson moving average mechanism does not cope well with the seasonality of this data series - I might look at a seasonal decomposition next time I work with this data.

The total revenue story ... might be holding. It will be worth watching this series for the next couple of months.

Expenses growth has slowed noticeably since 2010. Looking at the bar chart, this is the fourth June with above monthly expenses (probably additional payments and prepayments of 2012-13 items).

Looking at the underlying cash balance ...

The headline surplus/deficit ...

And finally, a quick comparison ... we can see that revenue has been growing faster than expenses since mid 2011. If the government can keep these trends going - faster revenue growth while constraining expenditure growth - it just might just get to a balanced budget on time.

Evaluating Budget Forecasts

George Megalogenis made the following observation in today's Australian.

The long-run budget forecasts the Treasurer issued at the depths of the global financial crisis - which warned that the surplus would not be restored until 2015-16 - have turned out to be closer to the truth than his more recent upward revisions that promised a surplus three years earlier, in the present financial year. Mr Swan insists Labor will secure that surplus with extra spending cuts to be announced and implemented before the next federal election.

So let's have a look. The following charts present the actual Federal Budget outcomes against various government forecasts. The forecasts labeled with a B are from the May Budget. The forecasts labeled with an M are from the Mid-Year Economic and Fiscal Outlook (MYEFO) which typically comes out in November. The forecast labeled with a U comes from the Updated Economic and Fiscal Outlook (UEFO) in February 2009 (which accompanied the announcement of the second stimulus package). I have charted the underlying cash balance and the fiscal balance against the various projections.

With MYEFO in 2008 (which included the first stimulus package) and UEFO in 2009 we see the first two of three downward estimates. The May 2009 forward estimates are the most pessimistic, and have proved the most accurate to date. Since MYEFO 2009, the government has consistently overestimated its capacity to return the budget to surplus.

One trick statisticians use to measure the accuracy of a forecast is to look at the out-of-sample mean square error of the forecast compared with the actuals when they come in. The formula follows.

Unfortunately not all of the actuals are in for the budget forecasts from 2009, but for those that are in, the next two tables gives us an idea how the government performed with each of its forward estimates.

Fiscal Balance:

	Series	MSE	Count Forecast Observations Checked	Percent Forecast Checked
1	Budget07	2145.04	5.00	100.00
2	MYEFO07	2954.19	4.00	100.00
3	Budget08	3734.83	5.00	100.00
4	MYEFO08	2830.80	4.00	100.00
5	UEFO09	323.28	4.00	100.00
6	Budget09	9.52	4.00	80.00
7	MYEFO09	96.92	3.00	75.00
8	Budget10	397.16	3.00	60.00
9	MYEFO10	609.98	2.00	50.00
10	Budget11	309.22	2.00	40.00
11	MYEFO11	145.95	1.00	25.00
12	Budget12	6.16	1.00	20.00

Underlying cash balance:

	Series	MSE	Count Forecast Observations Checked	Percent Forecast Checked
1	Budget07	1998.95	5.00	100.00
2	MYEFO07	2735.79	4.00	100.00
3	Budget08	3261.88	5.00	100.00
4	MYEFO08	2384.63	4.00	100.00
5	UEFO09	224.44	4.00	100.00
6	Budget09	30.35	4.00	80.00
7	MYEFO09	55.76	3.00	75.00
8	Budget10	332.91	3.00	60.00
9	MYEFO10	513.74	2.00	50.00
10	Budget11	224.82	2.00	40.00
11	MYEFO11	44.09	1.00	25.00
12	Budget12	0.44	1.00	20.00

As can be seen, the Budget 09-10 forecast (which is 80 per cent evaluated) is only beaten by the Budget 12-13 forecast (which is only 20 per cent evaluated - and that evaluation was of a single forecast of a result two months hence from the date of the forecast). The MSE can only stay the same or increase as future actuals become available and the estimate series is evaluated against them.

Final Budget Outcome 2011-12

Yesterday, the Treasurer and the Minister for Finance released the Final Budget Outcome for 2011-12. Some charts from the report follow.

First, the cash accounting charts ... (and a quick point of clarification 3e+05 = 300,000 - ie. receipts over the past two financial years exceeded $300 billion).

In accrual terms ...

And moving from annual to cumulative, net debt ... record breaking in nominal dollar terms ... but some way to go as a proportion of GDP.