Stats, Data and Damn Lies – When Vanity Metrics Attack

I was writing something recently – actually, I’m actually writing something now, and this thought came to me, and rather than continuing to scribble it on the piece of paper I keep handy for just this sort of thing to stop me from getting distracted, I’m going to run with it. Hope it’s worth this rather shapeless intro.

I was making mention in the original document of the high percentage of Australian innovation workers in higher education and government, as opposed to business: hardly new statistics and, as always, I felt a little uneasy about them because as bad as they are, I always expect them to be worse.  For those who haven’t heard me on this particular soapbox, the official stats show 58.7% of Australian researchers are employed in higher education, with a further 8.9% in government. In fact, of the 34 countries surveyed, Australia was in the bottom 5 for its number of researchers employed in business*. 

And this is where it gets muse-worthy. When I say “official stats”, I mean that these are the figures provided to the Organisation for Economic Co-operation and Development (OECD) by the Australian Government.  As a member nation, we provide this sort of information regularly, both for the OECD’s reports on a variety of economic issues but also for the Government’s own reports. And the reports that are generated from this data are quoted and used as evidence for policy development and public speeches and blog posts, and become accepted wisdom – most of Australia’s innovator minds are in higher education and not business.

Now, to be fair, these particular stats clearly say they relate to Australian researchers and the sectors in which they are employed, and my interests are in innovators and creators, which recognises a whole lot more occupations as key to driving innovation, including most of the information technology and communication industry and its sectors as well as designers, architects and other creative industries peoples**. Few of whom are researchers. And including them would certainly affect both my expectations and the end result of any consideration of the public-private employment of Australian innovators.  But let’s stick with the limits of the defined data for just a moment.

You see it occurs to me that here, as in most things, a lot turns on definitions. I remember going to an IT industry event a few years ago at which a guest speaker from the US was discussing unemployment figures. He pointed out that the administration of one US President who shall remain nameless changed the methodology for collecting employment data by excluding from the term “unemployed” everyone who hadn’t had a job for over five years. Not because they had stopped looking, nor had they stopped relying on public support – the administration just gave them a different label, which excluded them from the calculation. And he remains nameless because, whilst well aware of the reason for what some might call an under-report in comparison to what the average citizen thought was still included in the term, subsequent administrations of both persuasions have been very happy to work with the new term.

Which brings me back to the official stats.  There are only three categories against which to report: higher education, government and business, and the countries in question provide data against each category which the OECD uses for its purposes without validation. Because they can’t and, frankly, shouldn’t because the members include all the developed as well as the leading emerging economies who should all be able to count their own people and things with some certainty. Not to mention the OECD actually wants the data, and who would give it if their integrity was going to be questioned?

Now a university is clearly in higher education, and an employee of the Department of Primary Industries is clearly in government, but without an explanation of terms, we’re left making assumptions as to what is business. Which is fine, because everyone knows what a business employee is, right? Right, just like everyone knows what an unemployed person is…

I’m not suggesting sneakiness, or worse, in any public institution in Australia or anywhere else. I am, however, suggesting vanity metrics are at least as prevalent in public policy as they are in the start-up sector about which Eric Ries was writing when he first brought the term to prominence. Vanity metrics is the term Mr Ries uses in ‘The Lean Start-up’ to refer to metrics that are easily manipulated and do not necessarily correlate to the numbers that really matter.

For example, how many of those “business” researchers are actually employed in organisations where the only operational money they have comes from sales and commercial investors ie people who believe enough in what is being done to put their own money in it? Or, more precisely, how many are employed in publicly funded research institutes (PFRI), where a substantial proportion of all funding comes from public sector commitment of public monies.

To be clear, I’m not saying the reliance on public funding in any way lessens the importance of the work being done at PFRI or increases the importance of work done with private funding. But nor is that why we measure the sectoral split of employment. We measure sectoral split because we want to know whether the insights and productivity improvements that come from innovation are getting taken up into our wider economy, and the most direct way for that to be achieved is if the people producing those insights and improvements are already in the wider economy. In business.

And if the “business” category includes PFRI, which are not ‘higher education’ because their role is not to educate, and not ‘government’ because they are independent entities with boards and executives even if they do report to Parliament and receive a budget allocation from government of between 80-95% of their operational and program budgets***, then we need to know that. Because, given our objective in taking this measurement, they amount to what an average person in the street considers a ‘business’ about as much as a person out of work for 7 years in the US is not unemployed.

Vanity metrics and creative definitions are not evil in themselves – as a former lawyer who spent quite a bit of time negotiating Key Performance Indicators in funding agreements for the public sector, believe me: I’ve had to consider it. What they do, however, is encourage us to think the problem is not as bad as we think it is. In fact, their entire reason for existence is to make us feel good about things, to feel as if something is being achieved and progress is being made. And it is: it just may not be towards the goal and at the speed one might have wanted.

And that is when one runs the risk of complacency. Of believing that everything is not as bad as we thought and worse, that there’s plenty of time to do something about it. That’s when the real dangers set in: when we lose our sense of urgency and slip into inaction. And I can’t help feeling we have.

To quote my own personal Superhero Australia’s Chief Scientist Professor Ian Chubb AO,

We’ve been talking about this for a while but as I get older, I’m getting less patient with inaction

*These figures are from the OECD’s Science, Technology and Industry Outlook 2014, which is available on-line through their iLibrary at

**For those interested in the creative industries, United Nations Conference on Trade, Aid and Development (UNCTAD) identifies 9 core sectors within the industry: audiovisual, performing arts, traditional, cultural sites, visual arts, publishing, design, services (architecture, advertising, creative R&D, cultural) and new media (software, video games, digitised creative content), of which YCF’s focus is only on the last three.
While the sector was hit hard by the Global Financial Crisis, according to Australian Government figures in 2008-09 creative industries contributed $31.1 billion to industry gross product and had an average growth rate of 3.9 per cent in real terms, faster than the broader economy over the ten years to 2008-09.

*** these percentages are for illustration only: totally made up and not based on any PFRI.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s