Media Measurement is Broken

No matter what method is currently being used, media measurement is broken.

Personally, I’m a big fan of primary research methods... the old way of doing it. Sample, validity, probability, regression, trend, cluster, segmentation... and the miracle of the normal curve - but these are far too slow in the modern digital world, with expectations for real-time data being the norm. And why not! Quite right too! And you know... one self selecting random sample is much like another statistically speaking.

The difficulty is the lack of mixed method. It takes a bit more nous to handle these, but, with the derth of valid digital interpretations of the real world at present, it seems no matter how good the single method approaches for media measurement, they’re just not working.

It’s hard to have this chat without naming names, and I absolutely concede that providers are doing their best within the limits of their methods, but things are being missed. Huge chunks of behaviour are knowingly omitted for want of a mixed methodological mechanic.

In the US, for example, Sandvine figures show that 20% of all downstream internet traffic from fixed access (non-mobile) locations during peak periods is from Netflix.

But Comscore’s recent figures miss this. That’s a huge chunk of an audience – missing. (We used to cry about losing one record!)

The reason for the disparity is simple. Different methodologies. Sandvine looks at data networks and so captures mobile access points, as well as fixed ones. It’s network centric. So, they capture iPhones, PS3s, as well as desktop PCs and Macs. Comscore has a panel that monitors web use on specific machines – so its panel centric. And I’m told, it doesn’t count Macs (have to check this though as it seems incredible).

With the growth of iPhones, iPads, WiFi connectivity, Web over TV and a TV over Web, this gap really matters. I’m not thinking into the future here either. The old methods are missing usage today, and the impact of these problems for advertisers and media in general is huge.

Let’s take Irish radio as a local example. It’s collected with a very large and rolling sample with four releases of data per year (second only to the CSO). Those who represent the medium like this because a rolling sample gives no huge peaks and troughs. No rich-today poor-tomorrow stuff. No surprises. Not very webby though and it plasters over important changes and truths. But here today, gone tomorrow does happen. We who know the web are aware that traffic is in fact capricious. Look at Bebo! Why should it be any different for radio programs, or stations? Well, it isn’t. It’s just that the measurement is slower and uses 75% of the last wave, with the new wave. This softens the curve considerably. The findings when presented as the facts for today are in fact only 25% of the facts for the recent past, and 25% from a while ago, another 25% from age’s ago, and a final 25% from when God was a child. It’s like saying our economy is fine... when averaged out with the height of the Celtic Tiger. Dangerously misleading, and if presented as today’s news – wrong!

It means, for example, that despite the death of the very popular Gerry Ryan a long time back, the audience he won is in fact still being counted and attributed to someone else in his spot. So, many thousands of advertising euros have been spent for an audience that just isn’t there anymore. That’s just not right. Is it! It’s wrong.

And... what really bothers me, is the lack of digital. Digital radio is omitted as the current method is not reaching the at-work mobile radio user. Why? They aren’t home on a door-to-door sample. Are they! The tremendous popularity of digital audio consumption – and I don’t mean DAB, is being largely ignored, and in doing so ever valuable advertising euros are being lost.

Video is another problem. It is measured almost wholly for TV sets in Ireland, though it includes Web based TV in other countries. How often have you or someone you know viewed video over the net, on YouTube, RTE player or another TV over Web format. A smartphone for example. Once? Twice? Well, that isn’t being counted. It could be you, or maybe some people you know very well. 1 out of 10 maybe you know? Do a count... That could be a considerable percentage of all adults, and it’s not being counted. 10% of people are probably being lost, and what they are doing via their smart device is also being lost.

There was an item in The Times on analogue switch off for TV only the other day. Digital media communication is becoming pervasive, like it or not. But, more importantly, digital is multi-platform. It is not like TV, or Radio, or Print. With digital, it doesn’t matter, while with current research methods - it does. And these methods will not and are not coping with the modern digital media world. They were not designed to do so.

And bigifying the sample won’t help either. It’s the method that’s the problem. Recently, for example, Nielsen in the US increased their TV sample from 8700 to 25000. Big numbers. But, even with this huge count they did not detect any activity from non-network TV sources. That is... no iTunes, no Amazon, no YouTube... etc, etc. I read that as – no up-to-date media consumption. Old school. Stone age. Rubbish. We know its rubbish! And if it’s professing to be true, or right, or definitive which it is - it’s just wrong.

They are viewing a digital world through analogue glasses.

Needless to say, I’ve thought about this a good bit. And, it really annoys me that these issues aren’t being addressed, as not doing so is costing practically every medium consumed by Irish people a fortune in lost revenue. The audience... the true audience, for almost every medium I can think of is in fact much, much bigger... and the audience doesn’t give a damn about the device being used. Do they? Nope.

An interesting bit.ly link hits me via twitter. I click blind to a newspaper article. Wow! That’s interesting. Did I buy a newspaper? No. Did I read one? Yes. Did I see the ad at the top of the page? Yes! Am I up to date? Yes. How often does this happen every day... on twitter alone. 20,000 times? So, perhaps 20,000 newspaper reads – per day, missing. Across all sorts of titles I’d never buy. Thats 600,000 reads a month. 7.2 million reads a year. What’s that worth in Ad-land? Anyone? I dread to think.

And also, needless to say, there are solutions. Lots of good, doable, realistic, affordable solutions. What cannot be fixed on the other hand is the terrified and protective research ideology that pervades most media committees and organisations.

But there is a need to try some new things... And soon, so that little by little we can deploy new mixed method research approaches which I know will work sufficiently well to increase media consumption counts massively.

These will achieve:
• • Real time reporting
• • Platform and operating system agnostic audience measurement
• • Fixed and wireless (mobile and stationary) measures
• • Aggregated audience counts

And these steps need to be taken now, before mega-corps like Google, Apple and Microsoft become too dominant and become the de-facto auditors of measurement and message truth, in pretty much the same way Google is the de facto truth for search engine visibility. That took about three years. The impact of waiting for that to happen is that ad euros will leave the country... moving into other networks, with bidding platforms and prices commanded by the audience, not the publisher.

It also needs to happen before too much advertising money is lost by lost audience. While more real time measures may be jumpy, they can be averaged over time. But ignoring the digital for fear of its capricious nature ignores huge tranches of the audience of today, and any independent view on the audience of tomorrow.

In Pavilion Digital, we recently trialled some new methods, addressing a diverse set of data sources. Facebook outputs, iPhone app outputs, primary research, website posts, newspaper sales and classified ads, mobile research datasets... funky stuff! And I have to say, it was fun. Real Rubix cube territory, but what it brought home, from the outset, is that traffic, eyeballs and commerce leaves a digital footprint, and once you get your head around that, bringing all that data together is a challenge, but a hugely beneficial and rewarding one. And, with a single medium for example – you only have to get your data modelling right once... maybe tweak it a little, and if you’re good to go, you’re good for good... until another channel for audience consumption opens up. This is a good problem to have if you have it. Good for us and for the medium being reached.

Currently TV, Print and Radio are treated as silos of data which cannot, or are not counted together because they represented different industry interests and their respective ommittees. But nowadays, with it all being digital, they can represent the same interest, regardless of the platform of delivery. In the end of the day, data is just data. It doesn’t care where it’s from, or what it represents, or where it is delivered.

Take RTE. What is their media interest would you say? Well, perhaps TV, analogue Radio, DAB radio (RTE Choice is brilliant!), digital Radio (Grab Radio streams), Web Radio, Print (on website), iPhone apps (4 or 5?), TV over Web (streamed), RTE player (recent TV re-broadcasts), Website traffic, Blog reads and postings... etc, etc, etc. I’m sure I’ve forgotten plenty... (forgot Aertel). So, one interest – RTE, requires platform agnostic, OS agnostic, fixed and mobile digital delivery and consumption, and digital interaction. What are they counting? TV set viewers and mostly analogue radio listeners... and maybe website traffic. Oh, and the information is out of date.

This is a ridiculous state of affairs and as a licence fee payer funding our national public service broadcaster – I find it objectionably incompetent at best and a regrettable lost opportunity.

Really and in real time it is time for media measurement and the interests it represents to wake up. There’s an audience shift alarm clock going off loud and clear. And no, it’s not a windup clock - it’s digital.