The impetus for the media industry’s new JIC (joint industry committee) is at least partly due to Nielsen’s inability to accurately measure TV viewing during the pandemic. For reasons that seem obvious, Nielsen couldn’t keep sample homes in-tab and the decline in sample caused a reduction in TV viewing estimates.

The problem was related to sample quality. At the same time, the JIC wants to open its certification to companies that are using less rigorous sample methodologies.

The data that comes in from TV distributors and manufacturers that comprise the basis for these companies’ measurement are neither a census nor a random sample. Moreover, the companies have yet to even apply for accreditation to the Media Rating Council (MRC), whose  requirements which are tough even for Nielsen.

When you think about it, these facts don’t align.

If the “alt” companies’ -- which is now what they’re starting to be called -- samples aren’t random or a census, then what are they? If they’re not random or a census, they’re not representative of a population.

Their data samples are what some researchers might refer to as “convenience samples.” You get the name comes from the fact that their data is conveniently available.

The researchers don’t need to create a sample frame, call or visit lots of homes to ask people to participate or send out a lot of surveys. They simply collect it from TV set-top boxes or smart TVs.

There is no creation of a new sample frame with the data from each new census, no random choosing of census block groups or of homes on a block group. There are no rules that must be followed to use another home on the same block if a home drops out. You simply get what you get.

This is not to say convenience samples have no value. The sample sizes are enormous. They are fantastic for certain types of research, just not for traditional TV measurement. At least not as standalone services unsupported by panels.

Back in the day when ErinMedia and Nielsen were attempting to make TV ratings out of newly available TV set-top box data, there was another startup that didn’t believe the data was suitable for TV measurement. It instead believed that the best use of the data was for TV ad targeting and media return on investment research. The name of that company was TRA, which was acquired by TV and is now called TiVo Research.

If you’re not familiar with the company, TRA was started by the legendary Bill Harvey and his co-founder, Mark Lieberman.

Bill was a long-time Arbitron executive. He is often credited with having invented the area of dominant influence or ADI. He was the first winner of the Advertising Research Foundation’s (ARF’s) Erwin Ephron Demystification Award. The ARF’s highest honor. He beat out the legendary David Poltrack, CBS’ then Chief Research Officer, who won it the next year. It wouldn’t be a leap to say that he is the father of TV set-top box ad targeting. There is probably not a more respected researcher in all of media research.

This isn’t to bash the “alt” research companies, or for that matter, the use of convenience samples. Convenience samples are phenomenal for ad targeting. For this use, the data doesn’t need to be perfect. What you need is lots of highly granular data that enables the advertiser to send ads to homes with people who are most likely to respond to the advertising. You’re not measuring anything.

They’re also amazing for gauging return-on-media-investment. As TRA and TiVo Research taught us with their product when it first landed on the scene, you could double blind match large TV set-top box databases with purchase data from loyalty card programs and know exactly which homes were exposed to ads and which homes bought the product. This was and is done with no modeling. In research, it’s the Holy Grail – it’s as good as it gets.

The issue is that the JIC isn’t looking to use the “alt” research companies’ data in this way. It’s positioning it as a Nielsen replacement for currency measurement.

Unless the industry is planning to change the basis for how it transacts business from cost-per-thousand (CPM) to some other metric based on conversion, convenience samples by themselves are not a good replacement for a proper, randomly chosen, probability sample.