Representative Samples - at the start or at the end of a survey?

​

How many research projects do you work on where you've been asked to deliver a nat rep sample? Our guess is a fair proportion.

And is that among survey completers? Usually, yes.

​

However, there are plenty of times when it would be more appropriate to have a representative sample of survey starters rather than completers. In fact, it can provide much more actionable information, allowing you to better understand and quantify your target audience.

​

How so?

​

If we want to interview ground coffee buyers, we'll want to understand who they are, and how they differ from the adult population at large. If those who complete the survey (ie ground coffee buyers) are represented to census proportions, we don't get that information. 

 

Switch to a nat rep sample of survey starters, and we will know how the profile of category buyers differs from the wider population. We can size the market and the opportunities. If survey qualifiers (ground coffee buyers) are allowed to complete without further demographic quotas applied to them, we'll also get a naturally representative sample of this group.

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Things to watch out for

​

Going down this route does present some obstacles, namely in terms of understanding what's required, and applying the correct processes.

​

Let's take understanding first - if we are looking for a nat rep starting sample, the overwhelming understanding among sample providers is that they should target survey invites to a nat rep plan (eg x% to 18-24s, y% to 25-34s and so on).

​

What's wrong with that? Well, the main problem is that different groups respond to invites to greater or lesser degrees. The sample vendor will think their job is done - they sent out invites strictly in line with nat rep quotas. The researcher will look at the starting sample and see that it isn't nationally representative (where are all the young men we expected to see, and why do we have so many older women?).

 

A solution is to be a little smarter and take typical response rates into account, so proportionally more invites are sent to younger men, and fewer to older women. That might work, but you still don't have any guarantees that it'll fall out as planned. For that, you need to set quotas on the starting participants. 

​

This is where the process becomes important. Survey platforms apply quotas to survey completers, not starters, so to overcome this we need to convert all starters to completers - in other words, respondents are not screened out, but simply answer the demographic & screening questions before either completing their "shorter" survey or continuing into the "longer" survey.

​

We therefore have a survey in which everyone qualifies, whether they buy ground coffee or not. Simply that some qualify for a shorter survey and others for a longer one. The quotas track completers, but it's a combination of those who don't buy ground coffee and those who do...which, in reality, means survey starters.

​

Cost considerations

​

Let's assume we want 1000 completers of a 10 min survey, and the expected incidence of ground coffee buyers among a nat rep starting sample is 35%. We are going to need close to 3000 starters to deliver those 1000 buyers.

Rather than a cost based on 1000 @ 35% IR, we will actually have 3000 @ 100% IR. Of these, about 2000 will do a 2 min survey, and 1000 the full 10 mins. How does this stack up?

​

​

​

​

​

​

​

​

​

​

​

We used pricing from a well-known sample provider to crunch the figures. In this case, the sample cost in scenario 2 is about 20% higher than scenario 1, and that gives you both the market sizing information from the nat rep starters and a naturally representative sample of category buyers. We think that's pretty good value for money.

​

A quick recap

​

• Nationally representative samples can be specified for survey completers or survey starters.

​​

• If you ask for nat rep completers, make sure the screening qs don't distort the demographics of qualifiers from nat rep (census) proportions.

​​

• Nat rep starters are useful when you need to size an opportunity or are unsure of the demographic make-up of your target.

​​

• “Nat rep starters” is not the same as “Nat rep invites”.

​​

• Quota controls for survey starters actually means survey completers with different LOIs, but all at 100% incidence (all starters qualify for either a shorter or longer survey).

​​

• The cost impact of turning screenouts to completers isn't as big as you might think.

​

​

Next Steps

​

1. Make a conscious decision about the representativeness you require for each project, and whether "starting" or "completing" quotas are most appropriate.

2. Check with your sample supplier that they understand what is expected, and that they have the necessary experience to deliver.

3. Ensure your survey platform can handle the specified quotas, and that your survey scripters know how to manipulate the background logic & routing to make it happen.

4. If you're unsure whether your partners and processes can seamlessly deliver, have a word with us. We'll use the most effective combination of sample and research platform for your specific need and you'll have an expert team on hand throughout the project process.