You need to put together a winning research proposal. You're up against large, well-resourced agencies.

You must have something which makes you stand out.

​

Knowledge

 

You'll be one step ahead of your competitors if you can anticipate some key issues with online sample usage and demonstrate practical solutions. A little time questioning assumptions and the “we've always done it this way” attitude will give your proposal more chance of success, and will in all probability save costs (not to mention the time you don't have to spend fixing things when they go wrong).

Here's what you need to know about buying and using online sample. The focus is not on how to choose an online panel partner, but practical steps to handle the challenges posed by the feasibility, proposal and project processes.

The success of your project rests on you knowing the tricks of the online sampling trade, so let's have a look at what you might encounter - and what actions you can take.

 

​
 

1) How will respondents be targeted?

​

A nationally representative sample is based on a combination of demographic quotas. That's fine, and any online panel worth their salt will have this information profiled. What about if you want fizzy lemonade drinkers? That's probably a bit too defined for panel profiling, but “regular drinkers of carbonated soft drinks” is a decent start. There's no point in asking people who you know don't drink fizzy drinks if they drink fizzy lemonade. It's a simplification, but if the gen pop incidence is 25% for lemonade drinkers, and 60% of the population drink fizzy drinks, your incidence rate is going to be around 40% by targeting fizzy drinks drinkers from the start.

Upshot: Know what targeting/profiling is available, and test your panel contact's knowledge of how their existing profiling can help you. If there's a proxy profile to get closer to your target audience, use it. Check with various panels - you'd be surprised at the differences in what they do and don't capture.

 

​

2) When there's no available profiling

​

In the absence of any relevant targeting, some panels will offer an incidence check or mini-poll. These are slightly different (& each supplier will use their own terminology).

An incidence check will (usually) send a single closed question to a gen pop sample of anywhere between 200 & 1000 respondents. The info you get back tells you what % of the sample answered the question one way or another. You need to make sure you get back a respondent-level datafile including demographics (minimum of age & gender) - not just a summary.

 

Why? Because there will be demographic differences and these incidence checks don't tend to use demographic quotas. In other words, first-come, first-served until the target count is hit. Even if it's a representative outgo it usually means older women will be over-represented and younger men under-represented (young men respond slower to survey invites), so you should take the raw data and have it weighted to nat rep proportions.
 

Some panels provide incidence checks free of charge, others charge only if you want more than their standard respondent count, others charge a (refundable) fee ahead of commissioning a project, others always charge. It's a mixed bag.

A mini-poll tends to be an uprated version of the incidence check. It might include more questions (useful if your screening criteria are more complex) and might go out to a large proportion of the panel. Some panels use these to populate their profiling with new information, so it's a resource that can be tapped into again. What can also happen is that invitations to your new survey can go out only to people who answered the mini-poll in a certain way, meaning your incidence rate is now going to be high. You can usually expect to pay for a mini-poll but, used correctly, your overall project costs will come down.

Pre-screening is another version of a mini-poll which sits directly in front of your survey when it goes live. The aim is to filter out non-qualifiers before they hit your survey and redirect them to another one they are likely to qualify for instead. This can help both you and your panel supplier. You benefit from high incidence on a naturally low incidence study, and they maintain panelist satisfaction by not screening out lots of potential participants. There are some downsides (for example, panelists can be pre-screened numerous times before they qualify for a survey, and you don't want fatigued ones entering yours).

Upshot: If you don't know what incidence to expect, try asking for an IR check or (if more complex) a mini-poll. Ask for fully-detailed output. A mini-poll can often pay for itself - and more - if you only invite respondents to your study who have “qualified” in the mini-poll. If pre-screening is suggested, find out exactly what that entails and how it might impact the respondent mix for your study.

 

​
 

3) What are you being charged for? And why?

​

Here's a real-life example - I needed whisky drinkers for a survey, and I knew these accounted for 20% of adults in a particular country. I also knew that around 70% of whisky drinkers would qualify for the survey. So, if the panel could target whisky drinkers, the baseline incidence would be 70%. So I asked for 2 costs. Targeted whisky drinkers at 70% IR, or 14% IR among gen pop. The answer back was that, yes, they could target whisky drinkers, and their gen pop cost at 14% IR was $x.
What? You can target, but you're going to charge on the basis that you can't? It's not the most compelling pricing model I've come across and there isn't any obvious client benefit to it. Beware of this one.

Another example is premium pricing for specific target groups. Some are understandable, since they are in short supply, or difficult to engage, and need extra incentivization. Usually, this is on the B2B side of things, but can also include high net worths & healthcare targets. Mums are a “premium” target for some panels. I'm not convinced they should be - it's not exactly as if they're in short supply, and I can guarantee that they are not being offered a premium incentivization for the vast majority of surveys they take. Given that, what's the rationale?

Be aware also of another scenario, the “partially-profiled” panel. This is where x% of the panel has been profiled, so they offer a certain number of respondents at one cost, and the remainder at another (higher, unprofiled) cost. You might be better off combining 2 panels to ensure all respondents are profiled (and to build in a little back-up...more on that later).
My position was always that if we had some respondents profiled, we would charge as if all were profiled. It wasn't the client's fault we had only partially profiled the target, and anyway we'd be filling some of the missing profiling from the project data, so our Ops team could (at least in principle) update it. It very much depends on the person you speak to whether that approach is applied.

Upshot: Check the panel assumptions and pricing model - you could end up paying a lot more than you need. If one panel's offer doesn't make sense, and they don't budge, move on. There are plenty of good respondent sources out there.

 

​
 

4) A quick sanity check for sample feasibility

​

All feasibilities make a series of assumptions, and all are tied back to the total panel count. If you have feasibility confirmed for 1000 men aged 30-55 @ 20% IR, you will need 5000 survey starters. If the panel has a total of 20000 men aged 30-55, the lowest response rate (RR) to deliver those starters is 25%.

You need to think about what the lowest RR could be, and make back-up plans. Men, generally, are both less likely and slower to respond than women, so be prepared to cut your expectations. A rule of thumb I use is to assume a gen pop RR between 5% and 10% (or take half the declared RR) and chase that back to panel counts. It's better to engage with two suppliers and share the load than risk it all on one and have them struggle. Also, bringing in another supplier for emergency “top up” is stressful and potentially expensive if you only need a few extra and minimum fees are involved.

Upshot: If the numbers look tight, they are too tight. You don't need the stress & extra cost of unexpectedly repairing something that shouldn't have failed, so have a back-up plan or (better) give yourself extra coverage by sharing the load from the start between 2 or more vendors.
 

​

​

5) A special mention about Tracker feasibilities

​

All panels lose active panelists, but they might not be very upfront on the numbers involved. One proxy for this is to ask how many new panelists they recruit each year (most panels maintain stable counts, balancing lost panelists with new ones).
 

Your starting point is how many interviews a panel could support for a given spec in one go. Divide that by the number of interviews you need for a single wave, and you have a simple yardstick for the number of waves you can safely do. If it falls short, you can still look at the panel churn to see how many “fresh” respondents will be available to you by the time you get to that point.
 

Eg, you have a monthly tracker planned, at 500 interviews/month for a year. Your initial feasibility tells you you can get 3000 interviews, so that's equivalent to 6 months. If the panel loses/replaces 5% of panelists each month, 30% of the panel will be “new” after 6 months which means you'd get another 1-2 months of interviews. On top of that, your recontact rules might allow you to invite participants again after 3 or 6 months which would give you further flexibility.

Overall, you're probably looking at 8 months max so it'll require another respondent source to deliver the full year, and if you split the load 50/50 from month 1 you'll build in extra coverage just in case.

Upshot: Make sure you take panel churn into account when calculating tracker feasibility. It renews the panel base and continually provides new respondents. There are lots of assumptions, though, so be cautious and try to combine 2+ sources from the start on lengthy tracking projects.
 

​
 

6) Nat Rep starting samples - why and how?

​

Sometimes (in fact, surprisingly often) it's better to set quotas on survey starters than completers. Why? A balanced sample of survey starters is useful when the demographic make-up of a product or service is unknown, or is known to differ significantly from the wider population. By referencing qualified respondents back to a representative starting sample we can understand how these differ from the wider population, and “size” the market or opportunity.
If you set nat rep quotas on completers, you may find it difficult to deliver on certain demographics because those are less likely to qualify on your survey criteria. If that's the case, you're forcing the data to nat rep proportions when its natural distribution is quite different.

Delivering a nat rep starting sample is not particularly difficult if you know how, but neither is it as straightforward as it sounds. First, no survey platform sets quotas on starters - these are always on completers and, in the absence of a solution to that, the absolute best a panel can offer is a guess on the demographic make-up of starters. It's not enough to assume a representative outgo will do the job - that's influenced by response rates, so some demographics will be under/over represented. Even if you attempt to mitigate this by weighting of the invite counts, it's still without direct platform controls.

How do you get round this?
By turning starters (or, more precisely, screeners) into completers! This makes things interesting. All of the screeners have to count as completers to trigger quota controls & management. Therefore, 300 respondents for a 15 minute study at 20% incidence becomes both a 2 minute study at 100% incidence for the 1200 “screeners” and a 15 minute study at 100% incidence for the 300 “completers”.
Respondents can be incentivized differently depending on whether they only qualify for the 2 minute version or the full version. They are all captured as completers, so everyone qualifies (within the nat rep quotas which are set).

What do you get out of this?
We now have balanced demographics for survey starters. In other words, those who start the survey accurately represent the wider population. Among these, we can analyse the demographics of both buyers and non-buyers separately, by filtering on the “long” and “short” hidden variables which we generate in the survey script respectively.

This delivers the natural demographic make-up of each group, whilst the combination of the two groups delivers a nat rep sample.

Upshot: Be very clear on this with both your scripting and sample teams. You need your scripting team to change mindset and not think of non-qualifiers as “screeners”. They are short survey completers. On the sample side of things, if everyone qualifies the short survey completer cost could be balanced by a lower cost for long survey completers (100% vs 20% IR).

 

​

7) The concept of a blended incidence - and how to get it right

​

You can read a more in-depth article on this here but, as an example, if we need 200 main buyers each of 3 brands, and those 3 brands all have an IR of 15% among an invited target, what does the overall sample spec look like?

Is it 600 @ 15%? No, that's averaging the IRs and assumes we sample for Brand A and screen everyone who doesn't buy it. Then we sample for Brand B and screen everyone who doesn't buy it. Then we do the same for Brand C. If we did that, then 600 completes at 15% incidence would indeed be the result, but that's highly wasteful and not what happens in real life.

In real life we check if a respondent qualifies for A, B or C. If not A, then maybe B or C. If not B then maybe A or C. If not C, then maybe A or B. If none of them, we screen.
How do the numbers fall out on this basis? Well, you'd get 600 interviews at 35% incidence. This is what I term an Accumulative Blended Incidence and it makes quite a difference.

It gets a bit more complicated when you have a main sample (eg category buyers) and you need a certain number of specific brand buyers which won't fall out naturally within that main sample. In other words, you need to boost the sample to get those brand buyers. How many will fall out naturally, how many boosts do you need, and what's the blended incidence of that boost?
I tested this out with 10 online panel providers and not one got it right - that's going to cost you money if you're not on top of it.

Upshot: Be very careful when requesting boost samples or working to specific product quotas. Don't rely on your panel contact to get it right or understand the difference between an average & blended IR. Work it out beforehand with our calculator!
 

​

 

8) Dictate the sampling plan

​

One of the most common complaints I hear is that of missing standard demographic quota targets. All the simple ones are filled and, naturally, you're left scrabbling to fill unintended nested quotas for low incidence groups which are never going to appear because you've now run out of sample and time.

 

Solution? Relax quotas (but it's a cop-out and you shouldn't need to get to that situation).
 

The real solution is to look at your quota targets and break them down from hardest to achieve to easiest to achieve. We know that men are harder to reach than women, and that younger ages are harder to reach than older ones (over 60s respond really quickly!)

If we've got quotas on age/sex/region we will end up at the end of the fieldwork needing to fill specific combinations. Usually, that's something hellish like men aged 18-24 in rural County Nowhere with precious little sample available at the start who were mainly older women.
 

What you should do is spend the first day or so concentrating solely on the young men. Let them fill up naturally across the regions. Then do the same for the young women. Once you've covered off these younger ages, open up to older ages of both sexes and your remaining nested open quotas will be much easier to fill.

Upshot: Controlling the sampling plan is the single biggest thing you can do to deliver an outstanding project. Make sure the panel follows your plan by setting specific limits - by day if you need to - and be aware of potential pitfalls before they appear. Decouple quota management from sample supply. You don't want the tail to be wagging the dog.

 

​
 

9) Don't (always) believe the in-field incidence rate

​

First, you need to define what goes into your IR. Generally speaking, the IR is calculated as Completes divided by Starters. You have 1000 starters, 300 are screened, 700 complete. The IR is 70%.
What about those who are screened because they're over-quota? Well, are they over-quota on something targetable, such as demographics? If so, they're not valid screeners because they are preventable starters. The same thing applies for straight-out screeners who are targetable. They need to come out of the calculation both in terms of starters and screeners.
The IR then becomes Completes divided by (Starters minus Preventable Starters).

We ran a study about petfood. The panel we chose was able to target on pet owners and we saw that the IR was in line with expectations. Great - except that there were a lot of screenouts for not having a pet. Quite simply, the panel was not targeting pet owners...but the IR was in line. A chat with the panel revealed that they were quite content with the IR and saw it as valid because it matched what we had told them to expect. In fact, by removing the preventable screeners the IR was significantly higher than we had expected, and we secured a cost reduction for our client. It was an avoidable inefficiency by the panel which led many people to be unnecessarily invited to a study for which they wouldn't qualify.

It's advisable to run a soft-launch for every study you run. This helps highlight any issues with a small number of respondents, rather than have to deal with a massive problem involving 100s or 1000s of respondents.

Upshot: Don't assume your panel is targeting correctly - check where screenouts are coming from. You need to go into the fieldwork metrics and see how survey starters fall out into Completers, Preventable Starters and Valid Screens/Over-Quotas.


 

10) Recruiting To Follow-Ups and other research

​

Although I've concentrated on the online survey environment, respondents can be recruited to all sorts of other types of research. The “danger”, from a panel perspective, is that they lose control of their panelists, so they might not want their them to provide contact information for follow-ups. In some cases they're scared you'll “steal” those respondents. I've even seen the term “poaching fee” being used in a client proposal involving a follow-up! I'm not certain it was well-received.

The reality is that people don't just want to fill out surveys. Trying a product out, doing a mystery shop, participating in an online forum or recording a video diary appeals to many and it should be down to the respondent to decide if they want to participate or not.

It comes down to the culture within each panel that ultimately determines whether they are a good partner or not for research that goes beyond online surveys. I've dealt with panels who have immediately escalated things to their legal team, some who have flat-out said no, others who will allow it for a premium and a few who are incredibly relaxed as long as we ask for consent & contact information, handle invitations & deliver incentives ourselves (ie take over the participant relationship for that non-survey research).

Upshot: Follow-up recruitment occurs all the time but your preferred panel partner might not necessarily be a good fit to carry it out. There are legitimate reasons why a certain approach may not be feasible or appropriate, so work with them to determine whether their boundaries are really internal (not comfortable with it) or external (legality).

 

​

Bonus - When it goes wrong

​

I had a survey scripted, a sample source in place and was ready to start testing for soft launch. Then I got an email from my contact about the survey content (it was asking respondents to rate a series of statements about a long-standing conflict on an agree/disagree scale). After a few more hours another email came along informing me that, unfortunately, the survey would be inappropriate for their respondents as it was highly sensitive and they would not now be able to assist...but good luck with it in any case and hope it all goes well! (Gee, thanks!)
 

Luckily, I had a back-up source. We were testing with them within an hour, and we went into field the same day - on time, on budget and no disruption to our client. As it turned out, there was an open question that asked the respondents what they thought of the study and no-one said it was inappropriate. In fact, a number commented that it was a welcome change to be challenged like that and was much more interesting than the surveys they usually did.

Upshot: Always have a Plan B and don't be afraid to pose challenging subjects. Some panels can be over-protective on certain subjects (often for fear of offending) but respondents are adults, not snowflakes, so use common sense.