Launch HN: Prolific (YC S19) – Quickly find high-quality survey participants

A friend of mine runs into this periodically, with her response pools over-sampling on lower income and older individuals, with a lot of geographic skewing (she does local/regional research, so appropriate sampling at a census block and/or zip code level is needed).

I've worked with her several times to fill that gap, by leveraging Facebook ads for recruitment to balance out the deficiencies in her response pool. Ad cost + incentive tends to work out to the ~$10 range per qualified response. Which is too expensive as a recruitment method for her entire pool, but she's found fantastic to reach the otherwise unreachable demographic gaps she'd have. Different demographics respond better to certain incentive structures and ad copy than others, but in general it's been effective for capturing those hard-to-get groups.

Leveraging the same avenue for re-contact is also handy if your response pool is large enough to create a custom audience to target for follow-up ads. Even out of date contact info can be useful for this. Although your IRB (if relevant) may shut that down if you didn't account for it if the language in your initial contact didn't account for it, since doing this involves disclosing the subject's PII to a third party (Facebook) for creating the custom audience.

Our clients are about 80-90% academic, and we find that there’s a strong move within academia towards fair payments. In the long term we think that this will show in the data quality and reliability of samples such that paying fairly will result in the best value. Regarding cost, we're about $2 total for a 15min survey right now.

> How large is your pool?

Lack of transparency about pool size is one of the most frustrating things about online panels. This is one of the reasons why we don’t report our total pool size, but only our ‘active and accessible’ pool (participants who have been active within the past 90days). As a result, you can expect > 50% of the numbers of eligible participants we report to actually take part in the next 24/48hours. We have ~70,000 active participants (20,000+ in the US). I expect we would be able to get a sample of 5,000 people in <48hours, but would only be about to repeat this for about 10weeks with unique participants. Our ‘pool’, as measured by traditional panel providers is approaching 500,000, but we don’t think this is a useful metric as the majority no longer respond to invites.

> a) how you recruit high income people into your pool;

To be honest, we haven’t cracked this nut yet and we expect our pool to be unrepresentative for high income people also right now. Ideas we have to help address this problem are to 1) introduce charitable donations for those who aren’t motivated by cash incentives 2) improve non-financial incentives (e.g. feedback on the impact your data is having) and 3) have highly targeted invites, we hypothesize high earners would be willing to help out if we need someone of their demographics in particular, and if this was communicated well. If you have suggestions, we’re all ears!

> and b) what percentage of your pool would be high income (say HHI > $125k a year or so).

According to self report we have ~2,000 participants from a HHI (>$100k/year on our screener), though it’s possible this suffers from slight inflation and we don’t (yet) have a way to verify income levels.

> Finally, what is your pool attrition rate?

Our annual retention rate is ~40% or so (e.g. if a participant takes part in a study, they’re about ~40% likely to do so again 12months later). There’s a balance between ‘refreshing’ our pool and keeping high engagement, and we’re working on keeping “naivety” high while allowing for studies over a long period!

I can’t go into too much detail about the attention and engagement checks we have built in, but if you sign up as a participant and pay attention you might spot some. In the long run, we think good feedback systems and high trust will be key so that our pool iteratively gets better and participants don’t feel as incentivized to cheat. It’s key to make sure that participants feel that their high effort responses get fairly rewarded (both financially and with social appreciation).

[1] https://blog.prolific.co/how-to-improve-your-data-quality/

[2] https://blog.prolific.co/bots-and-data-quality-on-crowdsourc...

[3] https://researcher-help.prolific.co/hc/en-gb/categories/3600...

On a very basic level, the best way to validate your market & marketing ideas is talking to the right target audience. Often you can do this in your personal circles, but sometimes you need a less biased sample (in which case Prolific might be able to help). Good luck!

Right now we’re available to participants in OECD countries and it’s probably going to be another while before we can expand globally. Within the 36 countries that we’re in, we’re currently trying to work out what incentives might work best: How can we encourage the average citizen to sign up and take part in online research? What about very hard-to-reach demographics like professionals?

We feel that cash generally works well as an incentive, but it’s not everything. A lot of the time people want to contribute to projects they personally care about. So on Prolific’s end, it will come down to matching projects with the right participants. Another type of incentive could be to offer to participants that Prolific donates their earnings to charities on their behalf, and perhaps Prolific could match their donation?

We’ve recently launched quota-based representative samples for the US and UK [1], where we stratify based on sex, ethnicity, and age. A by-product of this feature is that it specifically invites niche demographics to participate, helping fill gaps. We hope that having more surveys available for demographics that are in the minority (e.g. certain ethnicities and age groups) will improve our ability to recruit these participants (although there’s a bit of a chicken-and-egg problem here).

Another thing we’d like to do is launch a mobile app for participants, so anyone can take a quick survey on the go (while waiting in line, chilling at home, commuting). Making our surveys more accessible to participants through different channels should help represent more people in society.

And then there’s user experience. We’re working on making our site as self-explanatory as possible, so anybody can sign up and start participating, even if you’re someone who doesn’t spend much time on the internet.

What do you think––what might be other ways to diversify our participant pool? Very curious

[1] https://researcher-help.prolific.co/hc/en-gb/articles/360019...

1) You can run follow up studies through our platform using anonymous identifiers we provide.

2) ~40%

3) We have many more filters when you create an account. The "Number of children" filter is actually a followup to a "Do you have children?". If you select "No" you'll see we have ~ 38,755 active participants who are childless.

4) Good question, I don't have good stats on this right now. The process of setting up a study is mostly touchless for us, especially for 'expert' users who have run online research elsewhere. Support mostly consists of billing issues, custom filtering, and complex studies (we sell to universities, so billing can be complex). We're working on improving onboarding and making the whole process as self-serve as possible.

In addition, SurveyMonkey audience uses a survey exchange provider CINT [1] and as a result you don't have any transparency of where exactly the participants are coming from, how they're vetted, the user experience they're having, and you can't communicate directly with participants or retarget them for longitudinal studies. All of which is important for research and are features that Prolific provide.

[1] https://www.cint.com/press-release/surveymonkey-audience-exp...

It would probably cost about $5-15 (it depends on how much you would pay the participant, I would recommend being generous for a study like this).

If you'd like to tell us a little more about your goals, we can get back to you with some more specific information: https://prolific2.typeform.com/to/aczPoI