How many users you need for UX research?
Deciding how many users you need for UX research is not easy. You want to balance speed and accuracy. The risk is to be either too slow or fast but inaccurate. Early-stage companies that do UX research have to juggle with very limited time and resources. So a good assessment is crucial to succeed.
The good news is that there are good practices you can follow to check how many users you need for UX research. This way you’ll eliminate the majority of guess work when calculating how many users you need for your tests.
The test you pick determines the number of users you need
The amount of users you need for UX research largely depends on what test you’re running.
The first step is to decide what you want to test and what is the best way to test it. The type of test you run determines the number of users you need to have significant results.
According to the Nielsen Norman group, these are the rules to decide the amount of users you need:
- For usability testing 5 people are enough to highlight 85% of the usability issues of your product. Higher-end projects would do with 8 participants, while lower-end projects should aim for at least 2 participants to test with.
- For quantitative studies 20 people are enough to have statistically significant numbers.
- For card sorting, a test on information architecture, you should aim for 15 participants per user group.
- For feedback on low fidelity prototypes 2 users are enough to get feedback on the design.
Qualitative tests are a great way to get high value quickly. You don’t need many users and you can collect valuable feedback fast.
Also according to the NN group, the tests with higher impact on business KPIs are qualitative. “The vast majority of your user research should be qualitative – that is, aimed at collecting insights to drive your design, not numbers to impress people in PowerPoint.”
How many users you need for A/B tests
For A/B tests it’s not only important to have significant numbers, but also a significant difference in result between the two variables. In other words, not only that more people click on A rather than B, but that the number of people who click on A is significantly higher than the ones who click on B. For this reason A/B tests need far more people to be conclusive.
The amount of users you need for A/B tests varies from a few hundreds to multiple thousands. The number depends on how many variants you’re testing with and what level of confidence you want from the test.
A/B tests are very used in conversion rate optimization (CRO). They tend to better suit the needs of companies with larger traffic volumes because they require a large number of users.
Also other test types used in CRO (e.g. elimination tests, multiple-variant tests) need large amounts of users. That’s why, to reduce the number of tests you need, it’s always a good idea to start with qualitative research.
Easy tools to calculate the users you need for A/B testing
To help you calculate the number of users you need for an A/B testing, you can use this calculation tool from Optimizely.
Here is how the tool works: the first thing you have to insert is your current conversion rate. You can find it on your current analytic tool (e.g. Google Analytics).
Then you insert the relative change in conversion that you find relevant. A value of 100% means that any change is relevant for the test, even only one person more or one person less. To be on the safe side, try a value between 70 and 80%.
Lastly, Optimizely has pre-filled the statistical significance of your test. This number tells you to what degree of confidence your test is a real positive. Pay attention to this number before modifying it. If e.g. 80% of confidence seems high to you, also think that its converse, i.e. 20% of false positives, means that there is one chance in five that the positive result is only determined by chance.
If you need a template to track your A/B test, you can download a free template here.
For how long should your test last?
The duration of a test is less relevant, as long as you get to the number of participants you need. Yet, you shouldn’t make your experiment too long, to avoid that the first results you collect are not representative anymore.
Also avoid testing too many things at the same time, since one experiment can influence the other. The result may be that you can’t use what you’ve spent time testing.
How to do research when you have a few users
What if your website has a very low amount of traffic? Can you still test? If your website still has little traffic, you can still test but you need to be realistic and pick the tests that deliver the highest value.
In this case, it’s even more important that you pick tests with a high impact on conversion and engagement. Focus on qualitative insights, which are more likely to deliver game-changing results.
Do you want to know more on how to use data for decision-making? You can find more on this post.