fbpx

How many users do you need for your UX tests?

When running a UX test you want to balance speed and accuracyThe risk is to be either too slow or fast but inaccurate. For early-stage companies, which have to juggle with very limited time and resources, a sound assessment is crucial.

The good news is that there are good practices you can follow to decide how many users you need. This way you’ll eliminate the majority of guess work.

Here we present how many users you need for different types of tests and how to prioritize in case you’re short on time, users or both.

 

 

The test you pick determines the number of users you need

How many users you need for a UX test largely depends on what test you’re running.

The first step is to decide what you want to test and what is the best way to test it. The type of test you run determines the number of users you need to have significant results.

According to the Nielsen Norman group, these are the general rules you can apply:

  • For usability testing 5 people are enough to highlight 85% of the usability issues of your product. Higher-end projects would do with 8 participants, while lower-end projects should aim for at least 2 participants to test with.
  • For quantitative studies 20 people are enough to have statistically significant numbers.
  • For card sorting, a test on information architecture, you should aim for 15 participants per user group.
  • For feedback on low fidelity prototypes 2 users are enough to get feedback on the design.

Qualitative tests are a great way to get high value quickly. You don’t need many users and you can collect valuable feedback fast.

Also according to the NN group, the tests with higher impact on business KPIs are qualitative. “The vast majority of your user research should be qualitative – that is, aimed at collecting insights to drive your design, not numbers to impress people in PowerPoint.”

 

A different case for A/B tests and CRO

For A/B tests it’s not only important to have significant numbers, but also a significant difference in result between the two variables. In other words, not only that more people click on A rather than B, but that the number of people who click on A is significantly higher than the ones who click on B. For this reason A/B tests need far more people to be conclusive.

The amount of users you need for A/B tests varies from a few hundreds to multiple thousands. The number depends on how many variants you’re testing with and what level of confidence you want from the test.

A/B tests are very used in conversion rate optimization (CRO). They tend to better suit the needs of companies with larger traffic volumes because they require a large number of users.

Also other test types used in CRO (e.g. elimination tests, multiple-variant tests) need large amounts of users. That’s why, to reduce the number of tests you need, it’s always a good idea to start with qualitative research.

 

How to calculate the users you need for A/B testing

To help you calculate the number of users you need for an A/B testing, you can use this calculation tool from Optimizely.

Here is how the tool works: the first thing you have to insert is your current conversion rate. You can find it on your current analytic tool (e.g. Google Analytics).

Then you insert the relative change in conversion that you find relevant. A value of 100% means that any change is relevant for the test, even only one person more or one person less. To be on the safe side, try a value between 70 and 80%.

Lastly, Optimizely has pre-filled the statistical significance of your test. This number tells you to what degree of confidence your test is a real positive. Pay attention to this number before modifying it. If e.g. 80% of confidence seems high to you, also think that its converse, i.e. 20% of false positives, means that there is one chance in five that the positive result is only determined by chance.

If you need a template to track your A/B test, you can download a free template here.

 

How long should your test be?

The duration of a test is less relevant, as long as you get to the number of participants you need. Yet, you shouldn’t make your experiment too long, to avoid that the first results you collect are not representative anymore.

Also avoid testing too many things at the same time, since one experiment can influence the other. The result may be that you can’t use what you’ve spent time testing.

 

How to do UX tests when your website has few users

What if your website has a very low amount of traffic? Can you still test? If your website still has little traffic, you can still test but you need to be realistic and pick the tests that deliver the highest value.

In this case, it’s even more important that you pick tests with a high impact on conversion and engagement. Focus on qualitative insights, which are more likely to deliver game-changing results.

Do you want to receive advice on how to use UX tests to improve your product? Feel free to get in contact with us here.

 

Do you want to know more on how to use data for decision-making? You can find more on this post.

Never miss an update!

The best UX updates delivered straight to your inbox

Looking for more personalized help?

Get in touch with us today to book a free consultation call!

Never miss an update

Leave your email address to receive the best updates on UX, growth hack and digital strategy.