A/B Split Testing Done Right

A/B split testing is a popular method that digital marketers use to compare different versions of ad campaigns, offers, or sales pages to determine which performs better. A/B testing lets you make data-driven decisions about what works best and is crucial for any successful marketing campaign.

Key to this is understanding whether or not the result is statistically significant. Is the improvement the result of the change or happenstance?

This post will touch on setting up a valid test, explain how to measure the statistical significance of the results (Click the link to jump to the calculator), and discuss why it’s important.

Don’t Leave Empty Handed!

Get Digital Tips & Tricks Delivered to Your Inbox

  • This field is for validation purposes and should be left unchanged.

We will never share your email address.

Setting Up a Test

As with most things, a successful outcome depends on planning. It’s essential to clearly understand what you are trying to understand with the test and, therefore, which metrics will be measured. 

This starts with identifying the goal of your test. What is the purpose of running this test? Be sure to test big things. Trying to increase conversions or click-through rate is a big thing. Blue versus green isn’t. 

Once you have identified the goal, decide which metric will be used to measure success. This could be conversion rate, click-through rate, page views per session, or any other metric that helps you track progress toward your goal.

It’s important to isolate the changes so that if your test does demonstrate a statistically significant change, you know exactly what caused the difference. If you change more than one thing in a test cell, you won’t know which change caused the result. 

Finally, be sure that the hypothesis for your A/B test can scale. Having a successful test that you cannot afford to implement is not a good use of time and budget.

A/B Split Testing Best Practices

When it comes to A/B testing, there are a few best practices that you should keep in mind. 

First and foremost, you should consider your sample size and the minimum detectable effect. Small sample sizes lead to low-confidence results. It is important that your sample size is large enough to detect changes in conversion rates or other metrics of interest. 

There are tools available to determine minimum sample sizes. Generally, you need at least 100 in the test cell. A good rule of thumb is 10% of the population up to one thousand. You can use the chart below as a guide.

Example sample sizes to measure statistical significance

By understanding the key concepts and principles behind A/B testing, measuring statistical significance, and following best practices, marketers can confidently use A/B testing to make data-driven decisions. 

Plan on Multiple Tests

Please don’t think that testing is complete after a single test. It’s best practice to re-run the same test multiple times to increase your confidence in the result. The initial result might be an outlier, or external factors may have influenced the results. Things as simple as changes in the weather, the time period, or changes in customer preferences over time can influence the results.

It’s also a good idea to gradually increase the size of the test cells each time you rerun a successful test. This is sometimes called confirmation testing.

You need to do both things to ensure that you make the best decisions for your business.

Measuring Statistical Significance – the Math

The two most important concepts to understand are the p-value and confidence intervals. 

measure statistical significance - Featured Image


The P-value measures how likely it is that the differences between your variations were caused by random chance. For example, a low P-value (less than 0.05) indicates that the difference is unlikely to be random chance, and therefore the result is likely statistically significant. 

Confidence Intervals

The confidence interval measures how confident you can be that the result is accurate and will be consistent across multiple testing cycles. It’s typical to have three confidence levels: 90%, 95%, and 99%. Select 95% most of the time. 90% is a bit loose, and 99% is too tight. Use them to give you a better sense of how the test performed.

I’m not going to get into the math here. I’m not competent to do this, and I don’t want to bore you into leaving. If you’re not comfortable doing the statistical analysis yourself, you can use the calculator below to test your results for statistical significance.

Measure Statistical Significance



Conversion Rate






Significant result!

Variant B’s conversion rate (5.20)% was higher than variant A’s conversion rate (4.33)%. You can be 95% confident that variant B will perform better than variant A.

p value


Interpreting Results

Once an A/B test is complete, it’s time to interpret the results.

If the change is statistical significance, you can be confident that the winning variation will perform better than the control. But, as mentioned above, don’t rely too much on the results from a single test. This can lead to inaccurate conclusions. You should always run your tests multiple times with gradually increasing sample sizes and measure statistical significance each time. You want to understand the data’s overall trend before making decisions. 

Finally, don’t rely solely on a tool to provide the answer; consider your own interpretation as well. 

You can confidently make data-driven decisions if you conduct the analysis, measure statistical significance, and add your own interpretation.

Conclusion – Measure Statistical Significance with Confidence

Hopefully, this guide has provided an overview of the basics of A/B testing, from the importance of measuring the statistical significance of observed changes and how to set up and interpret tests. 

Whether you’re a beginner or an experienced marketer, A/B testing is a powerful tool that you can use to understand marketing performance and, when you measure statistical significance, confidently make data-driven decisions. 

With the right resources and best practices, any marketer can incorporate A/B split testing into their strategy to get meaningful results!

Good luck and happy testing!

Author: James Hipkin

Since 2010, James Hipkin has built his clients’ businesses with digital marketing. Today, James is passionate about websites and helping the rest of us understand online marketing. His customers value his jargon-free, common-sense approach. “James explains the ins and outs of digital marketing in ways that make sense.”

Use this link to book a meeting time with James.