Free A/B Test Significance Calculator

Check if your A/B test results are statistically significant in seconds. Calculate p-values, z-scores, and confidence levels -- or plan your next experiment with our sample size calculator. No sign-up required.

Instant Significance Check

Get p-values, z-scores, and confidence levels with a clear yes/no significance verdict

Sample Size Planning

Calculate the exact number of visitors needed before you start your experiment

Actionable Insights

Get clear recommendations on whether to ship, iterate, or keep testing

How it works

1Enter visitor and conversion counts for control and variant

2Choose your desired confidence level (90%, 95%, or 99%)

3Get instant statistical significance analysis with p-value and z-score

Understanding A/B Test Statistics

The essential concepts behind statistically valid experimentation

What is an A/B test?

An A/B test (also called a split test) is a controlled experiment where you compare two versions of a page, email, ad, or other asset to determine which performs better. Visitors are randomly split between the control (version A) and variant (version B), and their behaviour is measured against a predefined goal such as conversion rate, click-through rate, or sign-ups. A/B testing removes guesswork from optimisation by providing statistically validated evidence of what works.

Key statistical concepts

Statistical Significance -- the probability that the observed difference is not due to random chance. Typically measured at the 95% confidence level.

P-Value -- the probability of observing results as extreme as yours if there were no real difference. Lower p-values indicate stronger evidence.

Z-Score -- the number of standard deviations the observed difference is from zero. Higher absolute z-scores indicate more significant results.

Statistical Power -- the probability of detecting a real effect when one exists. Standard minimum is 80%.

Minimum Detectable Effect -- the smallest relative change your test is designed to reliably detect.

A/B testing best practices

-Calculate sample size before starting to ensure your test can detect meaningful differences
-Run tests for full weeks to account for day-of-week patterns in visitor behaviour
-Never stop tests early -- peeking inflates false positive rates and leads to incorrect decisions
-Test one variable at a time so you can attribute any difference to a specific change
-Document and share results to build institutional knowledge and avoid repeating failed tests

Common testing mistakes

-Ending tests too early based on initial results that have not reached significance
-Testing too many variants without adjusting for multiple comparisons
-Ignoring sample size requirements and running underpowered tests
-Testing trivial changes like button colours when bigger levers (copy, offer, layout) have more impact
-Not segmenting results to understand which audiences responded differently

Frequently Asked Questions

Everything you need to know about A/B testing and statistical significance

What is statistical significance in A/B testing?

Statistical significance means the difference in conversion rates between your control and variant is unlikely to have occurred by random chance. A result is considered significant when the p-value falls below your chosen threshold -- typically 0.05 for 95% confidence. This means there is less than a 5% probability the observed difference is due to noise rather than a real effect. Understanding significance is critical for making data-driven decisions in your demand generation strategy.

What confidence level should I use for my A/B test?

The 95% confidence level is the industry standard for most A/B tests. Use 99% confidence for high-stakes changes such as pricing pages, checkout flows, or anything that directly impacts revenue. Use 90% for lower-risk experiments like headline copy tests or image swaps where the downside of a wrong decision is minimal. Higher confidence levels require larger sample sizes, so balance rigour with practical constraints.

How many visitors do I need for a valid A/B test?

The required sample size depends on four factors: your baseline conversion rate, the minimum effect size you want to detect, your significance level, and your desired statistical power. For a typical B2B website with a 3% conversion rate testing for a 10% relative improvement at 95% confidence and 80% power, you would need roughly 30,000-40,000 visitors per variant. Use the sample size calculator above to get exact numbers for your specific scenario.

What is a p-value and how do I interpret it?

A p-value is the probability of seeing a difference as large as (or larger than) the one you observed, assuming there is actually no difference between the two versions. A p-value of 0.03 means there is a 3% chance the result is due to random variation. The smaller the p-value, the stronger the evidence that the difference is real. In A/B testing, a p-value below 0.05 is generally accepted as statistically significant, but this threshold should be set before the test begins.

What is the minimum detectable effect (MDE)?

The minimum detectable effect is the smallest relative improvement your test is powered to detect. For example, if your baseline conversion rate is 5% and you set a 10% MDE, your test is designed to detect changes from 5.0% to 5.5% or larger. Smaller MDEs require substantially more traffic. Choose an MDE that represents a meaningful business impact -- typically 5-20% for B2B experiments. Testing for very small effects (under 5%) often requires impractically large sample sizes.

How long should I run an A/B test?

Run your test until you reach the pre-calculated sample size, and always for at least one full week to capture day-of-week effects. Never stop a test early because early results look promising -- this practice (called peeking) dramatically increases false positive rates. For most B2B websites, expect tests to run 2-4 weeks. If your site has low traffic, consider using the cold email strategy to drive more targeted visitors to your test pages.

What is statistical power in A/B testing?

Statistical power (also called sensitivity) is the probability that your test will detect a real effect when one actually exists. The standard minimum is 80%, meaning there is an 80% chance of identifying a true winner and a 20% chance of missing it (a false negative). Higher power reduces false negatives but requires more visitors. For important business decisions, use 90% power to minimise the risk of dismissing an effective variant.

Can I test more than two variants at once?

Yes, you can run A/B/n tests with multiple variants. However, each additional variant increases the total sample size needed because you must adjust for multiple comparisons to avoid inflated false positive rates. For most B2B websites with moderate traffic, sequential A/B tests (two variants at a time) reach significance faster and yield clearer insights. If you do run multivariate tests, apply corrections such as Bonferroni adjustment to maintain statistical rigour.

Related Resources

Explore more tools and guides to optimise your go-to-market strategy

Ready to Build a Data-Driven Growth Engine?

Our GTM experts help B2B technology companies implement systematic experimentation programmes that continuously improve conversion rates and reduce acquisition costs.

Get a Free GTM Audit Explore Our Services