Testing tools are getting more sophisticated. Blogs are brimming with “inspiring” case studies. Experimentation is becoming more and more common for marketers. Statistical know-how, however, lags behind.
This post is filled with clear explanations of A/B testing statistics from top CRO experts. A/B testing statistics aren’t that complicated—but they are that essential to running tests correctly.
Here’s what we’ll cover (feel free to jump ahead):
- Mean, variance, and sampling;
- Statistical significance;
- Statistical power;
- Confidence intervals and margin of errors;
- Regression to the mean;
- Confounding variables and external factors.
And just in case you’re uncertain about why A/B testing statistics are so essential…Keep reading »
From the outside, it seems like data is impartial. It’s cold, objective, accurate.
In reality though it’s more complicated. In the hands of someone with an agenda, data can be weaponized to back up that viewpoint. Even in the hands of someone benevolent, data can be misinterpreted in dangerous ways.
While this method is scientifically valid, it has a major drawback: if you only implement significant results, you will leave a lot of money on the table.