A/B Testing

A/B Testing Statistics: An Easy-to-Understand Guide

Testing tools are getting more sophisticated. Blogs are brimming with “inspiring” case studies. Experimentation is becoming more and more common for marketers. Statistical know-how, however, lags behind.

This post is filled with clear explanations of A/B testing statistics from top CRO experts. A/B testing statistics aren’t that complicated—but they are that essential to running tests correctly.

Here’s what we’ll cover (feel free to jump ahead):

  1. Mean, variance, and sampling;
  2. Statistical significance;
  3. P-values;
  4. Statistical power;
  5. Confidence intervals and margin of errors;
  6. Regression to the mean;
  7. Segmenting;
  8. Confounding variables and external factors.

And just in case you’re uncertain about why A/B testing statistics are so essential…

Keep reading »
How to Create a Unique Value Proposition (with Examples)

A value proposition is a promise of value to be delivered. It’s the primary reason a prospect should buy from you.

It’s also the #1 thing that determines whether people will bother reading more about your product or hit the back button. On your site, your value proposition is the main thing you need to test—if you get it right, it will be a huge boost.

Keep reading »
Top 20 A/B Ecommerce Test Ideas

There’s nothing that always works and pretty much nothing that never works either. Websites are highly contextual.

That being said, there are tests that tend to have a very high win rate. These are the test ideas that, while they don’t work 100% of the time, work more often than not.

Naturally, everything depends on the specific implementation — a good idea implemented poorly will not yield any results.

The following 20 testing ideas come from our own client-based research done over the years. Keep reading »

Predicting Winning A/B Tests Using Repeatable Patterns

If you ever ran a highly trustworthy and positive a/b test, chances are that you’ll remember it with an inclination to try it again in the future – rightfully so. Testing is hard work with many experiments failing or ending up insignificant. It’s optimal to try and exploit any existing knowledge for more successes and fewer failures. In our own practice we started doing just that. Keep reading »

How to Segment A/B Test Results to Find Gold

You run an A/B test, and it’s a winner. Or maybe it’s flat (no difference in performance between variations). Does it mean that the treatments that you tested didn’t resonate with anyone? Probably not.

If you target all visitors with the A/B test, it merely reports overall results – and ignores what happens in a portion of your traffic, in segments.

Keep reading »

How to Analyze Your A/B Test Results with Google Analytics

A/B testing tools like Optimizely or VWO make testing easy, and that’s about it. They’re tools to run tests, and not exactly designed for post-test analysis. Most testing tools have gotten better at it over the years, but still lack what you can do with Google Analytics – which is like everything. Keep reading »

Google Optimize

Chances are, you’ve heard of Google Optimize by now. It’s Google’s solution for A/B testing and personalization. It launched in beta in 2016  and left optimizers around the world waiting in line to try it out.
Now that it’s out of beta, you can give it a try without the wait.

But what can you expect? How do you configure it properly? How do you run your first experiment?

Keep reading »