Statistics for A/B Testing [Online Course]

Become a proper optimizer and know your stats

If you’re not fluent in A/B testing statistics, you won’t be able to tell whether your tests suck.

Online course:
Statistics for A/B testing

By Georgi Georgiev, Owner of Web Focus LLC

Course length: 3h 30min

Some of the companies that train their teams at CXL Institute:

A lot of your “winning” tests are probably not winners at all. Learn to call bullshit when needed, and be the person who advocates proper scientific approach in your team.

In 10 sessions, you’ll learn

  • How to run A/B tests with a sound statistical design in a variety of scenarios – multiple test variants, multiple outcomes, non-binomial data, and others.
  • How to align the statistical design with the questions at hand to get business insights while avoiding common mistakes.
  • The logic behind statistical hypothesis testing and concepts like statistical significance, confidence intervals, statistical power, and others.

Introduction video (4min)

video

Understand the complexities involved in planning and evaluating A/B tests

Avoid costly testing mistakes stemming from misuse and misunderstanding of statistics, and improve the ROI of all your A/B testing efforts, with Georgi Georgiev’s guidance.

Anyone interested in genuinely understanding the math behind CRO and A/B testing absolutely has to go through this course and take the exam as well. My whole team had a great time discussing the course material and aligning our views on the process we use and to come up with the ideas to implement in the future.

Radvilas Šeputis, Digital Strategist @ Evolvery

This course is right for you if…

  • You can’t define statistical significance correctly without looking it up on Google.
  • Your A/B tests produce a lot of “winners,” but your clients aren’t seeing improvements.
  • You’re planning and analyzing A/B tests, but you don’t understand the statistical underpinnings of the testing process.
  • You’re not confident in the outcomes of your tests and are unsure how much trust to put in them
  • You have an in-house statistical tool you want to improve, or you use a third-party A/B testing software you want to understand better

This course is probably not for you if…

  • You are just starting with CRO and have little to no practical experience with A/B testing.
  • You don’t employ A/B tests as a primary method to evaluate CRO work.
  • You are a professional statistician or experimental design specialist.

Skills you should have before taking this course:

  • Some experience in conversion rate optimization.
  • Basic understanding of how A/B testing works.
  • Some experience with an A/B testing software.

About your instructor,
Georgi Georgiev

Georgi is an expert internet marketer and statistician working passionately in the areas of SEO, SEM and Web Analytics since 2004. He is the founder of Analytics-Toolkit.com and owner of an online marketing agency & consulting company: Web Focus LLC and also a Google Certified Trainer in AdWords & Analytics.

His special interest lies in data-driven approaches to testing and optimization in e-commerce and internet advertising and Georgi is also the author of three papers, multiple articles on A/B testing for conversion rate optimization, as well as the book “Statistical Methods in Online A/B Testing”.

In just 10 sessions, you’ll be able to

  • Plan A/B tests by taking into account relevant error probabilities
  • Correctly interpret A/B testing statistics like p-values and confidence intervals.
  • Navigate the complexities of MVT, segmentation, multiple KPIs, and concurrent tests.
  • Efficiently communicate A/B test statistics.

Your full course curriculum

Statistics for A/B testing

Lesson 1

Basics of Causal Inference

Lesson Objectives:

  • Understand the difficulties involved in making sense of data in a noisy world
  • Learn how experiments help us deal with some of those difficulties
  • Establish the role of statistics in making business decisions
  • Translate business questions into statistical hypotheses

Lesson 2

Statistical Significance & Other Estimates

Lesson Objectives:

  • Understand the need for the p-value and what it means
  • Learn how to calculate the p-value for absolute difference in proportions
  • Introduce confidence intervals as an alternative presentation of the discrepancy between observations and a model

Lesson 3

Statistical Power & Sample Size Calculations

Lesson Objectives:

  • Understand type II errors, the concept of statistical power of a test, and how to account for them in planning an A/B test
  • Learn how to determine the required sample size for a simple A/B test
  • Understand the relationship between power, significance threshold, minimum effect of interest, and sample size
  • Be able to plan a fixed-sample A/B test so it achieves a target power at a specified minimum effect of interest

Lesson 4

Multivariate Tests

Lesson Objectives:

  • Understand what challenges are posed by A/B/n designs & the different tools for addressing them
  • Understand the implications of increasing the number of variants tested
  • Explore the trade-off between testing many variants and testing quickly

Lesson 5

Running Concurrent A/B Tests

Lesson Objectives:

  • Explore the complications introduced by running concurrent tests
  • Understand why commonly-proposed approaches do not get the job done
  • Learn how and if to engage in concurrent testing

Lesson 6

Tests With Multiple Outcomes

Lesson Objectives:

  • Understand the difference between primary and secondary outcomes
  • Learn how to recognize situations which require FWER control and which do not
  • Design tests with multiple primary and secondary outcomes

Lesson 7

Non-binomial data

Lesson Objectives:

  • Understand the three types of metrics in A/B testing
  • Cut through common misconceptions about statistics based on non-binomial data
  • Calculate statistical estimates for non-binomial data

Lesson 8

Statistics for Percentage Change

Lesson Objectives:

  • Explore the difference between absolute and relative or percentage difference
  • Define a proper statistical model for percent change
  • Calculate statistical estimates for working with percent change outcomes

Lesson 9

Asking the Right Questions

Lesson Objectives:

  • Recognize when a one-sided test is appropriate
  • Understand the need for correspondence between substantive(business) and statistical hypothesis
  • Learn how to translate different business questions to appropriate statistical hypotheses

Lesson 10

Communicating Statistical Results

Lesson Objectives:

  • Learn how to prepare stakeholders for using statistical insights
  • Explore best practices for presenting statistical insights through graphs, stories, tables, etc.

Show off your new skills: Get a certificate of completion

Once the course is over, pass a test to get certified in Statistics for A/B Testing.

Add it to your resume, your LinkedIn profile, or just get that well-earned raise.

CXL Institute is an authorized LinkedIn education provider

You can add your education, certificates, badges – everything you learned and earned at CXL Institute into the Education section of your profile.

How to find CXL Institute on LinkedIn

Get this course and 50+ others when you sign up for CXL Institute

This course (and all others – full list here) are included with the CXL Institute subscription.

At CXL Institute, you can take online courses on growth, digital marketing, optimization, analytics, persuasion – all in a single subscription.

  • World’s leading practitioners teaching you their best stuff.
  • 50+ online courses on all things data-driven marketing
  • New courses added every single month

Sign up for CXL Institute and get access to

  • 50+ courses (full list here) on all aspects of data-driven marketing (new courses added every month)
  • 4 Minidegrees that give you advanced level skills
  • Talk recordings by world class practitioners at CXL Live and Elite Camp

Got a team, multiple users? Get our Teams All-Access plan instead.

The CXL Institute guarantee:

  • 7-day no questions asked money back period
  • You can immediately apply learnings and improve your results.

Some of the companies that train their teams at CXL Institute: