Statistics for A/B testing
Become a proper optimizer and know your stats. If you’re not fluent at A/B testing statistics, you won’t be able to tell whether your tests suck. A lot of your “winning” tests are probably not winners at all. Learn to call bullshit when needed, and be the person who advocates proper scientific approach in your team.
In this 8-class live training program, you'll learn:
- How to run A/B tests with a sound statistical design.
How to ask the right questions, avoid common mistakes, and get insights through statistics.
An in-depth understanding of statistical hypothesis testing and concepts like statistical significance, confidence intervals, statistical power, and others.
This live course starts on Monday November 6th, 2017.
This course is right for you if...
- You can’t define statistical significance correctly without looking it up on Google.
- Your A/B tests produce a lot of “winners,” but your clients aren’t seeing improvements.
- You’re planning and analyzing A/B tests, but you don’t understand the statistical underpinnings of the testing process.
- You're not confident in the outcomes of your tests and are unsure how much trust to put in them
- You have an in-house statistical tool you want to improve, or you use a third-party A/B testing software you want to understand better
This course is NOT for you if...
- You are just starting with CRO and have little to no practical experience with A/B testing.
- You don’t employ A/B tests as a primary method to evaluate CRO work.
- You are a professional statistician or experimental design specialist.
Skills you should have...
Some experience in conversion rate optimization.
Basic understanding of how A/B testing works.
Some experience with an A/B testing software.
This class will give you all the tools you need to understand the complexities involved in planning and evaluating A/B tests.
Aside from mining specific information, you’ll get deeper and more accurate insights from your data in the process.
Avoid costly testing mistakes stemming from misuse and misunderstanding of statistics, and improve the ROI of all your A/B testing efforts, with Georg Georgiev’s guidance.
About your instructor:
Georgi Georgiev is the owner of WebFocus, a digital marketing consulting company delivering world-class marketing and analytics services for the past 10 years.
He’s the mastermind behind Analytics-Toolkit.com, a SaaS used by web analysts and CRO practitioners from agencies across the world.
Georgi is a lecturer in multiple marketing events, as well as a Google Regional Trainer in AdWords & Analytics. He is also the author of three papers and multiple articles on A/B testing for conversion rate optimization.
Why we hand-selected Georgi to teach this course
Georgi is thinking about and solving the problems most folks in CRO don't have the training or the time to pursue. After you've learned from him, you'll be able to apply experimental analyses that can save you lots of sample size and test run-time for your experimentation program.
Talking with Georgi for only a few minutes reveals an incredible depth of knowledge that rivals and often surpasses trained statisticians working in more theoretical disciplines. The value he brings is one of practical application, strikingly simple explanations of complex mathematical concepts, and a deep passion for the craft of CRO.
In just 8 sessions,
you’ll be able to:
Plan maximally efficient A/B tests.
Correctly interpret A/B testing statistics.
Navigate the complexities of MVT, segmentation, multiple KPIs, and concurrent tests.
Plan and analyze sequential tests.
Your 4-week live class schedule:
The 45-60 minute live classes will take place on Mondays and Wednesdays at 11 a.m. US Central time. Folks in Europe can tune in in the evenings.
Recordings of all classes are made available so you don't have to watch each class live, although we recommend joining live if possible.
Those who attend live will have the opportunity to take part in Q&A.
Everyone will have access to the forum where Georgi will answer your questions and provide feedback. Your instructor will be involved in the forum only during the weeks that the course is live. However, the forum will remain open for students who want to discuss ideas even after the course ends.
Statistics for A/B testing
Why A/B test? Basics of causal inference
In the first class, we’ll lay the groundwork that's required in order to understand more advanced concepts in subsequent classes. We’ll go over basic concepts that are crucial for developing a probabilistic mindset.
- Correlation and causation. Observational analysis versus controlled experiments.
- Sampling and natural variance and their implications for drawing insights from data.
- Null-hypothesis statistical tests – history and basics of causal inference.
- Control and randomization in A/B tests – why we need them and how they work?
- One-sided and two-sided tests, composite vs. point hypothesis.
Statistical significance & confidence intervals
Statistical significance is one of the most abused concepts in conversion rate optimization. You will learn what it is, really. We’ll discuss common misuses and misunderstandings, their consequences, and how to avoid them.
- What is statistical significance.
- Common misuses of statistical significance and how to avoid them.
- Common misinterpretations and how to avoid them.
- A/A, A/B/A, A/A/B/B testing – when are they appropriate, and what can they be used for?
- Confidence intervals.
Planning A/B tests: Sample size & statistical power
Why is statistical power so important, and yet so neglected? You will learn about the trade-offs involved in planning A/B tests, and how to avoid under- and over-powered tests.
- What is statistical power and why does it matter?
- The relationship between power and other statistical parameters: significance, sample size, & minimum detectable effect.
- Under-powered and over-powered tests.
Multivariate testing & concurrent tests
Learn how to properly plan and analyze a multivariate test, avoiding common pitfalls. We examine the practice of running multiple concurrent tests, and how and when it's appropriate.
- Complexities introduced by testing more than one variant versus control
- When is an A/B/n test preferred to a simple A/B test?
- Do’s and don’ts of running concurrent tests
Segmentation, multiple KPIs, & non-binomial tests
Learn how to gain deeper insights by segmentation. We’ll also examine the fine details of using multiple outcome metrics for a test and cover non-binomial metrics such as revenue per user.
- Segmenting A/B testing data for maximum insights.
- Complexities in running tests with more than one outcome metric.
- Analyzing non-binomial data such as revenue and time on site.
Sequential testing is the future of A/B testing. Learn about different approaches to sequential testing, and how to plan and analyze a sequential test.
- Shortcomings of classical fixed-sample tests
- The issue of optional stopping
- The alpha-spending approach to sequential testing
- Planning and analyzing a sequential test
- Adaptive tests – benefits and drawbacks
Faster testing by asking the right questions
Planning ROI-positive A/B tests
You will also get introductory
In addition to classes, you’ll get access to snack-sized video lessons to bring you up to speed on the course topics.
Show Off Your New Skills: Get a Certificate of Completion
Once the course is over, pass a test to get certified in Statistics for A/B T
Add it to your resume, your LinkedIn profile or just get that well-earned raise you’ve been waiting for.
Why you'll learn more from live classes
In short: You'll actually make time for learning.
Most people don't ever complete self-study courses. It's difficult to prioritize learning over urgent items on your to-do list. (Hey, we’re all busy.)
How do we know?
- A 2015 study shows that the current average completion rate for MOOCs (Coursera, edX, Udacity) is approximately 15%. So 85% drop-off rate. The course that received a Coursera award for having the highest completion rate — wait for it — had a rate of 20%!
- Seth Godin reported that his courses on Udemy and Skillshare have an 80% drop-off rate.
- We've run self-study courses for years, and our numbers confirm this
Our previous live coaching classes have had over 90% completion rates. Around 60% attended every single live class, the rest watched recordings (due to time zone differences, etc).
Our #1 goal is customer success... That you actually master this stuff. We're confident that this live class format will do it.
efore it's too late
- 8 live classes with Georgi and your classmates.
- Access to a private, members-only discussion group with Georgi and your peers.
- To ask Georgi all of your burning statistics questions.
Need help convincing your boss to pay for this? Pitch to Boss
Enrollment ends in:
ENROLL NOW — $499
Teams of 2 and more get a 25% discount during checkout.
After completing the payment you can login to the course homepage.
7-day money back guarantee.