Become great at advanced experimentation analysis

Go way beyond what most optimization experts know

Online course

By Chad Sanderson,

Head of Product, Data Platform @ Convoy

Course length: 4h 30min

Sign up to All-Access: $289 / month

Updated 12/2023 English subtitles Certificate included

Some of the companies that train their teams at CXL:


From zero to hero in self-serve experimentation

After you take this course, you won’t need to rely on what the testing tools say about results.

You’ll be able to go into the data yourself, understand it, query it, and perform advanced statistical techniques to get more value than any tool can really give you. That’s independence.

Course overview

Get to the top of the food chain in terms of the best CRO practitioners in the world

Having the ability to dive deep into experimental statistics and analysis using data and code is a skill that very few people have. You will be one of those rare phenomena. This will put you many steps ahead of all those optimization experts out there.

In this course, Chad will peel back the curtain on the world of advanced experimental statistics, data science, and discuss deep experimental understanding. All using common sense language.

  • Learn what makes data analysis so important and why we should think very carefully about building our own metrics
  • Understand how to generate sample size (the right way)
  • Master principles of analysis and metric building
  • Discover how to build, launch, and analyze experiments offline
  • Use statistical techniques like regression and time series analysis for CRO
  • Go beyond traditional A/B testing, focusing on unconventional methods that maybe you haven’t heard of before
  • Generate powerful custom charts to communicate testing results
  • Start working with R (even – or especially if your most hated thing is jumping into code since you’re not a developer).

This course will teach you how to go beyond traditional A/B testing with advanced statistical techniques (no math background required), clear away numerous misconceptions about common and uncommon testing tropes, and open your eyes to the business-wide transformative strength of experimentation.

Ideal for you if…

  • You are a conversion optimization specialist looking to dive deep into experimental statistics and analysis using data and code
  • You have a hand in analyzing, interpreting, or reporting the results of digital experiments
  • Your business or client fully believes in the power of experimentation

This course is probably not for you if…

  • You are an experienced data-scientist, researcher, or statistician with a deep knowledge of experimental design
  • You are a CMO or VP. This course is less strategy, and more hands-on functional knowledge.
  • You are content with the surface level testing results reported by your testing tool

Skills you should have before taking this course:

  • Intermediate A/B Testing experience. We won’t be covering how or why you should be setting up experiments or using a testing tool.
  • Foundational understanding of analytics best practices and reporting. We expect you to know how to choose and report on important metrics.
  • Working knowledge of Microsoft Excel or an equivalent. We will be working with data!

Skills you will master

Experimentation analysisData AnalysisRunning experimentsMetric buildingStatisticsRA/B testing

Marianne M.

I feel so much more confident in analysis and optimization after taking CXL’s courses.

I learn practical techniques that are applicable to any business from the best in class.

Jonathan R.

CXL showed me how to go from no optimization program to a fully matured program, and how to handle all the nuances that pop up along the way.

Gajan R.

If you want to take the next step from dabbling to becoming a real CRO practitioner, the CXL material (plus a lot of practice) will get you there.

Learn processes for getting consistent results from leaders in the field.

Chad Sanderson

Head of Product, Data Platform @ Convoy

Chad Sanderson is a Digital Optimization, Testing, and Personalization specialist focused on the strategy, design, implementation, and analysis of winning experiments.

Chad is a passionate believer in the power of data, combining deep actionable insights with UX and Content strategy to create better websites and mobile apps.

Get a lesson preview

  • Learn the core principles of analysis & metric building

  • This field is for validation purposes and should be left unchanged.

Your full course curriculum:

Advanced experimentation analysis

1 Introduction

In this intro lesson, Chad explores the ‘why’ of experimentation analysis. When you come to understand the fundamentals, you will learn how to set you and your team apart when launching and managing experiments. By beginning to understand the various a

2 Principles of Analysis & Metric Building

8863894 Resources Lesson 2 R file Assignment 1: Create a metric hierarchy for your organization. Be sure to specify: Northstar metrics at the top of the hierarchy Second level lagging metrics in the middle Third level lagging metrics at the bottom Bonu

3 Diving into Experimentation Analysis

Topics covered: What is sample size reallyThe components of sample sizeBinomial metrics explainedSample ratio mismatch Continuous metrics explainedPractice using R StudioN2/N1 RatioPractice/case-study Lesson Resources: What are T-Tests and how do we ca

4 Deep Experimental Understanding

In this lesson, join Chad as he touches on the immense value of randomization and representational data and how to avoid alpha inflations when creating an honest analysis. Topics covered: The power of randomization Types of randomization Honest Analysi

5 I Have the Power (Analysis)

Topics covered: Sample sizeSize effectStatistical calculatorsStatistical power Lesson Resources: The first rule of not understanding effect sizes is that you don’t talk about not understanding effect sizesAn awesome open source power and sample size ca

6 Beyond A/B testing – Part 1

Topics covered: What can’t we experiment on using an A/B Testing tool?Regression Discontinuity Designs and Why We Use themInterrupted Time Series AnalysisPractice/case-study Lesson Resources: Lesson 6 R files Lesson 6 learning.csv Lesson 6 nv.csv Lesso

7 Beyond A/B testing – Part 2

In this lesson, Chad delivers practical advice and tips on experimentation for your organization at large. The lesson touches on areas of experimentation that may put your business at risk and ways you and your can team can design high impact experimen

8 Parting thoughts

Join Chad in the last lesson to cover all the extra details and quality controls needed for proper experimentation analysis.  Topics covered: Control vs. Treatments Setting up proper controls Response Surface Methodology Experimentation Brain Stor

Show off your hard-earned skills and strengthen your profile.

LinkedIn found that people who add credentials to their profile receive 6x more profile views than those who don’t.

Add your certificates to your profile, share your accomplishments with the world, and join our alumni network.

Subscribe to CXL and get access

  • 120+ courses (full list here) on all aspects of data-driven marketing (new courses added every month)
  • 10 minidegrees that give you advanced level skills

Got a team, multiple users? Get our Teams All-Access plan instead.

The CXL guarantee

  • 7-day no questions asked money back period
  • You can immediately apply learnings and improve your results.

Need help convincing your boss?

Click here for a pitch deck, swipe files and tips.

Subscribe to CXL

Get access to this and 100+ other training programs.

See plans and pricing

or

Request demo (for teams 2+)

Learn Experimentation Analysis: FAQ

Who is this course for?

This course is for you if:

  • You are a conversion optimization specialist looking to dive deep into experimental statistics and analysis using data and code
  • You have a hand in analyzing, interpreting, or reporting the results of digital experiments
  • Your business or client fully believes in the power of experimentation

Some careers that will especially benefit from this course are conversion optimization consultant, CRO manager, demand gen manager, growth manager, and data scientist.

What will you learn?

After taking this course, you will:

  • Learn what makes data analysis so important and why we should think very carefully about building our own metrics
  • Understand how to generate sample size (the right way)
  • Master principles of analysis and metric building
  • Discover how to build, launch, and analyze experiments offline
  • Use statistical techniques like regression and time series analysis for CRO
  • Go beyond traditional A/B testing, focusing on unconventional methods that maybe you haven’t heard of before
  • Generate powerful custom charts to communicate testing results
  • Start working with R (even – or especially if your most hated thing is jumping into code since you’re not a developer).

What does the course include?

Our Experimentation Analysis training includes 8 lessons, 9 downloadable resources, 4h 30min of video material, several assignments, and a final exam.

Are subtitles available for the lessons?

Lessons have English subtitles and transcriptions available.

Will I receive a certificate of completion?

After taking this course, you will receive the Experimentation Analysis Certification from CXL, with credentials that you can add to your LinkedIn profile.

What is the refund policy?

All plans come with a 7-day refund period .

How long will it take for me to finish it?

This is entirely up to you – it’s how many study hours per week you can put in. This course is 4h 30min long.