CRO & Testing

The world’s best archive of content on conversion optimization and online experimentation (A/B, MVT, bandit, AI etc).

Hold-Out Groups: Gold Standard for Testing—or False Idol?

You regularly run A/B tests on the design of a pop-up. You have a process, implement it correctly, find statistically significant winners, and roll out winning versions sitewide.

Your tests answer every question except one: Is the winning version still better than never having shown a pop-up? Keep reading »

9 Skills Marketers Need in the Age of AI

The robots are taking over. They’re going to be better than us, and then steal our jobs, and eventually turn on us and conquer the world. That’s the kind of hyperbolic language you’re probably hearing everywhere today. But some of it is true.

Will the skills you’re learning today be obsolete by tomorrow? Is there a point to learning new things if artificial intelligence (AI) is going to do it better? Is marketing going to become completely automated, and no longer require the human touch?

Keep reading »

How to Get Started with Machine Learning and AI for Marketing

Artificial Intelligence is powerful and will open doors we don’t even know exist at the moment. But there are some steps you can take today to ensure that you are making the most of this new capability and are not left behind.

As this is such a nascent area, it serves to begin with a brief description of Artificial Intelligence and Machine Learning. My assumption is that you have read a dozen different definitions and should be curious about which applies to this particular essay.

Keep reading »

Top 20 A/B Ecommerce Test Ideas

There’s nothing that always works and pretty much nothing that never works either. Websites are highly contextual.

That being said, there are tests that tend to have a very high win rate. These are the test ideas that, while they don’t work 100% of the time, work more often than not.

Naturally, everything depends on the specific implementation — a good idea implemented poorly will not yield any results.

The following 20 testing ideas come from our own client-based research done over the years. Keep reading »