Join the Conversation Add Your Comment

  1. Keith

    Really, an impressive article that provides insight about the various areas of A/B testing. Apart from this, other importantb areas of considerations are having a clear knowledge about your key performance indicators, analyzing the effectiveness of your A/B testing tools and running your tests for the scheduled time duration for achieving the desired test goals.
    Further, there are various A/B testing tools like Optimizely, Mocking Fish, Maxymiser that can provide accurate results and have relatively less testing errors so as to ensure an effective split testing results for your business organization.

  2. Georgi

    “The importance of testing to statistical relevance” sounds like a serious misnomer to me. There is no such concept as “statistical relevance”. Statistical significance is in no way a measure of “relevance”. Interpreting a statistically significant result as a “relevant” or “significant” result is a grave mistake.

    Also, testing “to” statistical significance is another commonly made error. Sadly most tools encourage you to make it…

    I explain those issues in much more detail here:

  3. david

    It would be really nice if the print button worked and this could be printed out

  4. Srihari Thalla

    I thought the Guide is a pdf and surprisingly this is a webpage!! I subscribed mainly because I’d download it and read it offline, NOT ONLINE.

  5. This is awesome! Thanks for sharing your knowledge! I totally agree with “Never stop testing – No matter how well your landing pages or e-mails may be doing, they can always do better.”

  6. Awesome tips, Jaan-Matti! I definitely have not done testing nearly as much as I should. I’ve bookmarked this page so I can reference it and all your great insight. Thanks!

  7. Farvede kontaktlinser

    Really a good post, which comes around the most, but you say, that we should use 302 and use rel=canonical, but I thought if you use Google new tool or payed like visual website optimizer, that such things was not needed?
    best regards

  8. sheriffbradshaw

    as a Conversion Optimization Consultant, I find this to be a huge resource for A/B Testing. Tank you!

  9. Derek Slater

    Wow – truly epic. The details are really useful (e.g. 302 redirects). Thank you.

    My own background is in B2B media organizations, which are mostly in the dark ages in regards to testing. A fallacy I encounter often is the notion of “I don’t care what results site X got, because their business isn’t the same as ours.” Of course their business isn’t exactly the same – but considering others’ results is still a great way to create a good, informed hypothesis for testing.

  10. Britt McCrimmon

    Great post. I am going to have to come back to this one and chew on it for a bit. You have got it packed with info!

  11. Steven Macdonald

    This is a fantastic resource for AB testing!
    Great job Jaan-Matti. I’ll be bookmarking this page for many years to come.


Leave a Reply

Your email address will not be published. Required fields are marked *

Current article:

How to Build a Strong A/B Testing Plan That Gets Results