Like what you read?

Get our blog posts sent to your e-mail
  • This field is for validation purposes and should be left unchanged.

Join the Conversation Add Your Comment

  1. Hi,

    Do you know what is the test performed behind mystatscalc (geek stuff)?

    Reply
    1. Peep Laja Peep Laja

      Im not sure about this site in particular, but theres lots of statistical confidence algorithm code samples around if google a bit

      Reply
  2. Christian

    Hi, good post. What tools are you using. I’m thinking of the two screenshots of statistical data. Doesn’t seem like it is mystatscalc.com.

    — christian

    Reply
    1. Peep Laja Peep Laja

      This is Visual Website Optimizer, I use this for split testing

      Reply
  3. Jente

    Nice read. Point 2 is very important, a lot of results of A/B tests I ran changed after the first days. Never make conclusions to fast, don’t follow your gut, only follow statistical proof.

    Reply
      1. Lauren Waters

        As an addition to Jente’s point it might be worth running tests with less variables so that you can reach the sample size that you want. Taking time to really plan and develop the content being tested can save time in the long run! :)

        Reply
      2. Posicionamiento web

        As you say, Analysis and understand what are you doing, and what your visitors are doing in your site is the key to improve your CR.
        I keep an eye on the tool, Visual Website Optimizer

        Thanks!

        Reply
  4. Bryant Jaquez

    Every client and new CRO-tester in the world should read this post. You did a great job at putting things simply. I am going to use this article as a reference to explain CRO to prospective clients.

    Reply
  5. Naomi Niles

    I’m glad you pointed these out, Peep! And I’m also glad to hear I’m not the only one who has these types of frustrations with commonly held beliefs.

    Reply
  6. guillaume

    Great post – I don’t think http://mystatscalc.com/ is reliable to calculate the confidence to beat the control, if you use it with the data from your screenshot, you will find out a 90.89% chance to beat control, where your tool show 95% (with my formula I found 95.46%).
    Anyway I think your conclusion is very true – it is all about the learning…and patience!

    Reply
  7. Agencia seo

    Great Post, The ab test is essential today for a marketing strategy long term oline serious, always thinking about the conversion.

    Reply
  8. Terry Kyle

    Good post Peep. Sadly, things like patience, realistic expectations and a thorough process of audience understanding (which do work) are very much out of fashion in today’s one-click solution (that doesn’t work) environment. Keep up the good work.

    Reply
  9. Agencia Marketing Online

    Thanks for sharing. We should take care with the % of the improvement if that is calculated on other percentages. I mean, % of improvement of 0.5% to 1% is 100%, looks like a completely success, when really is a logical trend when you are improving from the worse scenario.

    Reply
  10. Agencia Marketing Online

    The AB test is essential today for a marketing strategy long term oline serious. Great Post ;)

    Reply
  11. Steven

    Thanks for sharing Peep! We use Google’s A/B testing tool and you are correct with your conclusion, A/B testing takes time and patience and lot’s of small tweaks and testing ;-)

    Reply
  12. Http://inboxblueprintreview.co/

    It’s an amazing piece of writing designed for all the online people; they
    will obtain benefit from it I am sure.

    Reply
  13. Paolo Dossena

    Hi, I think you’ve reach the point by saying that at the end A/B test is about understanding your audience more than trying this colour or that picture size instead of the other. Only by really focusing on users, trying to gain knowledge of them by our test, one can finally increase conversion, but sadly most of the company thing backwards, targeting a goal and then by making small changes hope that they get there.

    Reply
  14. rajesh

    Thanks for sharing Peep! We use Google’s A/B testing tool and you are correct with your conclusion, A/B testing takes time and patience and lot’s of small tweaks and testing ;-)

    Reply
  15. Consultor SEM

    So true. A/B Testing it’s a necessary item in the checklist but the improvement not finish there! thank for sharing the post with this humor. ;)

    Reply
  16. Marketing online en Barcelona

    Thanks for sharing Peep! We use Google’s A/B testing tool and you are correct with your conclusion, A/B testing takes time and patience and lot’s of small tweaks and testing ;-)

    Reply
  17. Diseño Web Granada

    Great Post, The ab test is essential today for a marketing strategy long term oline serious, always thinking about the conversion.

    Reply
  18. A nice article and some nice tips. I think, however, that statements such as “statistical significance is everyting” or recommending a 95% significance as a rule of thumb is not very helpful.

    The first can make a person forget about statistical power and fall prey to arbitrary stopping (e.g. testing ends when reached signficance) which can basically increase your error rate from 5% to 100%. I argue strongly and in detail about those issues here: http://blog.analytics-toolkit.com/2014/why-every-internet-marketer-should-be-a-statistician/

    The second thing is that 95% significance might be too high a price to pay in case of small risk/cost to test. Also, it might be far too uncertain if the stakes are high or the decision – irreversible!

    Reply
  19. anamika

    Useful post Peep!, we tried A/B testing on all our major campaigns and final what we felt is.. Google revenue increased and we have wasted lot of money in testing the adcopies.

    Reply
  20. Matt Cutts

    A/B testing is for sure help us in improving our website to be more attractive to our readers.

    Reply
  21. Marcosdice

    Nice article! really tookme like a half hour to undestand but… now im like NEO on Matrix ehehe

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Current article:

Three Hard Truths About A/B Testing