Join the Conversation Add Your Comment

  1. Stan Eigi

    These are some very useful tips. And they make sense even to me, although I’m not very good at all this stuff. The “testing everything” part is especially understandable.

    Reply
  2. Hi Peep, great article.

    What would you consider the ‘minimum’ visits for A/B testing, in other words what would you define as “very little traffice”?

    We’ve always worked on the basis that 1,000 unique visits is sufficient for a statistical edge. Agree/disagree or that too arbitrary a question?

    Reply
    1. Peep Laja Peep Laja

      Visits matter little, 1000 is NOT valid sample size for most cases (unless you have extremely high conversions).

      Ideally you call a test when you have 250 conversions per variation (=500 total conversions).

      If your conversion rate is ~2%, then 1000 visits is like 20 conversions. VERY FAR from significance. So any “test” you called with 1000 visits is bogus.

      You have to aim for 95% or more statistical significance. You can calculate the significance of your test with a tool such as this one: http://mystatscalc.com/

  3. Jacksmith

    This is very nice explanation. Noting much but I totally agree that there are much things for solving the problem of traffic and getting the useful data. Your article solved out my queries. Thanks for sharing.

    Reply
  4. Mahdi

    drifted here from google plus without expecting much, but what I saw is amazing. Would come very handy for a a very new site which I have launched. Thanks Peep

    Reply
  5. I’ve just read a great post of guy who spent 4 years for a/b testing, and he got an amazing result from a very little change in detail. And now, your great post does help me to be motivated to take action immediately :). Thanks!!!

    Reply
  6. Interesting stuff, especially if you’re in a niche market in a small country like I am.

    Even with great positions in search engines, the traffic for certain terms just isn’t large enough to do A/B tests.

    I’ll definitely give sessioncam a go to, because with low traffic, it’s hard to justify the cost of Clicktale, which offers no entry level options (despite being a great solution)

    thanks

    Reply
    1. Peep Laja Peep Laja

      If you want to do user session recordings, Inspectlet is the cheapest option. I do have doubts about the accuracy of their heat maps though.

    2. I went with sessioncam eventually, as it worked out to be the best option, given the amount of traffic.

      So far, pretty pleased with their support as well, as responses came quick and the account was created quickly (despite the original indication that it would take 24 hours or so)

      Anyway, thanks for the suggestion of both, since I wasn’t even aware of these options.

  7. Francis Teo

    Great post. 100% agree. The only issue is that people do sequential testing and are not aware of the factors that may skew the results, such as seasonality which you mentioned.

    Reply
    1. Peep Laja Peep Laja

      I agree, sequential testing should be avoided whenever possible, too many factors at play.

  8. Steven Macdonald

    Great post, Peep.

    You’re absolutely right! Sites with low traffic can take months to reach a conclusion. One site saw 100 visits per month to a page we were testing and it literally took 6 months before we could identify a conversion winner…

    Having a basic CRO toolkit that includes usability testing, heat map analysis and customer surveys is more than enough to get started with the CRO process, all of which are pretty cheap ($100 per month?).

    Then when you have a few business cases, you ask management for more pricier tools such as Clicktale.

    Reply
  9. Blair

    Very great post as usual Peep!

    I get around 12,000 unique visitors a month right now, which is not crazy traffic or anything, but good enough to run some tests.

    I have been split testing my optins, but to get statistical significance faster, I have been doing what you just said by letting one form get all the traffic, looking at the conversion, then putting in a new one for about the same amount of views…and comparing the two instead of just doing an AB test.

    That should be pretty accurate right?

    I it is tough traffic, my site is a place where waiters submit funny stories etc, I am going to be developing a course for making bigger tips to try and monetize better…just added a feature box to top with email capture but just haven’t found a real winner yet.

    Thanks for the amazing content, your blog could be sold as a product! Speaking of products….any further development on the course you were working on?

    Consider one sold already!

    Thanks,
    Blair

    Reply
    1. Peep Laja Peep Laja

      Hey, with your traffic you should absolutely do proper split testing and not sequential testing, otherwise you really don’t know the actual impact of the changes and run the risk of using bad versions as its not apples to apples. There is no substitute for a/b testing. Sequential testing should only be used if your traffic is really low and you’re “testing” very different versions.

    2. Blair

      Hey Peep,

      Thanks so much for the reply and advice! Will keep at it! Really appreciate the help….I thought that by putting in my email here I would get notified that you replied? I did not, otherwise I would have thanked you sooner! oh well.

      BTW, have you ever considered going on Clarity.fm? (no affiliation)

      Basically you set a rate and charge people per minute of your time….if you ever go on there please let me know as I would love to ask some questions of you but wouldn’t dream of asking them if you were not getting compensated.

      Glad to hear course is coming out in October…thats when I am getting married…it shall be a wedding present to myself!

    3. Peep Laja Peep Laja

      Oh, and the course is probably coming out end of October. Thanks for checking.

  10. Oliver Stark

    Fantastic article and nice external resources.

    What is a “low traffic” level ? How much visitors / months ?

    Many thanks, I wish I could meet you in Boston…

    Reply
    1. Peep Laja Peep Laja

      Thanks. Low traffic = it takes forever (many many months) to run a split test until statistical significance.

      If you split test 2 variations, get 100 people / day to the page and your control converts at ~2%, it will take close to a year to run 1 split test. That’s low traffic :)

Leave a Reply

Your email address will not be published. Required fields are marked *

Current article:

How To Do Conversion Optimization With Very Little Traffic?