Join the Conversation Add Your Comment

  1. Garry Ponus

    Wow. So much information to digest. Rest assured I’d be getting my conversion analysis up and running!

  2. Mikaela Pole

    I am really interested in these kind of posts as they motivate me to learn online.

  3. Mike Hendrix

    Excellent read. Thanks for compiling and sharing.

  4. Phil Sharp

    I think you’re exactly right that running user tests can give you great ideas about what to A/B test. It leads to a whole lot of “Phil-slapping-his-head-because-he-didn’t-think-of-that moments.” Those moments are great for our company, but not so great for my forehead :).

    What I also like to do (and this is a bit unusual) is run user tests on my A/B test after a winner has already been decided. The A/B test gives me the data I need to know which test won, but the user tests help me to know why the winner one and the loser lost. This instantly sets me up with ideas for my next test.

    Since I work at I also want to quickly jump in and help clarify one small part of the post. With our service you can definitely recruit and use your own participants at no extra charge. There’s no need to use our testers if you don’t want to. Plus, for Enterprise members you can even recruit using intercepts, social media, your customer file, and 3rd party panels.

  5. Tyler

    Especially liked the usability testing discussion and examples of what you’ve learned from it. Another really interesting company that fits the Moneyball model is Relativity Media. They’re a film studio, and before they commit “to financing a particular movie — either through its slate deals with Sony and Universal or on its own — it’s fed into an elaborate Monte Carlo simulation, a risk-assessment algorithm normally used to evaluate financial instruments based on the past performance of similar products. Enough variables are included in the Monte Carlo for Wilson and his team to have reached the limits of their Excel’s sixty-five thousand rows of data.” (that quote’s from a great Esquire article about the company’s founder).

    I ran some quick numbers on the 12 movies they released this year. The total budget for all 12 movies was $622 million. The total box office those 12 pulled in was over $1.4 billion! And only 2 lost money (Wanderlust and That’s My Boy lost a total of $26 million).

  6. Now you’ve got me wanting to read moneyball! lol

    As Conversion-Rate-Experts have said “A doctor wont prescribe you medicine before they diagnose you. The same goes for conversion, you need to dig into the data and diagnose the problems so you can prescribe a solution.”

    And by doing this you can get high ROE (yes ROE not ROI) by discovering a list of small tweaks and fixes that can lead to an improved conversion rate. We can only get these smart wins by doing the research as you’ve explained with “evidence based marketing”.

  7. Diana quartin

    Hey Joseph, i know brooks will provide you a better answer but as for me I test even with low traffic, but I let the test run until i have 500 hits in total for that specific test. I just dont run multivariate tests. Awesooome article , thank u once again…. You are by far my favorite blog.

  8. Paul Sidwell

    Joseph, you can test with any amount of traffic. Use Google Analytics > Content > Experiments to set up A/B testing on your site. The one caution I’ll offer is this: Test only ONE thing at a time, for at least 30 days, preferably 90 or more. Then switch all relevant pages to the better-performing content, and move on to another test.

  9. Joseph

    Fantastic article once again. I understand the importance of testing and UX and all that. What if someone, like me, has a website that is just starting out and has 25 visitors/day? Is this enough for testing? If not, how can I ensure that my site is optimized for conversion as well as it can be for a new site? And at what amount of traffic can I start to seriously test? Cheers.

    1. Peep Laja Peep Laja


      As Paul said, you can do testing – but it’s just going to take *much* longer to get statistically significant results.

      As for doing research – you can always perform usability testing (amount of traffic you get is irrelevant). If you’re still in the early phases with your business model, you need to do customer development instead (seek out Steve Blank and everything he’s written

Leave a Reply

Your email address will not be published. Required fields are marked *

Current article:

Moneyball Design: Why the Research and Data Analysis is Critical to Getting Your Prospects to Convert