Your Test is Only as Good as Your Hypothesis [Video]

Your Test is Only as Good as Your Hypothesis [Video]

ConversionXL Live 2016 is coming up next March (get on the list to get tickets at pre-release prices). We’re going to publish video recordings of the previous event, and here’s the first one.

You run A/B tests – some win, some don’t. The likelihood of the tests actually having a positive impact largely depends whether you’re testing the right stuff.  Testing stupid stuff that makes no difference is by far the biggest reason for tests that end in “no difference”.


One of the speakers from the last event, Michael Aagaard – my dear friend, just joined Unbounce as their CRO manager. So it just makes sense to kick this video series off with a presentation by Michael. He walks us through how to make sure we’re testing stuff that actually matters.

You can get the slides in PDF here.

ConversionXL Live 2016: March 30th to April 1st. Get on the notifications list to take advantage of special offers.

Full transcript

Thank you very, very much. I’m so excited to be here, I’ve got to tell you. I can hardly contain myself. I love speaking at events, because it feels like coming home. It feels like a family is here. It feels like you’re back on your planet. Because when I hang out with civilians, mere mortals, they ask me what I do. It’s very hard to explain.

And I love my work so much, and I want to talk about it all the time. But people ask, “So what do you do for a living?” I start explaining. They go like, “Oh yeah, IT. Oh, yeah, yeah. Hey, I broke my printer the other day, you know, so can you help me?” And I’m like, “Ah, I don’t really optimize printers. It’s more like websites?”

“SEO, yeah, yeah, yeah, yeah. So, like, traffic and Google. Yeah, yeah. My brother has, like, a little web shop. Could you help him?” Wow. Then you start explaining what it really is, and you see people’s eyes, like, drift off. So, but that’s not the case here, so anyways, thank you very much for giving me this opportunity to hang out with my extended family. It’s going to be a lovely couple days.

All right. I would like to start by telling you a little story. There once was a company. They wanted to make a lot of money. They were operating in a dying industry. So they’ve got probably a couple of years left before their product is obsolete.

And so what’s a quick way of making money? Well, we could optimize a website. They have, it’s just one sales channel, a website, right? So what could they do?

Well, they sat down and they talked about it, and there’s this thing called responsive web design, right? That’s the best big thing. You know, they read the article where it says five signs that your website is outdated. It’s not responsive. They had the marketing consultant that was telling them, you know, “Behavior has shifted, and people buy from mobile phones now, not desktop.” Oh, okay, okay, okay. So it sounded like a great idea.

So they did it. They’re like, “Yeah.” You know, if we get a responsive website, we’ll make more money.

So they invested in a responsive redesign. Put a lot of money into it. They waited a whole year. So they were even closer to dying. They finally got it. They implemented it, and they were so excited. It was going to explode, make money.

So this was actually what happened. It was insane. It just went, [blows raspberry]. Revenue, [blows raspberry]. And conversions, everything went down. It was over 40% drop. That’s like a CRO horror story. It’s like the worst possible thing that could happen.

And these guys were shell shocked. They’re like, they had no idea what the hell happened. They talked to all the consultants, they got the responsive web design, and they’re losing money. That’s too bad.

So, and it gets even worse. Like, these guys are just burning money, because there’s the cost of the responsive web design itself, right? And there’s a serious loss of revenue. Then there’s cost of damage control, because the panic reaction is just display ads and just get more traffic, get more traffic, and they’re paying, they’re paying, right?

And damage control is also hiring someone to help figure out what the hell’s wrong with it, right? So, and then there’s a double work of kind of backtracking and finding out what the hell happened, and how are we going to fix this, right?

So my point now is of course not that they should have tested it and this . . . that would . . . lame point, of course. That wouldn’t . . . we’re not even close to that. What I’m talking about is, like, the basic hypothesis. It doesn’t seem like there was one. It just seemed like there was a good idea. It was something everybody else does, so we also have to do it.

So at one point, they contacted me, and I got involved. One of the first things I tried to do is try to find out, “So what went so horribly wrong?” So we’re looking at before and after, and there are some obvious things. Like, the whole checkout funnel was not responsive, right? So you go from a responsive experience to the most important part, and then you’re like, you try to get through the thing, and it’s very complicated. There are a lot of steps.

So that was a problem. Also, they weren’t addressing . . . and the drop-off rate in the funnel was crazy. It was over 90%. That’s serious. And they had other serious problems they weren’t addressing. They weren’t answering the questions that the clients had. They weren’t presenting them with right arguments.

So there’s a lot of stuff here that they could have done, but none of them thought of it, right? So this is kind of like, it’s kind of like this. This is what they’re doing. They’re trying. There’s obviously a big problem here, right? But what they’re doing is they’re focusing on the top of the funnel here, right?

So it’s like, “Please use the bins provided, because litter causes delays.” And right now, actually, it’s successful, because there’s two pieces of garbage here, and there’s one here. So it’s working. It’s working so far. Double. We doubled it.

Okay, so what else now? But still, it’s not quite what we hoped for. So we could change the sign, you know? We could change the color of the sign. We could even change this so that if you’re, like, if you put a Coke can in, it goes . . . You know? Puts it on a conveyer belt, it goes, like that.

You know? You keep on going. Oh, Cialdini. I read Cialdini’s book. There are all these different principles. We can test them all. And the copy here, and we can get more people. It’s like, this is where it’s going on. This is the horrible thing.

So that’s kind of . . . I used to do this all the time, you know? Just mindless testing up here. And it was insane, because I thought I was being scientific, you know? Because I was testing. So that in itself I thought made me a scientist.

And I also thought that, I was like, oh, you shouldn’t do guesswork. You should test, because then you stop guessing. But I was still just guessing. I was just testing my guesses. So it made no sense.

So this was, like, me. This is actually the, this is the young Michael Aagaard from about five years ago, right? This was like, we’ll split test, test like crazy. Testing is the coolest thing in the world. Everyone can do it. It’s really easy. You should just do it, right?

So this is me now thinking, dumbass. Wow, that was just so wrong, man. Like, if the old Michael Aagaard could meet the young one, oh, they would have it out.

Oh, by the way, this is what happens when you work with CRO for a long time and a lot of clients. Stressful, man. You age, like, 10 years in a couple.

All right. Okay, so back to the hypothesis part. A hypothesis is not a guess, and it’s not a good idea, and it’s not like, “Wouldn’t it be fun to do this? Wouldn’t it be interesting to see what would happen if we did this? I heard of that.”

No, no, no. A hypothesis is, it’s an informed solution to a real problem, and it’s not an arbitrary guess. And another very, very important point is that it’s the starting point for further research. So it’s not a definitive answer. It’s a starting point, right?

And a really good hypothesis consists of three things, in my opinion. And there are different ways of doing this. I think there has to be three things here.

We have to know the change we’re going to make, right? And we have to have an idea of how this will affect our prospects. And then we want to have in the hypothesis, also, the impact that we expect to see as a result.

And we have to have the three in there. And I see a lot of people actually forgetting about prospects. You’re forgetting about the people. And to me, that’s a major mistake, because you need that. It’s not . . . when we’re optimizing . . . that’s why I talk about optimizing decisions. It’s real people. It’s their decisions who make our conversion rates go up. It’s not just ones and zeroes. It’s not just numbers and analytics and spreadsheets. It’s real people that we have to impact. So if we take them out of the equation, it’s just not going to work.

So a very, very simple way of getting some practice with hypotheses is something like this. A very, very easy little template here. We’ve got everything in there. By changing X into X, I can get more prospects to X, and thus increase X. Okay? So we’ve got the whole thing in there.

It’s cause and effect. We have a change, we have the impact we expect it to have in the minds of our prospects, and the change we see.

So I think it’s very, very central to what we’re doing that we think in hypotheses. And I find it’s very . . . I do it all the time. I’ve tried to hardwire my brain to think in hypotheses constantly. And a big part of that is actually just saying why. I spend so much time with clients just saying, “Why? Why would you do that?”

And in a lot of cases, there simply is no good reason. So if you start thinking like that, in cause and effect, it will help you a lot.

And also, I think it’s important that we get a little bit away from only talking about test hypotheses, because we’re really talking about optimization hypotheses. If we’re focusing on test hypotheses right away, I think we’re a couple of steps ahead of ourselves. Because it’s not really the test that’s the main point. It’s the optimization that’s the important point.

The test can come along down the road, but the optimization part of it, we’ll work with that the whole way through. And I’ll give you some examples of it.

But first, I would like to show you a very, very stupid AB test that I personally conducted. Okay? So this is the landing page for a free ebook I have, and it has a big . . . it has BOB, a big orange button on there. And I know from other experiments that the button copy’s very important.

So this is button copy I started with. This is the control. “Get my free ebook.” Which is, in my experience, really, really good copy, because it talks about what you get, not what you have to part with. It has the benefit in there and everything, okay?

But for a while, I thought it was, there was just this idea in my head that it would be really interesting to play a little bit with psychology and stuff. You know, what if we don’t click this button. What would happen then?

And I could never, ever do this on a client’s website, right? But I finally had the chance on my own here. So I set it up, I ran the test, control versus treatment, don’t click this button, get my free ebook. Ran it for two months. I had almost 500 conversions, almost 500 downloads. And there was no difference. There was no difference.

So this is the kind of testing I used to do. I was pretty brain-dead. I mean, and the only thing I did right here was that I ran it for a long time and I got a lot of conversions, so that I was sure that it was really, really stupid.

And probably what I would have done earlier is I would have stopped it here, confirmation bias, when it showed me what I wanted to see. Then I would post a blog post, a case study, and say, “Ah, don’t click this button is . . . you know, you need to put it in there.” And everybody else starts doing it, all this stuff, we have all these “don’t click this button.”

But really, it didn’t matter. It’s fake. It’s not there.

And that’s the stuff, I mean, it was, like, Peep got the question, what was the thing that kind of shook him up most. And stuff like this really is . . . I was so arrogant. When I finally got to the point that I, like, admitted, man, I knew very, very little, then that’s kind of when things started.

And I just, I’m at the point now where I’m saying, I’m just going to keep on learning forever and be very, very arrogant, and think that I at some point can keep on learning.

One of the big reasons why is that, from having attended a lot of different conferences like this, and spoken at a lot of different conferences. I have had the privilege of meeting people like [inaudible 00:11:20] and Craig Sullivan, Andre Morris, [inaudible 00:11:24] Gardner, conversion scientists, all these awesome guys. We can share our experiences. We can share all these things we don’t understand.

Why do we keep testing and testing, and seeing lifts, and they’re not there? Why? We’ve developed together.

And I’ve been working with Craig lately, and he’s kind of been a mentor to me, and I’ve really learned a lot from working with him. So that’s also just the message here. You can really get a lot out of these conferences. Really, we’re all here to share our knowledge. We’re all still learning. Okay?

So let me show you an example of something I would do today instead of running this test. Which is, it’s not just stupid because there wasn’t a lift. It was stupid because there was no reason to do it. I had no hypothesis. I can after-rationalize and try to come up with something that could have been a hypothesis, but it wasn’t there. I had no idea. It was just because I thought it would be a sexy test. That is never a good reason to test, right?

So here’s another example. This is a client I’ve been working with in Denmark. They had a beauty clinic where they do laser treatment and stuff. So how do we optimize it? This was a home page. What are we going to do with the website?

Well, back in the day, I probably would have started with the button copy, I think. That would be the main thing. Nowadays I’ll do something different. Start with research now, start asking questions. And I’d say, well, there’s something serious missing here. This is a highly competitive product. You get these wellness deals all the time. There are all kinds of campaigns. There are tons of these guys in Copenhagen.

And what is the one thing people want to know? So how much is it? And there isn’t an overall pricing page on there. My competitors have it. And it’s, in my experience, it’s very likely that people are actually looking for information.

So maybe we, it’s not the right thing to have, like, tons of beautiful women all over the place, and we’re talking about, oh, your skin after this treatment will be da da da. My hypothesis was that people are going to need the facts.

So how do we qualify that? Well, in this case, I use a little feedback poll you can see down here. So this is in Danish, so there’s only one guy in the room who can understand it. You and me thing going on here. But actually, what I was trying to find out here was, the questions I was asking was, “Why did you visit the website today? Was it to check prices? Was it to learn more about our treatments? Or was it to book?”

And the client was sure that everybody was there to book. You know, that’s what we have to focus on. So we ran the feedback poll, and it turns out that 40% were there to check prices. That’s a pretty clear sign. And then to learn more, and then actually only 23% were actually there to book.

So what do we need to focus on? Getting clearer. Pricing is important.

So further confirmation here. When we go to the landing page for one of the products, they have prices on there. And we look at a click map, we’ll see that 26% of the clicks are right down here. That’s another indication that pricing is important.

So putting the pieces together, my hypothesis is, that there’s a big hole in the middle of the decision-making process here. There’s something very, very fundamental that I need to know, and I’m not being told. So if you can put that on the page we can bridge this serious gap, and we can motivate people to actually say yes, and then we give them leads.

Okay. So we created the pricing page here, and then we put it in also as . . . we’re going to use it in the global nav. And then we’re going to test it. We’re going to run tests and see if it works.

Let’s just stop for one second. It’s actually a hypothesis that we can test. That we can run proper AB tests. So you want to double-check that, okay?

So it’s a low-traffic . . . or, yeah low-traffic website. This is not the true conversion rate. It’s around there. I just put in two, okay?

So this is a likely scenario. I know they only get about 500 visitors a day. So the current conversion rate is 2%. And while it’s a little bit high, but let’s put in a 10% lift. We think we can get a 10% lift by getting this very important information.

And we have two variations. We have 500 visitors a day, include 100% of the traffic. It would take us 314 days to get a result. That’s insane. That’s ridiculous. Why would we do that?

I mean, I used to do this all the time. I’d start a test, and it would just be, blah, going into oblivion, and I had no idea.

So things like these, these are also part of the hypothesis we need to understand. Can we actually run the proper test? And there are a couple different considerations here, because one thing is, like, the geek in me wants to test everything, but, like Andre says, it’s too expensive. We have clients. We’re doing work here. It’s not just for my own pleasure.

So what would I need? I run most of my tests for four weeks. So how would I be able to get it down to around there? Well, I’d need a lift of 33%. Then I’d be down to 35 days. I’m not willing to risk that, because I’m not sure if it’s going to do that, so I could end up really hurting myself here.

So instead, we implemented it, we tracked it, and we saw just from the very first day, it was on the top three most visited pages on the website, and it keeps going. Another thing that happened was we started getting leads. We started actually getting leads from the pricing page, which was amazing. People are going, “Oh, wow, reasonable, 50%, oh, sure.” Okay? So we got another lead gen page on top of that, okay?

So this is another way of doing optimization. And the point here, of course, is that AB testing, like Andre was talking about, and CRO are not the same thing. People often introduce me as an AB tester. “Oh, Michael’s a great AB tester.” I get angry, you know? I’m not . . . I’m not the split test junkie any more.

That’s one of the tools that I can use if it is helpful. In this situation, it would not have been helpful. We would not have gotten the insight we needed. So we chose not to do it.

So really thinking about hypotheses like this helps us avoid stupid shit. So that’s the main reason why we want to always . . . we want to hardwire our brains into thinking in hypotheses.

So what happens if the hypothesis backfires and it doesn’t work? Well, let me show you a couple of examples. This was a betting site, a betting forum. I think I’d better walk down a little bit here so you don’t get seasick. I have a little bit more of a whiff I can span here.

Okay, so this was a betting expert, an online betting forum. And on the home page here, this is, like, the main landing page. We want people to sign up. We want as many people to sign up as possible.

Now, one of the things that I’ve . . . I’ve been fascinated by forms for a long time, and one of the things that I was seeing a lot of places is the privacy policy, okay? And what would be a likely hypothesis? That maybe people aren’t quite comfortable giving their information here. So maybe a privacy policy would work.

So I stole a privacy policy from a very, very popular American online marketing blog, because these guys must know what they’re doing, right? And this was my hypothesis. By adding a privacy policy, I can reduce friction, get more prospects to sign, to give me their info, and then I can increase signups. Okay.

So, put the privacy policy on there. I’m ready to run the test. I’m ready to prove to the client that I am the world’s most awesome optimizer. I can do incredible shit with just one sentence.

It didn’t work at all. So it mega backfired. And if I hadn’t tested this, well, that would have been a serious problem. And this was a situation where I actually could test, because we had enough traffic.

So, and stuff like this is really, really humbling, in my opinion. And you need . . . I need. Well, I’m just saying you. I need this once in a while, because I can get up, the ego kind of, gets really big, and then you do a test like that, and it kind of deflates. And that’s really cool. And then you can start learning some more.

All right, so what did we take away here? Does this mean that privacy policies do not work? Is that it? We disproved the hypothesis.

Well, not necessarily. Maybe it was just the wrong way to present it. We were just, maybe the hypothesis was fine, we just need to tweak it a little bit. Okay?

So if you know a little bit about how the brain works, there’s this guy called Kahneman, he has a book called “Thinking, Fast and Slow.” Awesome thing. He’s talking about how you have a very, very quick intuitive part of the brain. That’s System I, okay? And then you have System II, which is the more analytical. It’s not like your brain is divided into two different parts. It’s just two different ways of thinking.

And System I is very, very intuitive, and it’s built on experience, and all kinds of memories and stuff, and it fires very, very quickly. And actually, System II is very, very lazy, very, very . . . it takes a lot to wake it up, right?

So System I, for example, walking. That’s System I. You know, if that’s in your System II, then that would be like, we’d never get anywhere, and multi-tasking would be out of the question, okay?

So System II is like the lazy controller that we really have to wake up. It’s like, System I is zipping away, then System II is, like, asleep back there. And then once in a while, System I goes, “Hey, System II, I need your help.” And it wakes it up. And it’s very hard for us to use System II for a long time. We’ve got to feed it. It actually burns glucoside, right?

So in a situation like this, where you see the spam word, what do you take away from it? If you use System II just for a second, you will read it and say, they’re actually saying that they’re not going to do it. But when you see it very, very quickly, all you see is the spam thing.

So what if we take that away, right? And we can reduce friction, and we can get more prospects to sign up. And so I ran this test, and this one worked, right? Okay, so we’re working with the hypothesis.

This one worked. Okay, very, very cool. So if you have a very solid hypothesis, you should also be able to reverse it. So the other way around here, by removing the guarantee and adding the word “spam,” I can increase friction, get less prospects to give me their information, and thus decrease sign-ups.

So I tested this on my own landing page, and yeah, I hurt conversions very badly by doing it. Okay? But now I’m seeing something working, and I’ve seen it on different websites. So this is a pretty solid hypothesis.

So this is, for me, a very, very interesting way of working. And we’re always moving forward, always learning. Even when we got a negative result before, we got learning, and we can keep on working with that.

So what I’m getting at here, of course, is best practice, what I used to do, it would work sometimes. I was against best practice, but I was really just inventing my own best practices. And I couldn’t figure out why it was like, in this case it was just such a big success, and over here it just didn’t work. I didn’t have any form of methodology.

And that’s really what people need out there. And most clients, I meet, that’s what they need. They need discipline, they need structure, and they need a specific set of tools and a specific way of working. Okay?

So I used to only do this. And now basically what I spend most of my time doing is conducting research, hypothesizing, and then analyzing results afterwards. Okay?

So today we’re going to, of course, focus on research and hypotheses. So let’s get into some examples here.

So when we do website reviews, Craig and I, this is part of the methodology we use. So one of the things, I would probably do a quick walkthrough of the site, just get a quick idea of what’s going on, and then I would dip my toes in analytics and just look at some of the stats, get an idea of who’s missing the website, how they’re doing it, where they’re coming from.

So I would look at the device categories, for example, get an idea of that. Then I’d move on to looking at browsers. And here we can see Safari’s number one. Wow, very interesting. But no, that wasn’t quite actually the reality, because most of our visitors are from desktop, but Safari can be on all different kind of devices.

So let’s throw in a segment. Ah, it turns out that Firefox is actually the one. We’re looking at only desktop traffic now.

So this is not going to be, like, an analytics section. I just want to show you just a couple of examples here, and how you have to dig a little bit deeper. If you only look at the first numbers, well, you might be actually misled. Okay?

So some of the other reports I would look at here, just quickly, is, of course, top landing pages. Look at some site search, what are people looking for. It might be because they can’t find it. New versus returning.

Navigational review is one. And I use a lot, because in a lot of cases you’re trying to find out, so, why are people exiting here? Or, sorry, we want people to do something specific, but what is it that they’re doing? So with navigation summary, for example, you can go in there and just see, where are people going after they’ve been to this page? In a lot of cases, that actually tells you what it is that they’re looking for. So that’s another little sneaky one.

And of course you learn to use advanced segments. Very, very important. Changed my way of working.

But I’ll just walk you through the process now. So what I’ll do is I will have analytics in one browser, and then I have the website in another browser, and then I start digging for data, start trying to understand what you call the conversion scenario.

So this was one we did for a company called Zespoke in Ireland, and they do bespoke furniture. You order it, you chose your own design, and then they hand craft it in the UK and they send it to you, right?

So this is the home page. So just doing a little heuristic review here, I’m trying to walk through the whole funnel and understand it, and I’m trying to, you know, be aware a little bit of how am I reacting to all the things going on.

So first of all, of course, is a huge page. There’s so many products. And to me, that doesn’t really make any sense. It doesn’t seem like they’ve really thought about it. They have a slider that just has 10 slides or something, and some of them say “click here,” and you can’t click it. And some of the pictures are really, really, really, really bad. It’s just, like, 3D generated stuff.

Free shipping is not a button. Free samples is a button. What are the free samples? I don’t know. It’s very confusing. Free shipping is an awesome, awesome, awesome value proposition, but it’s kind of like it’s drowning there.

Another thing is it’s hard to identify what kind of furniture this is, and what does this mean, and why do they have all this stuff here, right? So, for example, the search bar is very, very small. If you click on the navigation, it’ll still be this one that is highlighted, so you can’t really understand where you’re going.

So there’s tons of things. And we’re writing all these things down and saving them for later.

Another thing I’m doing is I’m writing the URLs down as I go. Okay, so next page. That’s a category page. Another slider. And they’re using, like, four slides to explain something that you could explain in just a few bullet points, right? So that’s taking up a lot of space. It’s confusing.

We have misleading copy here, because you don’t add to cart, you read more. So I would definitely make sure to change that.

Some of the same comments as on the home page. Writing all this stuff down. Go to the next page, the product page. Now it starts to get interesting. My brain did a number on me when I got into this one. I simply . . . it took me so long to decipher it. It was . . . what are all these things out here? They’re actually images. All of them are images. And, well, this one is one image. That’s an image, that’s an image. This is one big image.

And this was just deeply, deeply confusing. Up here, you have a price, and then we just have a whole lot of interaction stuff here. So it takes you a long time to decipher everything.

And down here, this is a very, very important element. This is actually a little interactive editor. You can put in your different colors there. You can see the design you’re doing. But it’s all the way at the bottom of the page. And they have the video that’s, like, the instruction video, it’s also at the bottom.

So it’s very, very confusing. It’s a long page, weird hierarchy here. So this one is one I’m going to put a note on. I think this one is very, very interesting to return to.

So we found a very, very interesting bug, also. You’re supposed to choose the different colors here on the drop-downs, right? But if you go down first and use the interactive designer that you’re supposed to use, and you go up and you try to choose something, well, then it’s gone, right? So if you do what they actually want you to do, then your options disappear. So that’s a serious bug.

And what do we do? We’re just going to fix that. I mean, are we going to test that? It’s like, testing whether having a door gets more people into the shop, you know? Oh, I’ve got to hypothesize. If we had a door, we’d get more in. Let’s see what happens. It doesn’t make sense. It doesn’t make sense. That’s a no-brainer. We’ve got to fix that right away, as soon as possible.

We get to the cart. A quick run-down. It’s not horrible, but I would question a little bit about the buttons down here, and I think there’s a lot of white space going on here. I don’t really think that this is a terrible page, so this one is kind of, I’m not going to spend too much time on that one.

We get to the checkout. This gets interesting. Now we’re really getting somewhere. Okay, so this is actually three steps in one, and probably the idea is, the more steps you have, the more people disappear from the website. Not necessarily. It’s all about perception. If you see something like this, it can be very, very stressful for the brain. There’s a lot going on here. There’s a lot [inaudible 00:28:16]. So I’d probably want to simplify this.

Another thing is when you start going, filling out your information — and I’m from Denmark, so I want it sent there — it gives me all these options, which is great. It actually knows the cities in Denmark. I’ve never, ever seen this before. So I’m happy. I’m enthusiastic.

And then when I fill it out, pretty much what it says here is they can’t send it to Denmark. It was like, that’s pretty disappointing. And it says, “Contact us.”

So I click “contact us,” and so, what? This is where I end. And so this says something about getting color . . . I can’t even read it here. It says, “If ordering color swatch samples, please provide your address to be posted to,” exclamation point. I have no idea what that means. They have a great little caption on here. And then I can continue. So this is really a dead end. That’s just weird.

Then we get to the PayPal payment. I can’t do anything here, so why bother with that. Then we get to the confirmation page, the thank-you page. It doesn’t really confirm much. So I don’t feel like I’ve really gotten a confirmation. It doesn’t tell me anything, what happens, when is it, when am I going to get my stuff, how long is it going to be, and so on.

And then they have this security seal here. It says, “Secure your purchase now.” Secure my purchase? So I’m not sure I’m going to get it? I have to do something else with a third-party tool to make sure that I get my purchase? That does not make me feel very confident, okay?

So now we’ve done the heuristics stuff. We’ve done a little bit of analytics. We kind of know the situation. So now it’s time to go through the funnels and find out where the real problems are.

So I was pretty sure that the check-out was the worst thing. That was definitely what I would go for. But what we found out was that the biggest problem right now is the product page. It’s very surprising. Actually, the check-out worked fine right now. Amazing. So it looks like we have some very, very dedicated users actually getting to it and then getting through it.

So we should focus on the biggest problem right away. So this, go back to the product page. So basically, this is a very, very, very confusing layout. The information hierarchy is way off. So that’s my basic hypothesis. I think we can rearrange all this content here in a much, much more intuitive way for the users.

And also, another consideration is that they don’t have a huge budget, these guys. So what we’re suggesting to them, we can’t go, like, all crazy and build a highly technical new responsive website for them. They can’t really do that, okay? So we have to stay within what they can do.

So this is what you see above the fold. This is very, very confusing, right? You don’t even really know what you’re doing here, and it’s just a lot of different tabs you can work with. What does that mean?

So the overall hypothesis is that by rearranging the product page content, we can create a more logical user experience, we can reduce cognitive friction, and then we can be able to configure the product and increase purchases. Okay?

So I go straight to a wire frame from there. It’s like, we don’t want to do it too quickly, but we also don’t want to lose momentum. So go straight to visualizing it. Does this seem like something that might work? Yes, it does. It might be a good way to do it.

Then I go straight to a mock-up, and this is basically what we came up with. So we’ve got the price, we’re saying that there’s free shipping. We’ve got some reviews. There’s 60-something. We’ve got the description here. We’re telling them all this stuff about it’s handmade in the UK, the value proposition, all the information about sizes and everything.

Over here, I’ve set it up into a little picture slider here, so it’s very easy to understand kind of what the pictures are and how you can change between them. We’ve got the editor here, because it makes no sense that you have it at the bottom of the page, because you need it right here next to the options.

So probably the optimal thing would be is that when you’ve created your design here, then it fills out the tabs, and you’re ready to go on. Right? So this is what you would see above the fold, okay?

So now we have a treatment. And I’m not necessarily saying that we have to implement this one. This was pretty fast. But now we have something we can keep going with. We can do some further research. We can maybe do a user test. We can do session recordings. We can do all kinds of stuff to get it on there.

Next thing is we’re probably going to open the other steps, also. We can go back to the home page. So what should we put on the home page? Well, we can guess, or, well, we can go to ecommerce, and we can see what are the top selling categories, what are the top selling products? And now we have great ideas to put in there.

Do a little wire frame, and now, and visualize it. We can help the client understand what it is we’re trying to do. Okay?

Another thing when we’re doing funnel analysis. This is an example from another website review we did, okay? So this was a SAAS product. And so they had the typical thing. You have the home page, you have the pricing page, you’ve got the form, you get the confirmation, you get the email, and then you get the success page. And you can log in. You’ve got your account set up.

So what we saw when we went through analytics was that there were a lot of people who never got from the email to the confirmation page and logged in. I’m trying to figure out why.

So it all had to do with the email. So we want people to go through clarification. That is this link right here. All the other links go to other pages that we don’t want them to go to. And that’s because they’re using one general template for everything.

So that’s a major, major mistake. So we’re losing all these people who are going to the login, and pretty much what it says is, “You need an account to log in.” And you’re like, “Well, I just did it. I just did it.” So that’s really, really horrible user experience.

So we need to fix that. Money is pouring out, business just is pouring out. So we can fart around with the homepage forever. We can tweak headlines. We can tweak buttons. But we’re never going to get anywhere until we fix this major hole. And again, it’s a big problem on mobile here, even more.

So basically, what we need to do to fix it is just simplify the email. Take out the link, have one big button you can click. Okay?

So funnels like this, I spend a lot of time on. They’re very, very important. When you [inaudible 00:34:23], you can easily find out where the biggest potential for lift is.

So this is actually a real example from a client. So an 80% drop-off between Step 1 and 2. It’s pretty clear where we’ve got to start working now. The second biggest one is between Step 4 and 5, so that would be the next thing I would start working on here.

A little trick. If you don’t have funnels set up, you can hack them manually. So you’ll type in the URL. You’ve gone through the funnel, you’ve took the time, typed in all the URLs. You can find them if you go to All Pages Report. You write them down. You write down the unique page views, and you do that for every step, and then you have your funnel. Very, very quick hack.

So basically what we’re doing here is plumbing. It’s plumbing. So we’ve got the conversion scientist over here, and I’ll just be the conversion plumber. It’s not very sexy, but, I mean, I’ve come to terms with it. I’m not Don Draper. I’m actually just the plumber. Okay.

So when we’re talking about funnels, and just talking about the whole experience, it’s one thing that’s important. This is also from Kahneman. I love this quote. He says, “In the economy of action, effort is a cost. Laziness is built deep into our system.”

This is very, very important to remember. Make it as simple as possible for people to do what you want them to do to get through your website. A lot of websites, they look like this, right? And, yeah, of course, so this is a checkout. This is the biggest obstacle. We want that, save that for the last, when people are really, really exhausted, their brains can’t handle it anymore, then we have to get them to crawl through that, right?

So this is basically what I feel like with most websites getting through it. Oh, you’re fighting, and when you get out the other end, you’re like, “Yes. I did it. I’m a champ.”

We want the opposite to that. We want people to not quite even notice they’re getting through that.

So whenever you make me think little things can really slow me down. Really, really slow me down. I want to show you an example here. Just, this is kind of a cute little example I found the other day. I was going through a travel website, and another review, and we were having some trouble here on this page.

And so one of the funny little things I noticed here, so you put in your name, put in your email, and then it says, “Please have a special agent call me.” And for some reason, the first thing I thought of was just, “Daddy,” right? Or you could call me Al, maybe, right? So that was really, really confusing. What the hell does he mean? Any questions? Can I really choose what you want to call me?

So when you get the error message afterwards, they’re asking for the phone number. But little things like this accumulate. So you need to get rid of those. And other things like, for example, a page like this, where it just is impossible to multi-task in this way, and to understand everything on the page. This is a big, big killer, in my experience.

Okay, so one thing that’s really, really important in order to actually be able to work with hypotheses, and get anywhere, and be able to validate them, is that you need to run cling tests so that you actually are getting insight, right?

So here’s an example of a test I ran on . . . it’s a lead gen site, and they do shipping requests, okay? So you go through the form, and then you get, you write down what you have to get shipped, and you get some free offers sent.

So what I did was, I hypothesized that by removing the progress bar, we would actually hurt conversions, okay? Because I’ve seen progress bars all over the place. People talk about them being good. But let’s see, actually, and test them for ourselves.

So I took it away, I ran the split test. So look at the average. I ran it for nine full weeks. We had 1500 conversions, and we have, well, a 4% drop in leads, but it’s not really significant.

What did we do? This is, we spent nine weeks and wasted our time. Well, it’s because we only looked at the average. It’s very, very important to actually segment and know what’s going on in your test, right?

So on desktop, as I expected, we saw a drop of 9.8%, okay? Wow, that means it’s very important we have the progress bar. It was the opposite on mobile. On mobile, when we took it away, we increased conversions. So if we’d only looked at the average, we would have shot ourselves in both feet, right? Because we’d really, really hurt ourselves.

So this is super important. And the thing, actually, on mobile here is probably not that a progress bar doesn’t work. It’s because it was responsive, so it got squished together. So instead of being a progress bar, it just said one, two, three. So that’s a weird user experience.

This is very important. You have to integrate your test data in your analytics account, and you have to segment. You have to be able to know and get the full picture.

Okay, Andre was talking about quality of research before. It’s very important. I used to not spend much time on it. I found out that it’s one of the best sources of information, one of the best ways of building hypotheses.

So I used to do a lot of customer interviews, and then I started dealing with sales and customer service. But actually, what I spent most time now is actually talking to sales and customer service. These guys have so much knowledge. They spend days and days and days just talking to people. They understand all their problems. And actually, nobody wants to talk to them in companies. So these guys are so happy when you actually listen to them, right?

So I have basically five very simple questions that I ask them. There’s just a slight variation between sales and support. So I would say one of the first questions I ask is actually, “What are the top three questions you get from potential customers?” That means we’re not answering them on the website. We need to know this.

When I got those . . . actually, sometimes I’d spend half an hour talking about this, and then they actually start answering the other questions. But this is the process.

So this, the follow-up question would be, “What do you answer when you get these questions?” These guys spend so much time explaining and solving questions, solving problems. So all the answers they give, I can write them down and can use them on my landing pages for my hypotheses.

So, number three, “What is the biggest barrier keeping people from buying the product?” We need to understand this. This is essential, essential stuff. And then, of course, “Are there any specific selling points that work particularly well? If so, which ones?” I can take these directly, use them on my, in my treatments.

And then I’ll round off just asking them, “Is there anything important that we’ve missed? Is there anything more you want to talk about?” And usually there’s a couple of really good points we get there.

Customer service, slight difference. Same, the top three, what do you answer. Then I say, “Are there any particular aspects that people don’t understand? What aspects of the product do people like least and most?” Right?

So it may seem like a very basic exercise, but it’s incredible how much information you can get out of this. This is the why. This is how you get into the minds of your prospects.

I mentioned before, you need to ask yourself whether the hypothesis that you can test on your website is real. Very, very good asset test is to do little calculation like I showed you before and just see how long it will take to run the split test, and does it make any sense for us. Right?

Okay, rounding off. Test hypothesis only . . . your test is only as good as your hypothesis. So the better the research, the better the insight. The better the insight, the better the hypothesis. The better the hypothesis, the better the test. And the better the test, the better the learning. So this is a whole process you’ve got to remember.

And so, final closing remark, don’t be a split test junkie. Be the conversion plumber. That’s how you win.

Thank you.

Join the Conversation Add Your Comment

  1. Can we get the a link to the slide deck as well? Right now it’s Michael being entertaining and fired up about stuff we cant see.

  2. Yes, slides would be extremely helpful. Can’t really understand what’s going on otherwise.

Comments are closed.

Current article:

Your Test is Only as Good as Your Hypothesis [Video]