Research-Driven Conversion Optimization Video Course

Free Course

Research-Driven Conversion Optimization

The free data-driven conversion optimization video course by CXL

video
59:31
 
Have you been getting inconsistent results from conversion, optimization or A/B testing, or maybe no results at all? It's probably because you're doing it wrong. (uplifting instrumental music) Maybe you're copying random tactics from a blog post, maybe you're copying competitors or market leaders or maybe just following best practices. That's not how you optimize a website. There is a process, a framework for conversion optimization that is industry agnostic so it works for SaaS or E-commerce, B2B, media, you name it, it works across the board and gets you insane results and I'm going to teach it to you.

Conversion rate optimization is a way to make more money.
But how do you do it? (uplifting instrumental music) What is conversion optimization, or as it's usually referred to, conversion rate optimization, CRO? Conversion rate is a mathematical formula. So if you have 50 people buy your stuff or sign up to your email, but 1,000 people came to your website or a landing page, whatever it is, you just divide it to two and you get oh, I have a 5% conversion rate.

And then of course, in order for us to make more money,
we need a higher conversion rate. They say that conversion rate optimization is all about increasing the conversion rate so it will be 6% or 7%, but I kind of disagree or rather I don't like that terminology. Actually, I don't like the R in conversion rate optimization because it's not about the conversion rate. Let's say you build a brand new website, you just tell your mom about it and she goes to your website, one person, and she buys something.

So one visit, one purchase, 100% conversion rate.


Amazing, right?
Yet you just sold something to your mom and that's it, that's not a business. So conversion rate optimization is not about the conversion rate. Another way you can increase conversion rate if you want is like make every product you sell free or one cent. So let's say you build a new Amazon, sell everything that Amazon sells but everything is free. Your conversion rate will be insane. Everybody will purchase something for free, but you'll go out of business. So optimizing just for conversion rate is silly.

Don't do that.
So conversion optimization really is about growth. It's about growth. And what is it that we need to do to our website so we could grow our business sustainably and profitably? Now there are multiple ways to do this, and traditional way is we all have opinions. Hmmm, I think we should do this, we should make this button bigger, we should add more image sliders or (mumbles). Everybody has an opinion what should be done, and opinions are like assholes, everybody's got one and they all stink.

So opinions is not a way to optimize anything.
I've been doing this conversion rate optimization for years. 10 years. Now if I have to predict which change will result in more money, I get it right maybe around 60, 65%. That's just slightly better than flipping a coin! Basically random. You can't have an opinion about these things. Well, you can have an opinion but the odds that you're gonna actually get something right, oh, the odds are very low, it's like playing the lottery.

So that's not how conversion optimization works.
So the main question is what should we change? You know, because in optimization, we wanna change something about a website, add something, remove something, modify something so we would make more money, so more people would buy your stuff or you know, become a lead or whatever, it is that we're trying to accomplish. So what do we do? Now if we agree that opinions are the wrong way to go about it, the right way is conducting research, We need to figure out what are the current problems with our website, where are the problems and why these problems are problems to begin with? Only if we understand where, what and why these problems exist, we can come up with a better idea, an idea how to fix them.

We call this treatment.
We apply a treatment, maybe we do an A/B test, maybe we just change something on the website. So imagine you're going to the doctor's office. Let's say you have abdominal pain. (groans) And your doctor says, oh very well, please lie down here, we'll get the nurses, we'll just cut you right open and you know, like perform a surgery. And you're like whoa dude, I just got here and like, that's opinion. Like I have an opinion, it's your you know, whatever, your liver, we need to perform the surgery right here; whereas you would expect that they'll run a bunch of tests, blood tests, you know, CT scan and you know all that stuff, like they need to conduct research to figure out what's wrong with you and then prescribe a method of treatment, right?

So this very same thing on the website,
yet we jump to having just opinions and we wanna change stuff on our website just by looking at it. Doesn't work like that. So we need to conduct conversion research. And in the following videos, we're gonna dive right into how to do this. And so many people also ask me, so what are the super killer ninja secrets about conversion rate optimization? What are the magical things that always work, maybe some psychological trickery? Well, no. There no secrets, there's no magic in here, it's purely hard, hard work.

So as I see, conversion optimization consists of two parts.
It's doing research to understand the problems, to understand the problems, and then from the problems we come up with hypothesis

about what should be changed to address this problem,
to fix the problem. And then we don't know what will work, we'll come up with four, five, 75 ideas for treatments and then we'll need to run experiments to see which of these treatment ideas actually works and which one works the best. So it's research plus experimentation. So stop thinking about guesses, think about data-backed hypothesis and that's how you optimize websites.

Let's say that tomorrow you're hired
as a conversion optimization manager for a big company and they give you the goal of increasing their conversion rate 20% per year. How would you do it?

So if your answer about improving the conversion rate
of a website is instantly tactic coming, oh make the headline bigger, put the button here, (mumbles), you're doing it wrong. It's immediately how you can recognize somebody who's an amateur versus somebody who's a pro because pros focus on the process. As one of the management gurus, Deming said 60, 70 years ago, if you can't describe what you're doing as a process, you don't know what you're doing.

The same applies to conversion optimization.
So the process for any website is the same. If you're trying to optimize an E-commerce site, a lead gen site, media site, a SaaS, doesn't matter, the process is always the same. The tactics might be different, and they often are, but the process is the same. So largely speaking, conversion rate optimization process always starts with conversion research. You figure out what the problems are, where, why these problems are problems. Then once you have a list of problems, you turn these into hypotheses.

So hypothesis is if we change,
based on the data X, Y, Z, if we now change this and that, we expect to change this metric. You know, more people will do that, more people will do this other thing. So you have a list of hypotheses, you turn these into tests, into experiments, run A/B tests. And once the tests are done, you analyze the results and maybe you implement, maybe you discard and then you go back to step one, which is conversion research.

It always starts with research.
But how do you conduct research? There are multiple ways to go about it, there's no one right way to do this. I would recommend you start your conversion research process by adopting ResearchXL framework. ResearchXL is a framework that I came up with around 2013, and we have used it on hundreds and hundreds of websites over the many years since. And it works really, really well. So ResearchXL is, essentially, what it does is it uses six types of data.

So it gathers data input from six various different sources
with the goal to analyze what are the problems, where and why. One of the the first steps in this process is always technical analysis. So technical analysis means that we wanna figure out whether the website in question works with every single device, every single browser and a combination of the two. So if I'm on a random Huawei phone on a random whatever browser, the website should still work.

Because you can have the most persuasive website
in the world, but if it doesn't work with the specific browser, device combination I am on, it doesn't matter, right? So it needs to work. Recently, at the CXL Live Conference, there was this big case study where Dell launched a brand new website, a brand new website that they had been working on for a really, really long time, a lot of money was invested in it and they rolled it out to a small percentage of the traffic first to be sure that it's better and measured it, and revenue per visitor dropped and they're like what, how is this possible? And then they looked at their website, they talked about it, oh, like we need to make some UI enhancements here, we need polish the design here and make this process a little smoother.

So they implemented 20-plus changes
and ran a new test. The result? The revenue per user now dropped 45%. They're like, what the hell is the problem? And only later they found out that all of this was because of technical bugs. There were a bunch of JavaScript errors in their check out, things didn't work. Websites not working is the biggest conversion killer there is and is also the lowest hanging fruit. If currently you have some browser segments or devices that are underperforming, fixing those bugs is the easiest, the fastest way to more money.

So here's what you should do.
In your digital analytics platform, let's say Google Analytics, you pull out the browser report and you wanna compare the conversion rate per browser version within the same browser family. So for instance, if your website converts at 5% for Internet Explorer 11, but only 2% were Internet Explorer 10, that's like more than two times difference. Why? It's because of some nasty bugs or some whatever weird UX stuff. So now you need to go investigate it, figure out what is the problem.

If you don't wanna do it yourself,
there are QA people out there, quality assurance people or companies you can hire that will go in and do this technical testing for you. But basically, you wanna understand which browser versions convert less than some other versions. And same for devices. Of course, mobile typically tends to convert much less than desktop, but this is really dependent on what you're selling. In B2B, for expensive products like you know, we're talking like 10,000 a year or more or anything around thousands of dollars, mobile traffic doesn't really convert.

People are still not used to spending
a lot of money on mobile. Yet when it's like trinkets, small stuff, the conversion rate is very high, especially if people have transaction, they already know what they want, they'll go to your website and buy on a mobile phone no problem. But on average, we're seeing that mobile converts around 25% of what desktop does, and that's just a broad average, means nothing (mumbles) particular website. But tablets typically convert the same as desktop.

If you think about it, tablets are not mobile devices
because 90% of tablets sold are Wi-Fi only, so you can't use it on a bus, only at home, and you're probably sitting on a couch at home while using your tablet, right? So it's kind of like desktop and it's big screen. So if you see your tablet converting less than a desktop version, the tablet experience is probably suffering, there's something going on, it may be bugs. And of course you wanna actually eat your own dog food and open up your website on your own mobile phone or desktop, you know, both, and just walk through your website like a regular human being and check out and complete a purchase or whatever the thing is that you want people to do there.

So technical analysis, figure out underperforming
browser segments, device segments. Another thing about technical analysis is site speed. If pages are too slow, the people you know, might leave and not buy as much. So again, in Google Analytics, you can look at your page speed per URL. So you know, there are all these various page speed tools out there, and they're great, you should use them, there's one by Google called PageSpeed Insights, there's GTmetrix, various different kinds.

If you Google site speed analysis tools,
you'll find a ton. But bear in mind that if you just put in your homepage, they will only analyze your homepage, not your entire site. Your website has probably lots of pages. If you're an E-commerce site, you probably have you know, hundreds if not thousands or tens of thousands of product pages, right? So you wanna analyze your top 50, top 100 web sites

with the most traffic and see what the average
interaction time is. And notice I didn't say page load time. So what is page load time? Page load time is really how long does it take for the page to be completely loaded? So if it's a long page, meaning most of it is below the fold, we as users, if we're still looking at the above the fold area, we don't really care how long it takes to load the rest of the page there, even if it takes you know, 15 seconds, we just don't notice it. What actually the users care about is document interactive time, which actually means how long until the website becomes usable? So basically, the above the fold area.

Renders and we can click around
and interact with the website. That's what really users care about. So ideally, it's three seconds or less. And if you have high-volume web sites, those small differences might start mattering. If it's between say, four to 10 seconds, you know, it's kind of the average but you should get closer to three seconds or less. But if some pages are 10 seconds or more, you can be sure that this is hurting your conversion rate on those pages.

So typical culprits are too large images.
You know, like your images are like seven megabytes and people are on a known Wi-Fi network, takes a little while to load, or you have too many different CSS and JavaScript files so there's a lot of back-and-forth between the browser and the server and that all slows your site down, it's very frontend development heavy. And if you're not a frontend developer, you probably are not able to fix those issues yourself, but you can figure out which pages are too slow and send that to your frontend developer and say hey, there's a problem here with the site speed, with the page load speed, fix it.

So that's technical analysis.
This is always the very first step in conversion optimization. Once you've done that and you've fixed all the bugs and you have no more slow loading pages, it's time to move on to step number two, which is heuristic analysis.

Step number two in our conversion research process
is heuristic evaluation, heuristic analysis. (uplifting instrumental music) Essentially, what it means is it's an experience-based assessment of our website. So we're gonna conduct a walkthrough of a website page by page, we're gonna walk through the site like an average user. Desktop and mobile separately because you know, it's a different experience, different things matter and so on.

And this exercise is best done in a group setting.
People from a team, people who've never used your website, actual target audience members, you know, up to six people. And you're gonna go page by page and you show that on a big screen or laptop if you don't have a big screen and basically, we're gonna assess every single page against a set of criteria, which are written down here. There are various heuristic analysis frameworks out there, and some have seven steps and some have 17 steps and all that stuff.

Essentially, it all boils down to these four steps.
And you know, you could have sub branches of these four, but essentially it's always about these four things. So what do you do? You're not gonna comment on websites like oh, I don't like the blue and oh, I don't like the red. You know, that's just an opinion. So we need to be more structured in our feedback, in our observations. So everything that we're gonna comment about a web page, so either it's a home page or a thank you page or card page, whatever it is, we're gonna do it in a structured manner and we're gonna try to categorize each piece of feedback under one of these four.

So first thing is friction.
Friction is something that makes it difficult to use or it's difficult to understand. So for instance, if they want us to fill out a form with like 27 form fields, that's a lot of work. That's friction, like ugh, I don't wanna do that. Or if the website has a kind of scammy-looking design

from like 2002, blinking banners and da, da, da,
it looks sketchy, that's friction, it's like mental friction. Like ooh, I don't know about this stuff. People need to feel comfortable at your site. If you're Amazon, you don't have trust problems. If you're a relatively unknown site, you're gonna have big trust issues and so friction is a big, big issue. Anything that causes mental friction or is difficult to do or I don't understand what I should do next or I'm trying to click this button and it doesn't work, it all creates friction.

Second thing is distraction.
So every single page in your website should have one primary goal. So you know, typically the goal of your homepage is to get people off the homepage, like down the funnel. Or the goal might be segmentation, like choose here, are you like male or female, am I a small business or large business, things like that. Or I'm interested in this service or the other service. So anything on that page that is not contributing to people

taking that one action is a distraction.
So if you want people to click a specific button or fill out a form or whatever it is but there are all these other things there, there's like oh, latest four, five blog posts and there's this would you also be interested in (speaks gibberish) and there might be, or worse yet there's something moving, there's an animated banner, there's one of those automatic sliders every three seconds, something changes or the worst yet, a video background, like with music, (sings random tune) stuff that's moving around.

So what happens is like our human brain
is designed to detect and follow movement. You know, like ages ago, this was a useful survival skill. Like we needed to see if like a mammoth is coming to trample on us or a predator or an enemy and so on. But today on a website, as soon as you have something changing every three seconds, it's a distraction. We're gonna just look at it and we're not gonna read the value proposition. We're not gonna understand what your website does and why we should buy from you.

We're not gonna fill out as this form.
So anything that is not directly contributing to people taking that one action is a distraction. On a product page and E-commerce, the goal is add to cart. So anything in there that is not contributing to that, better remove. Motivation is more important than distraction and friction. So motivation is about making people want to take action. So we talked about friction, 27 form fields, fill out this form.

(blows raspberry) Never.
Well, if you fill out this form, you're gonna get a free Tesla P90D Model S.

Only 27 form fill, I could do 127.
So the amount of friction is very relative compared to the amount of motivation. Now of course your product is not a free Tesla, right? It's something else. It's not as good, it costs money probably. But still, you need to sell people on the value of your offer. If you're selling a vacation, you want to make us wanna go there; photography, the product copy, all of those things matter a lot.

So your first goal when you're trying to get people
to take action, whether it's to click somewhere, add to cart, fill out a form, is to make them wanna do it. Or not make them wanna fill out the form but make them yearn for that final outcome that they're gonna get. If you're gonna give us money, you're gonna get six-pack abs and all that good stuff, like of course if you only diet for two years and exercise. And then finally, relevancy. So relevance is that you can have an amazing value proposition and really short forms and like beautiful, beautiful design and minimal distraction.

But if it's about fishing equipment,
I personally couldn't care less because I'm not a fisherman. So the relevance is so important. So you need to be absolutely clear who this is for. Who is this for? The higher the relevancy, the higher the odds that I'm going to convert. So you wanna make sure that you address the fact that if I am your ideal target audience member, target user, why should I buy from you right now? So build that path so people would read it and recognize aha, this is for me.

And if you're using ads to drive traffic to a page,
of course you want to make sure there's continuity from the ad copy, as well as design, if it's a visual ad; and the landing page, copy and the visual, the way it looks like, it needs to match. So if you have all these four things in place, you're good to go. So you go through your website step by step, every single page that matters, mobile, desktop separately and just write down your observations on every single page.

The important thing to note is that these observations
is not the absolute truth. So 50% of your observations are probably going to be irrelevant, they actually don't matter. But what follows next in our conversion research process is we're gonna gather qualitative, as well as quantitative data to either validate or invalidate our observations.

Digital analytics will show you what is happening,
where and how much.

(uplifting instrumental music)
Third step in our conversion research process is digital analytics. So for most people, it's Google Analytics, but you can use whatever other web analytics tool that you know, is a better fit for you. Number one thing in analytics is always is everything being measured? So every single thing that a user can do on a website and every single thing that a user can experience on a website should be tracked and measured.

So for instance, let's imagine that
we have an E-commerce site. So what are all these things that people can do? They go to the homepage, maybe there's a slider there and they can manually click through different images. Every time they interact with the slider, we should mark it down. We should fire an event. That's what it's called in an environment, let me fire an event when they're interacting with a slider. They're scrolling down. Oh, how far down, 25% of the page, 50% longer? We record that automatically. There's a YouTube video maybe about a product overview.

Are they watching the video?
And if they are watching the video, how long? Did they watch everything or just the first two seconds? You need to measure that. They're searching for something. We should measure that. They go to the category page, there are all this filters so you know, size, filter by size, price, color, blah, blah, blah. Are they interacting with the filter? Which one? If they click add to cart, we need to click that. And obviously, all the purchasing data, the full funnel, they go to the cart page and all the multiple checkout steps, everything needs to be measured.

Because if things are not being measured,
we can't improve stuff. So let's say that you have a product filter for, filter products by color. Now how many people are using that? And if they are using it, what is the impact on user behavior? Like, are they more likely to purchase, less likely to purchase, no difference? And if like 1% of the users are using that filter and if they are using it they convert worse, it's probably a good idea not to have it to begin with.

But you wouldn't know that if you wouldn't track it.
Also, if you want to optimize your product page so more people would click on add to cart but you're not measuring specifically cart adds, you can't optimize for it because you don't know if the change you made increase cart adds or not. So it's very important that we measure everything. If you're using Google Analytics, then all this measurement you can set up at Google Tag Manager. And it's not difficult at all. In fact, if you're a marketer, you need to be able to use Google Tag Manager on your own, and you can set up all this tracking without any developer involvement.

There are other tools that record more stuff out of the box
like Heap analytics for instance, measures everything that needs to be measured right away automatically. But you know, as soon as you go above 50,000 page views a month, it becomes rather expensive. So first thing, make sure everything is being measured. Everything that is important for you. Number two, the data that we're measuring, is it accurate? It's so often that the data that, the funnels that have been configured and so on, it's actually not true.

Sometimes you see websites where they have
really low bounce rate. For instance, a bounce rate is like 1%, 2%, 3%, people are like (chuckles) I'm the boss, look at my bounce rate. Whenever you see this, like no, this is called broken measurement. So if you have the GA code loaded twice on the page, your bounce rate will be off and below 10%. If you have an event that is firing and it's not set to non-interactive, engagement is being recorded, again your bounce rate is artificially low.

So those things can screw up your data.
Also you see all the time where people have a five-step checkout funnel and it shows the kind of percent people go through all the steps. This never happens in real world so it's broken. Or the final purchasing count or the revenue does not match what we're seeing on the backend in our content management system or on our E-commerce system. So you need to verify that all these things are active, that they make sense.

When you look at the data, like on average,
people add 77 products to the cart. Really? I don't think that's accurate, right? Whenever you see something that's fishy, it's like it's probably not true. Also you see off the revenue being double counted. Or if people for some reason can reload their thank you page, which they shouldn't be able to do, again the transaction is loaded multiple tires, it can inflate the revenue and all these metrics. And then there are also things like you're using subdomain.

You know let's say your blog is on a subdomain
or you use multiple domains like your shop.domain.com and blog. and then you have

your main domain.
If people navigate between those sub domains, the sub domains you're tracking implement it, or every time they switch between sub domains, maybe it shows up as a new visitor even though it came from a paid Google ad. So that attribution gets lost if subdomain and cross-domain tracking are not properly set up. So it's very, very important. If you cannot trust the data, you cannot be data-driven. So these are very, very important first steps. Now assuming that everything is all right, we can trust the data, everything is recorded, now for conversion optimization purposes, digital analytics helps us identify three very important things.

Number one is where are the leaks?
Every single page on your website is leaking money. Meaning users are dropping off, they're leaving your website. And you have you know, some sort of a typical customer journey, a funnel, and so you wanna understand in which of these funnel steps people are dropping off the most? So typically, let's say on E-commerce cart page, 50%, proceed to checkout. So if you're in your site, it's like 20% to 30%, you have a big, big problem with your cart page. Or on your product page, typically an average site is like 10% of people add something to the cart.

If for you that's 1% or even less,
something's wrong with your product page, either you have the wrong people on the site, a relevance issue in your checkout. So what is the checkout completion rate? Should be like around 90%, which would be good. If only 20% of people finally put in the credit card and finalize the payment, again it's probably the checkout that has the problem. Of course these are all hypotheses, we don't really know but we can, we don't really know what the problem is but we can see where they're dropping off, very important.

And again, you have to look at this
across devices separately. So mobile separately, desktop and so on and so forth. We also wanna see, look at correlation. So the people who are buying something, what other behaviors are correlating with high purchasing rate? Now what we you don't know is like is it that people who were gonna buy anyway use site search or they just know what they want? Or was it that using the site search helped them find more relevant products, which increased their likelihood of making a purchase? So if that is the case, we should try to get more people to use the search and we will make more money.

Make the search box maybe more prominent and bigger
and so on and so forth. So we wanna understand what are people doing or not doing and how does that correlate with conversion rate or revenue per user? And of course, we wanna always segment all the data per traffic source, per device, any segmentation that might make sense for you. If on your website people are logged in, you might also be able to segment per gender. Or if it's B2B, by revenue, business type, what they do, all those things.

So very important.
What's a quick and easy way to figure out what people are doing or not doing on your website? Heat maps.

There's another category of analytics tool
usually referred to as customer experience tools or mouse tracking tools. Essentially, what they do is they give you a scroll map. So it's basically a visual representation in a form of the heat map to show how far down people scroll on a given page. So typically, what it tends to look like is you have a page like this and then at the top, it's like all reds, a lot of people are using this scrolling, scrolling, scrolling and then it gets yellower and yellower and yellower, and then it's blue, like nobody's scrolling here.

And what's interesting about scroll maps
is like where is this transition here? Where is the transition where the red becomes kind of blue? And this is important because sometimes important content is in this area so most people don't see. Or another way why this is important, we wanna understand where is this moment where people are dropping off? It's usually something about the design of the page that makes people stop scrolling.

It's typically a change in the background color.
So maybe here the background color was white and here is suddenly blue or black or whatever. So dramatic background color change. So people assume that this is where the page ends or that the content below is not relevant to the content above. So once we see this, there's an ah, look at this line here, let's remove that, let's have a uniform background color and scrolling can improve. Or you see that nobody sees this important value proposition, let's move this above the fold.

So scroll maps, really great way
to figure this kind of stuff out. Click maps, click maps basically show you again a heat map of where people are clicking on your page. You can of obviously see this data also in Google Analytics, but the nice thing about showing it in a heat map form is A, you quickly see where people are clicking or not clicking. And executives love this kind of reports so it's very easy for everybody to see a click. And sometimes, something that you really want people to interact with does not get any love at all from users.

Maybe it's not prominent enough,
maybe other things are more prominent. And also, you can see things that people are trying to click on that is actually not a link. So maybe you have something that looks like a button. Maybe you have an image and people assume that they should be able to click on that image and go somewhere, that it should be a link but it's not, so they're like clicking, sometimes they're rage clicking like (grunts) and nothing happens. So then it's like oh, kind of make that into a link. Then there are hover maps that'll like basically show where the mouse cursor has been moving, some call it attention maps.

That's completely useless, it's kind of a scam
that these vendors are selling you so don't trust that all because like the premise is that people look where the mouse cursor is so it's kind of like cheap, poor man's eye tracking but it's BS, it's not. Like imagine, think about the last time you read an article on a website. So with your mouse cursor where you're doing like this, like a reading line by line? Didn't think so. So we are basically just scrolling down and we're reading with our eyes.

So this is completely useless, don't do it.
And finally, good useful feature they have is session replays. So anything that users are doing on your website is filmed or like recorded as a clip, a video clip that you can playback without audio. So you don't know what they're trying to accomplish, you have no idea about their intent or what are they experiencing, but you can see what they're doing. So usually if in digital analytics I see a page where a lot of people are dropping off, they're like leaving the site, I wanna watch some videos of what are people doing or not doing on that page.

And often you find something really, really interesting.
So I had this case where it was a five-step funnel, they had to fill out an online resume. And in step three, massive drop off. After they had filled out two long forms on page one and two, it's like they already invested a lot of time in this, why are they leaving the site in step three? So I watch session replays and what I found was that there was a question on page three that ask you have to name three references and it was mandatory.

People didn't have them, so they left the site.
And thanks to video session replays, I understand where, like what specific question was the problem, took it out, conversions went up.

Why aren't more people buying your stuff?
And when they see your offer, what are they thinking? (uplifting instrumental music) Next step in our research process is qualitative research. So when quantitative research can answer questions like where, what and how much; qualitative research answers the question why. Why users behave this way or that way, why did they do this or that or why didn't they? It's not perfect, we can't answer everything with 100% accuracy, but to me this is the most insightful part of any research process because we're talking to our actual users.

This is so, so, so important.
Now there are multiple ways how to do qualitative research and we'll go over these real quick. Number one is customer surveys. So you wanna survey people who just bought something from you or they signed up for whatever, something, your software for a free trial. So now you wanna survey them, you wanna send them out a survey within like 24 hours, 48 hours of them purchasing something while they still freshly remember their purchasing experience.

Because if you survey them 12 months later,
they don't remember the details of your website or what went on, they're gonna bullshit and lead you astray. So you wanna survey them fresh. You can use any survey software, whether its Typeform, Google Forms, completely free. It really doesn't matter. You send the survey over email and you wanna ask open-ended questions. No multiple choice. You know, you can use multiple choice for segmentation if that matters, like male, female, your age range, something like this, whatever is relevant for your business or maybe none.

You don't wanna ask multiple choice
because that will assume that you already know what the possible answers are. So what do you want to ask? So one is you wanna ask about the friction in their purchasing process. You wanna ask what was the one thing that nearly stopped you from buying from us? You also wanna ask about their motivation. Like, what kind of a problem where you're looking to solve for yourself? Of course if they bought a pair of pants, might have been lack of pants so you don't need to ask that. But if they're shopping for a SaaS tool, it's different, right? There's a problem, there's a solution, there's a use case and you wanna understand their motivation, the user intent.

You also wanna understand how your offer
compare to other offers. So how many other websites did you check out before deciding to buy from us? What made you buy from us and not these other guys, to understand what, in their mind, was the competitive advantage? And also you know, I've done this type of surveys for years and I've never seen a case where people say oh, I never checked out the competition, I just bought from you. Always do their research, they always do competitive intelligence, they check out multiple sites. And it's very important for you to know how many different sites on average they're checking out, and what are those other sites so you can have a value proposition that stands out.

Because nothing is worse than sameness.
So customer service, very important. Another thing is on-site polls. So basically, this is serving people on your website who might or might not buy anything. These polls you wanna trigger when they're visiting one of your high exit pages, so pages where people are dropping off. And close to the money, so like checkout page, cart page, one of those pages. And so you wanna trigger a poll, a question, when they're leaving the page or when they've demonstrated above average engagement, meaning like they've spent on the page 20 seconds or something like this.

And so what you wanna do is you wanna ask only one question
and also open-ended like, and usually I like to ask what's holding you back from doing, whatever we want people to do on this page. So on a checkout page, what's holding you back from completing this purchase right now? Or on a product page, what's holding you back from getting this product right now? And people will tell you, it's wonderful. You can sometimes get a higher response rate if you start with a yes/no question.

That's like, is there something holding you back
from making this purchase? Yes/no. Because it seems it's really easy to answer so they click yes there is. Now then you ask them, type in what is the reason. So you can do that as well. So for instance we had this case where on a cart page on an E-commerce site, massive drop-off, people are not moving into the checkout and we're looking at the page and just you know, coming up with all kinds of hypotheses why that might be and then we ran this poll and 90% of people said it's the high shipping costs.

Just by looking at the cart page,
it was impossible for us to figure out that the highest shipping cost was the reason for people to abandon the page. But yeah, people told us, we asked them, they told us. And finally, you should interview actual customers, just talk to them one on one, and your moderation skills are everything. I also like to interview customer service reps. Ask them like, hey, when people are calling in, what are their top questions, you know? Because I might want to put that content onto the website. Like maybe people are calling, what kind of pre-purchase questions they get or what kind of technical problems people are complaining about? Interviews, also interviews with salespeople.

If it's like call now to buy blah, blah, blah, blah, blah,
I wanna ask what are the questions again people considering a purchase, what are they asking? And what type of answers work really well? Because the salespeople are picking up the phone call, they're usually very experienced. Finally, if you have live chat on your website, and you should, it really works, read through the transcripts. Read through the transcripts of what they're asking, what are the common questions. And again, you wanna pay attention to patterns. Like some questions, I guarantee you, are asked more often than other questions.

And if you go through these four steps,
you'll have a pretty good idea what your users want and how they want it. You can learn a lot by just observing users, just by looking at what they're doing on your website. (uplifting instrumental music) Last point in our conversion research process is user testing. User testing essentially means that we're gonna recruit people that represent our target audience, and we'll have them use our website, we'll have them perform certain tasks and we're gonna just observe how they go about performing these tasks and whether they encounter any problems, usability issues, any sorts of friction.

Number one and the easiest, fastest, cheapest way
of conducting user testing is remote unmoderated user testing. So this is when you use sites like trymyui.com, usertesting.com and so on where you just go and specify the number of people you want, more or less what the demographics should be like then you plug in your website URL and write down the tasks you want people to accomplish and bam, done. It's really fast, really cheap. Out of these three options, it's the least valuable but it's so much cheaper and faster, you can do it.

Now ideally the people performing tasks on your website


represent your target audience.
But anybody is better than nobody. So even your grandma, like (mumbles) your website should be usable for everybody. The specific lingo on the website might be you know, like if you're selling marketing automation, your grandma is not maybe gonna understand this so hence if you want somebody to assess your copy and your value proposition, you need somebody else. Then remote moderated is basically, you can use Skype or Zoom or any of these tools where the person that you're basically interviewing, you're moderating a user testing session.

So they're sharing their screen and using your website
and you're giving them prompts what they should do or what not to do. This is only successful if you're a good moderator. Because a terrible moderator will ruin this process. So you don't wanna do this. The best way to do this is in-person moderated. So basically you go to a user's home office, they come to you but like going to them is better when they're in their natural environment, they feel comfortable, relaxed, all that stuff. And again, moderator skills are 90% of the outcome.

But this is the most time-consuming and the most expensive.
So what kind of task should these people perform? I like to give them roughly three types of tasks. Do something highly specific. So if it's E-commerce, find a pair of pants size 34, black, under $30. You know, very specific and I wanna see how they go about finding that. But some people have very specific ideas what they want and you wanna test your site, can they find that specific thing? And then you wanna give them a broad task. Hey, it's your best friend's birthday coming up, find something they will like.

And so again, you don't say use search or use the menus,
you just wanna see how they go about it, no specific instructions. And finally, you wanna have them complete the funnel. So like complete the checkout, buy the product, et cetera, et cetera. And you wanna recruit five to 15 people. So less than five, it's hard to tell which user was an outlier and like, just weird. And more than 15 people is unnecessary because the same issues keep coming up and you don't get really any new insights.

So the ROI goes down beyond 15.
So you recruit five to 15 people. If you use unmoderated, then usertesting.com, trymyui will give you a panel to recruit from. If you do moderated, you wanna recruit people through different channels. Craigslist might work. Or generally, there's a Facebook group for any interest. If it's beekeeping, there's a beekeeping group. If there are runners in Oklahoma, there are probably 100 Facebook groups for those people.

And so find those Facebook groups and you can say hey,
join that group and say hey, I'm looking for a beekeeper to give me feedback on my website. And obviously, it needs to be paid. Depending on how much time you want from them, these people might be paid 25 to 100 bucks. It might not be cash, might be gift cards, Amazon, Walmart gift cards, whatever. And it's important that you just pay attention to what they do, not what they say.

Because I've had plenty of experiences where
people go through the website, they have a horrible experience mainly on mobile because mobile experience still sucks on so many websites. And like, they really struggle with checking out, like they can't figure out how things work. And then you ask them, so how was your experience? Oh, it was great! What did you like about the site? Oh, it was so easy to use! What would you change about the website? Oh nothing at all. And I saw that person struggle for seven minutes. So don't really listen to what they're saying, just pay attention to what they do.

Figure out what the problems are, fix them
and do a new round of user testing to see if those problems persist or the fixes that you implemented maybe dissolved the issues. But you'll discover new ones. What works the best? Does this work better, or this one? By how much? A/B testing is how you find out. (uplifting instrumental music) So once we've completed our conversion research process, we have a list of problems that we are aware of.

Now you gather a team of people from various teams,
you know a designer, a developer, business expert, you know, whoever. And together, you hypothesize what should be changed to tackle this problem that we have identified? And you wanna come up with as many different ideas as possible for treatments. Some people advocate that you know, like the ideas that you come up with should be all very similar, but there's also something to the idea that all ideas are fungible, meaning that there's no, no idea is better than the other idea, these are just ideas.

So you just need more ideas.
And then the more radically different the ideas are, the better. Because if everybody is like oh yeah, you know, that's the way to solve it! You're gonna be wrong. Because oftentimes, the idea for a treatment that nobody believes in ends up being the winner. So you wanna come up with as many ideas as possible, and now you wanna, based on your traffic, you have to decide how many variations can you test at once? Because A/B testing is really A/B/n testing, it could be A, B, C, D testing.

And A/B testing and the statistics around it
is actually, it's pretty complicated and the rabbit-hole goes really, really deep. So here I'm gonna just cover you, cover the essentials that you need to know, but there's so much more to explore for you down the line. So once you've figured out what are the treatments you wanna test, you can do some calculations around how many treatments can you test at once? Because there's a very important test. If we run an A/B test or A/B/C test, when is it done?

When is the test over and we can say
this worked or didn't work? So it's about stopping rules. Basically, you need to take into consideration three things, and one is sample size. Do we have enough people in the experiment to have statistical validity? Because we need a certain sample size, a certain population to be part of the experiment to be able to detect a difference between variations. Because you know, we're gonna compare which of these variations is going to increase sales the most. And the smaller the difference, let's say B is 2% better than A, we need way more people to detect the 2% difference versus let's say variation C is like 25% better than A, our control, then we need much less sample size to be able to detect that uplift, whether that test is a valid, has a valid winner or not.

For calculating how much sample size,
how many people you need per variation, there are all kinds of sample size calculators out there. So just Google A/B test sample size calculator, you'll find one, there's not much to it, it's pretty easy. So this is number one criteria. So the sample size calculators will tell you how many people you need, and then you can look at your own traffic and determine like how much time do I need to get that amount of people to be part of my experiment? If you're in Google.com, it takes a day, right? It takes 30 minutes to get you know, all those people.

So let's say that we need you know,
100,000 people per variation, our sample size calculator shows us. If your website has 100,000 people per month, then you can't run that A/B test. Statistics, it's math, and you can't change math. So small, low-traffic websites just don't have enough volume to run A/B testing. If you have less than 500 transactions, so it's like purchases per month or signups, then you can't do A/B testing, you don't have enough volume.

If you have 500 to 1,000 per month,
you can maybe run one test per month. But like, to implement the proper testing program, you need more than a 1,000 transactions. And the transaction could also be an email signup. Then of course you can only optimize for email signups, not purchases. But let's say that we are you know, google.com, and getting 100,000 visitors per variation takes us 30 minutes to a certain page.

Is the test done 30 minutes? No.
Because what we're doing is we're taking them a convenient sample of our traffic, not a representative sample. Because people behave differently in the morning and afternoon. Monday behavior is different from Friday behavior and weekend behavior is completely different. And also, let's say we run a test from a full week, so from Monday to Sunday and that might be fine, but how do we know that this week is not weird, that this week is not an outlier? Maybe this week our competition did something that impacted our test.

So it's better to always test
at least for two business cycles. For most companies, it's gonna be two weeks. And of course, if you can't reach your desired sample size in two weeks, you need to test longer. When it comes to building A/B tests, somebody needs to design them, somebody needs to code them up. So depending on the changes you need to build those treatments that are addressing the problems that you identified in the research process, the tests can be simple or complicated. So most testing tools have a visual editor, but the visual editor, it either is extremely limited, meaning that you can just tweak copy and maybe change button color, but you should really not use the visual editor because the visual editor, you know, as I said, you are highly, highly limited in what you can do, and your hypothesis should not come from what can I do with the visual editor, but they should come from what is the best treatment to fix this problem that I identified here? So for most 99% of A/B tests, you need a frontend developer to build this test for you inside a testing tool or in whatever code editor.

It's essentially manipulating the frontend code.
So you need to know CSS, jQuery, JavaScript, et cetera. So if you're not a developer, don't attempt to be one. Hire a developer. There are dedicated companies that build A/B tests for you, like code them up, design them, whatever you need. So no worries here. Now number one test result killer is a broken code.

So meaning that you create an A/B test, you run it,
your variation B is losing heavily, like minus 50%. Oh, our idea was bad. But in fact, very often and most often, the reason is that it just doesn't work, the variation B actually doesn't work in most if not all browsers, there a bug in there, something is off. So quality assurance testing for A/B test is extremely, extremely critical. You cannot miss this step.

Most testing teams all have a dedicated QA people.
QA teams, full-time QA people who conduct quality assurance

on their tests.
You need it to run A/B tests to make sure that there are no bugs in there. So once your test is done, now there are three choices what may happen. Nothing at all, meaning it's flat, there's no significant difference between the two. Well then, you know, there's something else. There's, you know, that your treatment lost. Okay, there's something else. Or your treatment was a winner, implement the winner. Now in case it was flat, or in case it was a loser, it's still learning.

Now you might go and look at your,
analyze the test results per segment. So you wanna look how the people in your test responded to the treatment. If you look new versus returning or different traffic source, and of course you should always test mobile and desktop separately. If you combine them, you wanna look at them separately. But the same rules apply here. In order for you to analyze results within a segment, you need enough sample size inside a segment as well.

So if you have a low traffic website,
you can't really do any segmentation here. But if you also discover things that don't work, that's progress. You wanna understand what works and what doesn't work. So learning, there are no losses, it's just only learning. But yeah, A/B test helps you figure out what really works and put an end to opinions and arguing or what is better. Just test it.

What students are saying about this course

This [course] has taught me more than 2 years in a UX role. THANK YOU!

Photo of Rachael Robinson

Rachael Robinson @ Ahoy Designs

About this FREE COURSE

Conversion optimization – when done right – is a systematic way to convert browsers into action takers (leads, buyers). It’s about making more money and growing the business without investing additional resources into acquisition.

In order to improve your conversions through optimization, you need a conversion research process. Research that tells you where are the problems, what the problems are and why those problems are problems, to begin with. That is why I created the ResearchXL Framework.

You can’t improve your site much just by looking at it. You need to start by asking business questions about user needs and behavior, you need to measure everything people are doing. The data that you get from this will help you answer those questions, and the insights will lead to hypotheses.

Once you have hypotheses derived from data, you can start running intelligent online experiments and finally start getting the results you want. That’s the real way of doing conversion optimization. And this free video course will teach you how.

About Peep

Peep Laja is the founder of CXL. He’s a renowned conversion optimization champion, was nominated as the most influential CRO expert in the world.

After running the CXL agency arm for 5 years, he started CXL Institute where data-driven marketers are trained.

Over the last 20 years, Peep has worked in web development, marketing consulting, B2B sales, SEO & PPC and SaaS.

Peep Laja