#5: Learning From Customers (Qualitative Surveys)

The last couple of lessons were about analytics, heatmaps, form data – quantitative research.

Now we’re getting into qualitative territory – the next 3 lessons (including this one) will be about qualitative data.

When quantitative stuff tells you what, where and how much, then qualitative tells you ‘why’. It often offers much more insight than anything else for coming up with winning test hypotheses.

Qualitative research enables us to peek inside the mind of the people we’re trying to sell to. Without knowing what they’re thinking, we’re in the dark. We can speculate all we want but it’s not going to get us anywhere.

Your best source of information – your customers

Asking feedback from anyone is good – but if they’re not your target audience, it might not matter what they think. You can ask me about what I like about tractors, but it’s not going to help you sell more tractors.

So how do you find people in your target audience from whom to get feedback?

The single best attribute to qualify somebody as your target: it’s somebody who took out their wallet, and gave you money. Or if you’re in B2B with a long sales cycle, somebody who filled out your lead gen form.

They had a problem, your offer seemed to solve / alleviate that problem. Becoming your customer is the best qualifier there is.

Getting feedback from customers

So we need to get information from our customers. How exactly?

The absolute best way to get valuable information out of people is through interviewing them. Some say phone interviews reveal more than in-person interviews as people are more open. Maybe. I prefer phone interviews because it’s faster and easier to set up. In-person interviews require a lot more planning and time.

But even phone interviews are time-consuming. So I always pair it with online surveys.

For optimization projects I recommend ~10 phone interviews with customers who seem to match the ideal customer profile + an online survey to hundreds of people.

You want to survey only recent first-time buyers. People who don’t have a history with you, no prior relationship. And no more than 30 days since their purchase (better if it’s like 5 days since purchase) – or they will forget what they were thinking when they shopped on your site (and then bullshit their answers).

The best online surveys have open-ended questions. No (or very limited) multiple choice questions.

You want to get in at least 100 responses, so your sample size would be adequate. I typically aim at getting 250 responses for better data. I often find that having 500 responses is not better than 250. It takes more time to go through the answers, but typically I don’t get additional insight.

Understanding what they want: questions to ask

The kind of questions you ask will determine the usefulness of the surveys, interviews. You need to nail it.

Avoid useless crap like “would you recommend us to others” or “rate our service from 1 to 10”. You might tickle your ego, but you will get ZERO insights.

We’re after insights that help us boost conversions, and nothing but! Do not combine customer satisfaction surveys or other stuff with your conversioninsight survey. Keep this 100% focused.

What to ask? It depends on the business / website. Broadly speaking, you want to understand 4 key things:

  • who they are – putting together personas;
  • user intent – what’s the specific problem they were solving;
  • shopping process – what mattered to them when choosing the product, what kind of comparisons did they do, how many / which other sites they looked at, and so on;
  • friction – fears, doubts and hesitations they experienced before making the purchase.

So the actual questions might be something like this:

  • What can you say about yourself? Watch how they self-identify. Get the demographical & lifestyle data, and see if there are any trends (e.g. generational socioeconomic, etc). If you’ve got a B2B business, ask about their industry and position in the company (and who makes the decision!)
  • What are you using [your product] for? What problem does it solve for you? Here you want to make sure you understand their problem. You might discover some unintended uses as well.
  • How is your life better thanks to it? Which tangible improvements in your life or business have you seen? This will tell you the end-benefit your product provides in the words of your customers. If some say really nice things, hit them up for testimonials or case studies afterwards.
  • Did you consider any alternatives to our product (prior to signing up / buying)? If so, which ones? You want to know who people compare you to. Next step is that you need to build a ‘compare’ page where you compare yourself to the competition and make a case for your advantages
  • What made you sign up for our product? What convinced you that it’s a good decision? Why did you choose us over others? You want to know what’s working for you in your current website + identify some advantages you might want to emphasize more.
  • Which doubts and hesitations did you have before joining? Identify main sources of friction, and address them (or fix them if they’re usability problems).
  • Which questions did you have, but couldn’t find answers to? 50% of the purchases are not completed due to insufficient information. This helps you identify some of the missing information your customers want.
  • What made you almost not buy from us? Identifying friction again.
  • Anything else you would like to tell us? Leave room for feedback about stuff you don’t know about.

I’ve found that it’s best to ask 7-8 questions per survey. It’s not too many to kill the response rate, and not too few to render the whole process useless.

Pay close attention to the wording they use when answering. You want to take those very sentences they wrote in their responses, and use them later in your website copy.

The best copy comes from the mouth of the customers. You’re reflecting back their own words, joining the conversation in their mind.

Yes – it will take time to go through ~200 responses to 7-8 questions. if you don’t have time to do this, you’re basically saying “I don’t have time to work on growing the business”. If you want better conversions / revenue per visitor / etc, then do it. It’s a high-value activity.

Or – if you have more money than time, you can hire a qualitative market research consultancy.

Motivating them to reply

The insights you get from customer surveys are very valuable. So is their time – it takes 5-10 min to fill out a survey, and interviews take even longer.

So you absolutely need to compensate their time. I personally dislike draws: “answer these questions and you have a chance to win free candy!”. That’s OK when it’s a quick 30-second survey, but you need to do better when you ask people to sit down, think and write.

If you want to increase your response rate – and you do – then you need to offer instant gratification to each and every person that takes the time to fill out your lengthy survey.

It doesn’t need to be anything expensive. Ideally you’ll find something that’s valuable to the customer, but not expensive to you. Like a free month of service, $5 Starbucks gift card etc.

For interviews where more time is needed, I recommend you offer $25 or $50 gift cards (Amazon is a good option as everyone uses it, or gift card to your own ecommerce site if you have one).

Tools

You don’t need no fancy tools to send out customer surveys. My favorite tool is Google Docs Forms, completely free. Sure, it might not look the best, but it’s highly functional and allows super fast survey creation.

If you want pretty forms, check out Typeform. It’s as good as it gets (it’s what we use).

In the next lesson we’re going to focus on learning from people who visit your site, but are not necessarily buying anything. Why not? We’ll find out.

  • #1: Mindset of an Optimizer

    You seek to understand your customers better - their needs, sources of hesitation, conversations going on inside their minds.

  • #2: Conversion Research

    Would you rather have a doctor operate on you based on an opinion, or careful examination and tests? Exactly. That's why we need to conduct proper conversion research.

  • #3: Google Analytics for Conversion Optimization

    Where are the problems? What are the problems? How big are those problems? We can find answers in Google Analytics.

  • #4: Mouse Tracking and Heat Maps

    We can record what people do with their mouse / trackpad, and can quantify that information. Some of that data is insightful.

  • #5: Learning From Customers (Qualitative Surveys)

    When quantitative stuff tells you what, where and how much, then qualitative tells you 'why'. It often offers much more insight than anything else for coming up with winning test hypotheses.

  • #6: Using Qualitative On-Site Surveys

    What's keeping people from taking action on your website? We can figure it out.

  • #7: User Testing

    Your website is complicated and the copy doesn't make any sense to your customers. That's what user testing can tell you - along with specifics.

  • #8: From Data to Test Hypotheses

    The success of your testing program depends on testing the right stuff. Here's how.

  • #9: Getting A/B Testing Right

    Most A/B test run are meaningless - since people don't know how to run tests. You need to understand some basic math and statistical concepts. And you DON'T stop a test once it reaches significance.

  • #10: Learning from Test Results

    So B was better than A. Now what? Or maybe the test ended in "no difference". But what about the insights hidden in segments? There's a ton of stuff to learn from test outcomes.

  • Conclusion

    Conversion optimization is not a list of tactics. Either you have a process, or you don't know what you're doing.