The Advanced Guide to Qualitative Research

You spend most days analyzing and interpreting numbers, right? You’re constantly sifting through Google Analytics dashboards, form analytics reports, Mixpanel data – the list is endless.

When you spend so much time focusing on the numbers, it’s easy to forget about the people generating those numbers.

That’s where qualitative research comes into play. Qualitative research sample size requirements are also significantly less than quant research, making it doable for most companies.

Qualitative vs. Quantitative: What’s the Difference?

Research is all about understanding human behavior. Qualitative research is conducted through observation and inquiry while quantitative research is conducted through data measurement, analysis, and comparison. Essentially, quantitative research will tell you the what and qualitative research will tell you the why.

Anyone who is familiar with the ResearchXL model will know that the two go hand-in-hand to deliver insights…

Let’s say your problem is customer acquisition (you’ve seen the numbers). Despite optimizing your lead gen form, which you suspect is the problem, you’re not seeing a major increase in conversions. What gives?

You decide to set up a quick, one question exit survey for visitors. You simply ask, “What prevented you from buying Product S?” You’re expecting responses to shed light on a specific problem with your lead gen form. Instead, responses tell you that your prices are too high or your value proposition was too confusing.

Now consider how much time you spent optimizing that lead gen form, how much testing traffic you wasted. A few tweaks to your value proposition or adding a price anchor could have produced much more significant results… if you had only known.

In the words of Michael Aagaard, “It’s very difficult to solve a problem you don’t understand. Vice versa, it’s pretty easy to solve a problem you do understand. That’s what research does… it helps you understand problems.”

Solve Problems You Don’t Know You Have

Conversion research, qualitative or quantitative, is vital. Without it, you’re wasting time improving and testing things that aren’t going to have a high ROI.

When Michael surveyed 100 marketers, he found that there were four common reasons for not conducting conversion research….

1. Time
2. Client / Company Buy-In
3. Budget
4. Don’t Know Where to Start

Qualitative research helps you answer the following three questions:

1. Who is my audience?
2. Why are people not converting?
3. What do I test?

Who can’t find the time to find the answers to those three questions? And what client / company would veto or not provide a budget to answer them?
If you still can’t get the buy-in or budget, consider the three main ways qualitative research can impact your site.

1. Create Accurate Personas

Your customer personas are representations of the different segments within your target audience. With personas, you can get a feel for exactly who your customers are, how they think and what motivates them.

• What’s the persona’s intent? For example, does she want to lose weight or have a healthier lifestyle?
• What’s the persona’s lifestyle profile? For example, is he a young professional or a retired senior?
• What are the persona’s objections / fears? For example, does she think it’s too expensive or too innovative?

You get this data through user interviews and buyer intelligence. With this information, you’re able to make more informed decisions about your design / UX, copy, calls to action, and even your product or service itself.

Jen Havice wrote an incredible article on this topic for CXL, How To Create Customer Personas With Actual, Real Life Data. If you want to learn more about creating customer personas, I encourage you to read the full article.

2. Craft Copy That Resonates

Startups struggle to find their value proposition, SME enterprises struggle to describe new features, large enterprises struggle to speak to different customer segments.

Crafting compelling copy is a struggle for businesses… of all sizes.

Of course, the person most fit to write your marketing copy is your customer. When you understand the reasons they converted, you get closer to your value proposition. When you use their words and phrases, you play on the science of familiarity.

Qualitative research helps you make more informed decisions about your copy.

In 2014, Sujan Patel was able to boost conversions by over 30% by spending two weeks working as customer support agent for his company. This exercise led to a shift in the company’s value proposition. Instead of focusing on productivity and time saved, they focused on the features he heard customers raving about most often.

3. Make Data-Drive UX Decisions

As a marketer or entrepreneur, you have the curse of knowledge. In other words, you’re too familiar with your site to look at it objectively, even if you think you’re capable of doing so. You can never experience your site for the first time ever again.

As a result, it’s very difficult for you to make objective, high impact UX improvements. The best way to identify UX issues is to simply watch your visitors interact with your site.

Using UserTesting.com, which allows you to watch videos of real people using your site (or app), StubHub discovered that a link was causing major confusion. A “See Details” link was designed to take visitors to the ticket purchase page, but it was too ambiguous.

After watching visitors run into the same issue over and over again, StubHub changed “See Details” to “Go” and conversions increased 2.6%. For StubHub, a 2.6% increase in conversions is worth millions of dollars in revenue per year.

The team at StubHub had the curse of knowledge… they knew what to expect when they clicked “See Details”. Imagine how long it would have taken them to realize that link was a source of confusion.

4. Implement Meaningful Updates / Features

You’re about to roll out a feature that you think your customers (and potential customers) will love. What if that feature is unintuitive and confusing? What if it’s not interesting to your audience at all?

First of all, qualitative research can help you decide what features are missing. If you design and develop product / service updates and new features based on what you think people should want, you’ll waste valuable time and money. Instead, base it on what they have demonstrated a need for.

Simply asking your customers what product / service they used before, what they’d like to be able to do with your product / service, what they find inefficient or time-consuming, etc. can be helpful. Of course, people rarely know what they really want. So, once again, watching people engage with your site or app can be helpful.

Second, once the updates and new features are implemented, qualitative research can tell you how well they’re being received. Is the site or app easier to use now? Is the new feature causing any confusion?

By adding UserTesting.com to the product development lifecycle, Evernote was able to increase user retention by 15% while also dramatically increasing user engagement numbers.

Philip Constantinou, former VP of Product at Evernote, wrote, “[Also,] UserTesting.com lets us try out experimental versions of our applications, which is something we could not do with traditional beta testers.”

So, now we’re left with the final barrier to conversion research: “Don’t know where to start.”

User Testing / Usability Testing

What Is It?

User / Usability testing is the best (and fastest) way to understand how real people experience and engage with your site.

It’s unique in that you’re not asking people to simply tell you about their experience, which is subject to bias and poor recall. Instead, you’re asking them to show you their experience, where you can pick up on issues and insights they weren’t even aware of.

Why Does It Work?

So far, you’ve see how well user testing has worked for companies like Evernote and StubHub. Watching how people interact with your site is the easiest way to…

• Gather major, unexpected insights about your UX / design and copy.
• Identify the need for a product or service pivot.
• Come up with high impact test ideas.

Since user testing is essentially eliminating the curse of knowledge, it often leads to powerful insights. Going back to our three main goals of qualitative research, user testing tells you why people are not converting and what to test next.

Ask yourself the following questions…

• Is my navigational system really as intuitive as I think it is?
• Is my value proposition really as clear, concise and compelling as I think it is?
• Is my lead funnel simple and clear? Is the next step obvious and non-intimidating?

User testing helps answer those questions. What’s more is that it helps answer those questions quickly.

For example, during their annual user conference, Atlassian was on a mission to collect as much qualitative data as possible. Using UserTesting.com, they were able to record over 200 sessions in just three days.

How to Get Started

1. Start user testing early. The earlier you can begin, the less time you’ll waste developing, designing and testing things that simply don’t matter. Most people believe user testing is expensive and impractical, but anyone can do it. Steve Krug and Jakob Nielsen have both said that even three to five participants is enough to identify major usability issues.
2. Conduct user testing often. User testing is not a one-and-done approach. Instead, it’s an on-going effort to improve usability and gather qualitative data. After you try it for the first time and gather your insights, make adjustments and try it again. Did you tweaks work the way you expected? What else can you optimize?
3. Be aware of your moderation influence. When you think of user testing, you likely think of a remote environment. Still, you are moderating the experience to some extent (e.g. “Think of a gift you want to buy someone. Try to find that gift on the site.”) You are asking them to visit your site and run wild.
   1. To avoid any validity issues, try to: (1) have users browse their favorite websites before starting the test, (2) bring them through two different experiences (e.g. finding your pricing page and walking through the checkout process), and (3) maintain a blind experiment by mixing in unrelated tasks.

Tools to Use

Here are some tools to help you with this type of qualitative research:

• Wynter – target customer insights for B2B companies.
• UserTesting.com – “Get videos of real people using your website or app, in 1 hour.”
• TryMyUI – “Watch videos of real people using your website.”

Live Chat Transcripts

What Is It?

Most of you know that live chat is a powerful customer service tool that helps reduce cart abandonment, reduce bounce rate, etc. It’s also a great way to gather qualitative data.

Think back to Sujan Patel and his experiment of working as a customer support agent for two weeks. Impressive results, right? Unfortunately, you might not have two weeks to dedicated to something like this.

Instead, you can review live chat transcripts, reading through conversations your agents are having with visitors, to gather the same insights.

Earlier this year, we wrote about making live chat convert in greater detail. If you’re considering using this type of qualitative research, be sure to take a few minutes to read the full article.

Why Does It Work?

Live chat transcripts tell you a few very important things about your visitors…

• What they are unable to find on your site.
• What is missing from your site.
• What is unclear or vague.
• What is causing friction or preventing them from converting.

Going back to our three main goals of qualitative research, live chat transcripts tell you who your audience is, why people are not converting and what to test next. Not too bad, huh?

Live chat transcripts work because you can identify points of friction throughout your site and conversion funnel. You’ll be able to identify patterns (e.g. 12 people have complained about poor product zooming in the last three days) and run meaningful tests to improve UX / design, copy, and calls to action.

How to Get Started

1. Use Wordle to save time. Reading through live chat transcripts is time-consuming, but word cloud tools like Wordle can help you identify recurring points of friction and target your search so that you know what to look for.
2. Set live chat triggers to encourage use. Be proactive in starting conversations. After visitors have been on your site for a certain amount of time, a degree of intent has been established. They might not be struggling enough to reach out to you, but they may be receptive to an invitation to chat (e.g. “Can I help you find what you’re looking for?”)
   1. Most live chat software has optional triggers (e.g. if the visitor is on the pricing page for 30 seconds, trigger a message) built-in.
3. Ask the right questions. Ask your support team to work with you on gathering qualitative data. Once they understand how you’ll be using the transcripts, they can begin asking key questions (while solving the issue at hand). For example:
   1. What are you looking for today?
   2. How long have you been looking for it (here or otherwise)?
   3. What has been the most frustrating part of your search today?

Tools to Use

Here are some tools to help you with this type of qualitative research:

• Chatlio – “Talk with your customers using your existing Slack service.”
  • 30 day free trial, $49 / month.
• Olark – “Answer their questions before they leave. Make happier customers.”
  • 14 day free trial, $12-17 / operator / month.
• Zopim – “Live chat is a faster and more personal way for you to engage your customers.”
  • 14 day free trial, free (only agent only) or $11.20-44 / agent / mo (billed annually).

1-on-1 Customer Interviews

What Is It?

It’s a good old-fashioned conversation with your customers. Instead of watching visitors use your site or drawing insights here and there from live chat, you’re having a direct conversation with someone who has purchased something from you.

These conversations can happen in person, of course, but they often happen online or even over the phone. You can also try a text-based interview using the same tools you’d use for surveys (more on that soon).

Why Does It Work?

1-on-1 customer interviews bring you one step closer to answering the following questions…

• Is your messaging aligned with what your audience wants to hear?
• Does your value proposition resonate with your audience?
• Was the proposed value delivered post-conversion?
• Were there any roadblocks to buying your product or service?
• What other options did they consider before choosing your product or service?

The list is essentially endless, but those are five core learnings you can expect.

Going back to our three main goals of qualitative research, 1-on-1 customer interviews tell you who your audience is, why people are not converting and what to test next. Once again, three out of three.

This is perhaps the most direct way of gathering qualitative data. The best way to get to know your customers will always be to sit down and have a conversation with them. Numbers can’t replace that experience or offer the same insights.

How to Get Started

1. Start with recent customers. These are the first people you should be talking to. Then, progress to interviewing repeat buyers and long-time customers. You want to know why people convert the first time and recent customers have the freshest memory, but you also want to know why people have stuck around and repeated the buying behavior.
2. Interview your sales / customer service staff. For the same reasons it’s a good idea to review live chat transcripts, it’s a good idea to interview your sales / customer service staff. They’re on the ground talking to your visitors and customers every day. What questions are they asked most often? They have insights that will surprise you.
3. Be aware of biases and poor recall. When asked what issues they encountered, customers might say “none” when really, they had trouble finding something or understanding something. They may be more inclined to tell you what you want to hear and, frankly, they may not be aware they had trouble finding something.
   1. Also, take precautions to avoid leading questions. As a marketer, you are bias as well.
4. Record the conversation. If you’re not performing a text-based interview, be sure to record the conversation. Most major voice / video chats have recording devices built-in or free apps available for download. Log the file in a spreadsheet so that you (and others) can go back to the conversation in the future.

Tools to Use

Here are some tools to help you with this type of qualitative research:

• Google Hangouts / Skype – Both solid, widely adopted tools to talk to your customers.
  • Free.
• 15Five – You can use this tool to talk to your sales and support staff.
  • 14 day free trial, $49 / month for the first 10 people ($5 / month for additional people).

Customer / Traffic Surveys

What Is It?

1-on-1 customer interviews, like reviewing live chat transcripts, are time-intensive. If you have even a 30 minute conversation with ten customers, that’s five hours of your week. Customer and traffic surveys allow you to gather data from more people at a faster rate.

Traffic surveys are concerned with people who have visited your site, but haven’t converted yet.

• Exit Surveys – Before a visitor leaves your site without converting, prompt them to answer a question or two about their experience.
• On-Page Surveys – While a visitor is still on your site, prompt them to tell you why they’re there and how their experience is going.

Customer surveys are similar, but concerned with people who have actually purchased from you in the past.

Why Does It Work?

According to FluidSurveys, 24.8% of people are willing to complete email surveys on average. So, if you sent an email out right now with a link to a survey, approximately one in four of your leads / customers would be willing to answer questions about their experiences.

Through customer / traffic surveys, you can find out…

• Who they are. This will help with creating personas.
• What their intent is. What are they trying to do or achieve? Do they think your product or service can help / has helped?
• How they shop. Did they compare you to competitors? What were the major benefits of your product or service, in their own words? How long did they weigh the decision?
• Where the friction was. What fears did they have about giving you their money? Did they have doubts about your product or service? Where did they have trouble within the conversion funnel?

Going back to our three main goals of qualitative research, customer / traffic surveys tell you who your audience is, why people are not converting and what to test next. Another three out of three!

Your visitors and customers are willing, even without incentive, to answer your questions. All you have to do is put together a survey that will get to the heart of the four points above.

Inman News used Qualaroo to nudge visitors to find out how many of them knew about their upcoming conference. As the conference date got closer, they switched from an information gathering nudge to an instructional nudge, sending visitors to their “last chance savings” landing page.

With a 4% response rate, Inman learned that 76% of visitors didn’t know about the conference. Via traffic from the nudge, Inman drove thousands of clicks to their “last chance savings” landing page and secured $5,000 in registration revenue in just two days.

How to Get Started

1. For visitors, ask about friction. What’s the number one action you want your visitors to take? Now ask them a question or two to identify the major points of friction surrounding that action. What prevents them from taking it? What hesitations do they have? Find out why they are so uncertain.
2. For customers, ask about retention. Optimizing for retention is a high impact, on-going task. The number one action you want your current customers to take is to, well, remain customers. What do they wish they could do with your product or service? Did you deliver the full value you proposed? What other benefits could you provide to keep them around?
3. Ask high impact questions. Your visitors want to help you, but there is only so much you can ask of them. (Getting answers to open-ended questions, while ideal, is harder than answers on a survey scale or multiple choice responses.) Before you ask a question, think about how you would use the data collected. If you don’t have an answer, don’t ask the question. Try to get the most value from the least amount of questions.

Tools to Use

Here are some tools to help you with this type of qualitative research:

• Qualaroo – “Qualaroo website surveys uncover customer insights that lead to better business results.”
  • 14 day free trial, $63-499 / month (billed annually).
• TypeForm – “Forms. Done awesomely.”
  • Free (for basic users) to $20 / month (for power users; billed annually).
• WebEngage – “The most powerful on-site customer engagement suite for your website.”
  • 14 day free trial, $49-949 / website / per month (billed annually).

Conclusion

Qualitative research is just as, if not more, valuable as quantitative research. Every once in a while, it’s important to venture outside of your dashboards and spreadsheets to actually engage with your customers.

Without qualitative research, one little word will plague your data and tests: “Why?” [Tweet It!]

Here’s what you should remember about qualitative research….

1. Qualitative research is the “why” to quantitative research’s “what”. They work together.
2. It tells you who your audience is, why they’re not converting and what to test next.
3. It helps you better understand the problem, resulting in: better tests, more accurate customer personas, awesome copy, improved UX, and meaningful new features.
4. Perform user testing early and regularly to avoid wasted time and effort (for both developers and marketers).
5. Review live chat transcripts to identify points of friction. Be sure you’re asking the right questions and use word cloud tools like Wordle to save time.
6. 1-on-1 interviews with your recent and long-time customers (and sales / support staff) will provide more in-depth qualitative data, but be aware of biases and poor recall when analyzing that data.
7. Customer and traffic surveys will help you gather data (who, what, how, where) from a lot of people quite quickly. Ask visitors about friction and ask customers about retention.