How to QA Ecommerce Email Campaigns to Increase ROI

I don’t have to convince you that email is important:

• Email is a cost-effective way to get sales. Converting customers from your mailing list costs less than converting the same number via advertising.
• Email includes your most-interested customers. The majority of your list is composed of people who’ve already placed an order, so you know that they’re willing to buy (though it still takes effort to win back customers.)
• Email maxes out revenue from big sale periods. Black Friday and Cyber Monday sales represented $123 billion in revenue in 2018. And, according to Klaviyo, emails initiated 30% of those sales.

But do you work on your email? Is email an actively managed and QA’d operation in your business?

In our experience, most stores seem to “set it and forget it” with email. People don’t go through email campaigns from the customer’s perspective to see what they’re receiving prior to or after an online purchase.

And, as a result, your email marketing may not fit your customers’ needs, which, in turn, reduces email conversions. In this post, you’ll learn how to QA your email campaigns so they work for you and your customers.

But first, what is email “quality assurance”?

Email quality assurance, or QA, is a process that lets you check whether your email campaign works as it should. (Most examples—and some steps—in this post apply only to ecommerce stores, but many apply to non-ecommerce companies, too.)

In short, you subscribe with separate email addresses from everywhere that your store gathers an email address, and wait to see what happens.

First, create unique email addresses for each behavior in your store, including:

• Signing up for your general mailing list
• Abandoning a cart
• Placing an order
• Signing up for an in-stock notification
• Ordering a subscription-based product
• Buying something that requires onboarding, first-time configuration, or assembly

Then, wait for your store’s emails to roll in. We’ll talk about the exact process for QAing in a moment. But first, here are some signs that you’re not managing your email campaigns effectively.

8 symptoms of poorly QA’d email

My consultancy, Draft, has been running a long-term research project on email for the past few months. For 30 stores, we did the same process above: We placed an order, abandoned a cart, signed up for the primary mailing list, and signed up for an in-stock notification using four unique email addresses—and waited to see what came in.

So far, we’ve found a lot of common mistakes that people tend to make with their email—even if they’re doing lots of other things right. QA’ing your email could solve all of the mistakes, earning new customers and reorders in the process.

Here are eight of the biggest ones:

1. Multiple review requests

We frequently got pinged to write a review from multiple apps. We presume that, had we written a review, it would’ve ended up in different places on the store.

Reviews should be requested from one source, with one touchpoint, and posted to one place on the store.

For example, Taylor Stitch asks for a review and then, later, a Net Promoter Score survey (likely the culprit of an app that they installed and forgot about).

2. Off-brand emails

Emails should always scan like they’re coming from the same source, with a consistent voice, tone, and appearance. Transactional emails, such as receipts and event invitations, tend not to be styled like the rest of the brand’s primary marketing list.

For example, Chubbies’ emails follow a consistent format:

Which makes it jarring when we get emails from Chubbies that look like this (replete with a broken image):

3. Poorly targeted emails

Personalization is tremendously important in any email strategy, yet we rarely saw a store doing any discernible personalization. There’s a risk to getting personalization wrong, of course—false positives can result in confusion and increase unsubscribes—but the potential benefits are significant.

Usually, poorly targeted emails announced products that were completely irrelevant to me or unrelated to the order that we placed. For example, Sportique recommended women’s products to a person named “Nicholas” after I placed an order for a gear bag:

If you’re going to personalize your emails, start at the highest level: If you sell products that are mutually exclusive for men and women, personalize by gender. If you sell products that naturally “upgrade” from one to the next, sell the next product up to purchasers of your introductory product.

4. In-stock notification issues

With many stores, products quietly come back in stock without notifying us—even though we signed up to be notified. Or they would be quietly discontinued, but we weren’t provided with any alternatives.

For example, Outdoor Voices restocked a product without telling us. And Jeni’s Ice Cream discontinued a t-shirt that we signed up for without telling us:

5. Letting the list go cold

This is 101-level, but it bears mentioning since we saw so many stores mess it up. You should be emailing consistently, lest any attempts to revive the list result in a bunch of unsubscribes. After three months of no activity at all, Taylor Stitch began emailing us a few times a week:

6. Link target mismatch

For example, on this Rothy’s email, “Shop Now” should go to a collection of pink products, not all flats:

7. Non-durable links

If a sale ends, the link to that sale should go to a page that provides alternatives, not a generic “all sale items” section or—worse—a 404 page.

For example, when Kettle & Fire’s sales end, clicking their calls to action takes you to their home page (albeit with a UTM tag appended):

Instead, the customer should be taken to a page that explains the sale has ended, apologizes for the inconvenience, and provides alternatives.

8. Unmanaged failure modes and edge cases

• If an out-of-stock product is discontinued, people should be emailed about it.
• If an abandoned cart discount is offered, it should work in perpetuity.
• If a call-to-action link is clicked weeks or months later, it should still work.

Now that you know what to look for, let’s talk about the step-by-step QA process to manage your email campaigns.

The QA process for managing email campaigns

First, create some smart folders to filter by each inbox and store name:

(This screenshot is for Apple Mail. Here are how-tos for setting up smart folders in iOS Mail, Apple Mail, and Gmail.)

Then, go through each email that’s sent and identify:

• What’s working.
• What isn’t.
• How to reproduce each issue.
• What needs to be done to fix any outstanding issues.

These are the key components to review in your email marketing:

1. Orders

We place orders for real products, without returning them, to get as close to a real customer experience as possible. Once that happens, we look for the following:

• Is a receipt email sent?
• Is a shipment email sent?
• Is a welcome email sent? According to Omnisend, welcome emails have an average open rate of 45%, versus 18% for promotional emails.
• Are other transactional emails (out for delivery, delivery, signature required, etc.) sent?
• Are there personalized emails based on what we ordered?

You’ll also want to place a separate order and return it to see if there are follow-up emails about the return experience.

2. In-stock notifications

In-stock notifications are something of a special case since the end state can differ significantly. For example, what happens when:

• The product is restocked?
• The product is discontinued?
• The product remains out of stock for over a month?

You’ll want to create a hidden test product that you can use to trigger these end states and see which emails fire for each case. (Here’s how you do it on Shopify to keep the product from being indexed by search engines or available on your own store’s search.) Then, sign up and immediately discontinue or restock the product to see what happens.

Here are all the potential elements you should QA for in-stock notification emails:

Overall

• Before an in-stock notification is offered, does the projected shipping time increase (to a maximum of 4 weeks)? That allows visitors to purchase products that will be out-of-stock for short periods of time.
• Does the in-stock notification contain a checkbox to sign up for the store’s mailing list? If yes, is the box checked by default? Every email touchpoint is an opportunity to collect the customer’s address, but explicit consent is important.
• When signing up for the in-stock notification, is the customer sent a confirmation email? This sort of feedback is valuable for the customer’s experience.
• When signing up for the in-stock notification, is the customer given a clear next step on the store’s website? Avoid dead ends whenever possible.
• If a product has been discontinued, is the customer notified?
• If a product has been discontinued because it’s been upgraded—and the upgrade represents a new product—is the new product linked instead?
• When customers need to be notified of a restock (or discontinuation) event, are they notified at a time of day that fits their time zone? Emails that land in a customer’s inbox at 4 a.m. are less likely to be opened.
• When clicked, do images of products go to their respective product detail pages or to a collection page? Is this the behavior that customers expect?

Restocked products

• Is an email sent to the customer?
• Does it contain a clear call to action…
• …that, when clicked, automatically adds that specific product (including size, color, etc.) to the customer’s cart?
• If the customer hasn’t made a purchase within two days of the product being restocked (and the product hasn’t sold out yet), is a follow-up email sent?

Discontinued products

• Is an email sent to the customer that provides alternatives to their request?

Products that remain out of stock

• Is a follow-up email sent at any point that provides alternatives to their request? How long after the request is first made? This may be a timeframe worth testing—if the volume of in-stock notification requests is high enough.

3. Subscriptions to the general list

• Is a welcome email sent?
• If an enticement (like a free product, download, or discount code) is offered, is the enticement sent, and does it work?

4. Cart abandonments

• Is the customer’s email address retained as they focus away from the email field? (Shopify app Recart does this by default.)
• If the cart is abandoned, is a cart rescue email sent between 2 and 4 hours after the transaction is initiated?
• Does the cart rescue email provide a link with the customer’s saved information, including the cart?
• Are any discounts offered as enticements? If so, are they one-time-use codes, or are they general-purpose codes that can be used by anybody? Unique, one-time-use codes are strongly preferable, lest your code end up on discount sites. (They provide higher conversion rates as well; Klaviyo creates these by default.)
• Are the discount codes time-delimited to promote scarcity? If so, is a timer provided in the cart rescue email? Is a follow-up email sent when time has almost run out?
• Do the discount codes work?
• Do the cart abandonment emails fit the overall design, voice, and tone of the brand?
• Is the customer signed up for the store’s general mailing list? If not, is a checkbox provided for the customer to provide consent to do so?

5. General

• Are transactional emails styled and worded similarly to other promotional emails?
• Are calls to action actually displayed as buttons? Are they styled as real buttons and not “ghost buttons”?
• Is the email’s primary call to action above the fold on smartphone screens?
• Is at least one marketing email sent each week?
• For emails that have clear segments—and you should be segmenting your email—are the criteria for qualifying into and out of the segment working?
• Is at least one email sent educating the customer for every three that sell something? Your email can and should be part of a broader content marketing strategy, which improves your brand and SEO performance.

Conclusion

In analyzing 30 stores, one of the biggest things we’ve learned is that stores rarely make email a line item in their marketing budgets. But they should. The ROI is there, and in the long run it could result in happier and more passionate customers.

Start by QAing your email. This is a process that takes a few minutes to set up, a morning to analyze, and could reap a significant reward for any business. In short:

• Sign up for your own store’s emails by triggering common behaviors, like placing an order or abandoning a cart.
• Wait for emails to come in.
• Analyze your emails for what’s working and what isn’t, and come up with a list of things to change.

With this post, you should be able to manage your email more effectively—isolating and resolving the most common issues that your campaigns face.