How to Conduct Icon Usability Testing (and Do Icons Even Improve Usability?)

Icon Testing

Symbolism is a tricky thing. Different symbols mean different things to different people based on different backgrounds and past experiences. That’s why icons can easily flop, if they’re not properly tested. After all…

If the user can’t use it, it doesn’t work. [Tweet It!]

But how do you test icon usability? Is icon usability testing the same as standard usability testing? Do visitors even respond well to them? It’s not as cut and dry as those covering the evolution of the hamburger menu icon would have you believe.

How Do Icons Affect Usability?

Technically speaking, an icon is a person or thing “regarded as a representative symbol of something”.

Web designers use them in navigations (e.g. click the house icon to go to the home page), in apps and tools for efficiency (e.g. click the B to bold your text, click the eraser to remove what you’ve highlighted), and in copy to bring benefits / features to life (e.g. a running shoe to indicate you’re faster than the competition).

But why? Aurora Bedford of Nielsen Norman Group explains the key benefits of using icons…

“AuroraAurora Bedford, Nielsen Norman Group:

  • “Icons make good targets: they are typically sized large enough to be easily touched in a finger-operated UI, but also work well with a mouse cursor (in contrast to words, which can suffer from read–tap asymmetry on touch screens).
  • Yet they save space: icons can be compact enough to allow toolbars, palettes, and so on to display many icons in a relatively small space.
  • Icons are fast to recognize at a glance (if well designed) — particularly true for standard icons that people have seen and used before.
  • There is no need to translate icons for international users, provided that the icons are mindful of cultural differences (for example, mailboxes look very different in various countries whereas envelopes look the same, therefore an envelope is a more international icon for an email program than a mailbox).
  • Icons can be visually pleasing and enhance the aesthetic appeal of a design.
  • They support the notion of a product family or suite when the same icons and style are used in several places.” (via NN/g)

But, there are disadvantages to using icons, too. Below is a screenshot from a site for a product that “takes out the pain of team inboxes and enables you to scale your customer support, your hiring, your sales and more”. Take a look at the icons and try to guess the benefit that it symbolizes…

Front Icons

If you’re anything like me, your answers are way off.

Here’s what the icons represent on the Front site (from top left to bottom right)…

  • Lightning-fast learning curve
  • Goodbye, messy collaboration
  • Get work done faster
  • Make better decisions
  • Find answers faster
  • See the whole picture

Ok, let’s try again. This time, with a group of software navigation icons. This is screenshot of a tool that helps you build landing pages. Take a look at the icons and try to guess what they all symbolize…

Unbounce Icons

This time was much easier, right? You were probably able to conclude that the T symbolizes text, that the picture of a tree symbolizes images, etc.

All of the icons allow you to add something to your landing page using drag and drop. Here’s what the icons represent…

  • Section
  • Box
  • Text
  • Image
  • Button
  • Lightbox Button
  • Form
  • Embed Video

How many did you get right? Unbounce, shown above, also shows rollover text for each icon, which can be helpful. Otherwise, your users might start to feel like this…

Confusing Icons
Image Source

So, are icons a good idea or not? According to Zoltan Gocza of Centralway, they’re difficult to remember and can really harm usability…

“ZoltanZoltan Gocza, Centralway:

“Many researchers have shown that icons are hard to memorize and are often highly inefficient.

The Microsoft Outlook toolbar is a good example: the former icon-only toolbar had poor usability and changing the icons and their positioning didn’t help much. What did help was the introduction of text labels next to the icons. It immediately fixed the usability issues and people started to use the toolbar.

In another study, the team of UIE observed that people remember a button’s position instead of the graphic interpretation of the function.

In most projects, icons are very difficult to get right and need a lot of testing. For abstract things, icons rarely work well.” (via UX Myths)

If you work in Industry A and I work in Industry B, a single icon could mean two different things. You could see the icon and think “contact page” and I could see the icon and think “pricing page”.

Worse, you could know exactly what the icon is of and I could have no idea. They’re subjective and rarely universal, which makes them risky, especially without proper testing.

But what if you add rollover / alt text like Unbounce or legends / keys? According to Craig Kistler of Small Farm Design, those aren’t true solutions…

“CraigCraig Kistler, Small Farm Design:

“These are workarounds and are not real solutions. Reason is each solution relies on the site visitor caring enough to figure out what the icons mean.

The first actually takes effort for the user to move the mouse, position the cursor just so and then, if using alt text, wait for the text to display.

The second increases the cognitive load of the visitor and adds a fairly significant degree of friction. To use use a legend the user must first think to look for a legend (remember we looking at a website and not a map), then scan back to the icon, take a glance, remember then icon, scan back to the legend, and then find the matching icon and description.

In both solutions the effort seems to outweigh the actual value the icon is trying to provide.” (via Small Farm Design)

Craig went on to explain three simple principles that apply to icons…

“CraigCraig Kistler, Small Farm Design:

  • “Label and image is better than just one or the other (image or text). However, if using only one text works better than just the image.
  • Icon images will be learned, the position of the icon is learned quicker. If you change the image, but the location remains the same , visitors usually won’t notice. However, if you change the the location and keep the image the same visitors will become frustrated.
  • The speed the average visitor will recognize an icon’s meaning from the image alone is directly proportional to how quickly the team can decide on which icon to use. Meaning, things that are obvious to a designer (i.e. question mark for help) are more likely to be obvious to a visitor but things that aren’t as obvious, say maybe return policy, are more difficult to understand.” (via Small Farm Design)

The conclusion is a familiar one: you have to test icons for yourself.

Should you use text? An icon? Text and an icon? Do your visitors recognize the icons you use? Are they used by different sites in different ways or are the icons universal?

Suddenly, there are a lot of questions to be asked. So, what’s the best way to test your icons? It depends what you’re testing them for.

The 5 Elements of a Useful Icon

According to Zoltan, icons are most useful at specific times under specific conditions…

“ZoltanZoltan Gocza, Centralway:

“Where icons add value:

  • Universally understood icons work well (ie. print, close, play/pause, reply, tweet, share on Facebook).
  • Icons can serve as bulletpoints, structuring a webpage (ie. file type icons for PDFs, DOCs, etc.).
  • Good icons can make the look of a webpage more pleasing.” (via UX Myths)

But it’s a bit more complex than that.

There are five elements of a useful icon. It should be: easy to find, identifiable, easy to comprehend, necessary, and visually appealing. Each of those elements can be tested, either in context or out of context. The better your icon performs based on each of the five elements, the more useful it is.

1. Easy to Find

Making an icon visible on a page isn’t enough. Icons also need to be where they’re expected to be, according to Jared Spool of User Interface Engineering, who ran an experiment on icon placement…

Jared Spool” width=Jared Spool, UIE:

“In the first experiment, we changed the pictures of the icons, but kept them in the same location. We found, in general, users quickly adapted to the new imagery without much problem, particularly for commonly used functions.

In the second experiment, we kept the original pictures, but shuffled their locations on the toolbar. To our surprise, users really struggled with this. It really slowed them down, and, in several cases, they could not complete common tasks. (The icons were all visible, they just had trouble finding them in their new locations.)

From these results, we inferred the location of the icon is more important than the visual imagery. People remember where things are, not what they look like.” (via User Interface Engineering)

For example, if you visit an eCommerce site, you expect the shopping cart or bag icon to be in the top, right-hand corner of the screen. When you’re logged into a SaaS, you expect your user profile and account settings to be symbolized by a person icon (or your headshot) in the top, right-hand corner of the screen.

If someone changed those familiar placements, it would be difficult for you to find the icons.

Icons are also competing with the rest of the page. Do they stand out in size? Where are they on the visual hierarchy? Is it obvious that they’re clickable? Are they being overshadowed by other content that’s calling more attention?

2. Identifiable

Once they’ve found the icon, they need to be able to identify it. This is more difficult than you realize.

Right now, in WordPress, I’m looking at an icon symbolizing “paste as text”. It’s a simple clipboard with a “T” through it. It’s common and quite familiar to anyone who writes on a computer often, right?

But when asked, a teenager born in 2000 was unable to identify it. He knew what it did, but not what it was (a clipboard). Why? Because he’d never seen (or even heard of) a clipboard.

Unless they’re universal like the “paste as text” icon, which is rare, icons need to be identifiable. Being able to identify the icon helps give it contextual meaning. If it’s too small or the icon is too obscure, identification (and thus, comprehension) will suffer significantly.

3. Easy to Comprehend

Aurora agrees that few icons are universal, adding that the absence of standards is hurting usability and UX…

“AuroraAurora Bedford, Nielsen Norman Group:

“There are a few icons that enjoy mostly universal recognition from users. The icons for home, print, and the magnifying glass for search are such instances. Outside of these examples, most icons continue to be ambiguous to users due to their association with different meanings across various interfaces. This absence of a standard hurts the adoption of an icon over time, as users cannot rely on it having the same functionality every time it is encountered.

The 3-line “hamburger” icon is an excellent example of an icon that is striving to become universally known. While the majority of interfaces use this icon to represent the main navigation menu, some use the same (or a very similar icon) to stand for a list.” (via NN/g)

(You can read more about the “hamburger” icon here.)

Before using an icon, you have to be aware of how it’s used on other sites / in other apps. How likely are your visitors to use those other sites / apps? So, to summarize, here are your icon recognition options…

  1. Universal – Everyone recognizes and comprehends this icon.
  2. Multi-Use – Most people recognize it, but it has multiple possible meanings.
  3. Unknown – Few people recognize or comprehend this icon.

Universal is safe, multi-use is risky without extensive research and unknown is just plain silly.

Peter Vukovic of FOUNDER.org explains why icon comprehension is so important…

“PeterPeter Vukovic, FOUNDER.org:

“Are you using Gmail? I am. Before the latest update, Gmail had a very clear text navigation on top of the page — Calendar, Drive, Sheets and other Google services were readily available at the click of a button.

Then Google decided to ‘simplify’ and move everything behind an abstract icon. The result? Most people never noticed the icon and Gmail started to receive a flurry of support requests.

People avoid and often ignore things they cannot understand — that’s basic human nature. Avoid designing interface elements that make people wonder what they do, because no one will bother finding out.” (99designs)

If your icon doesn’t clearly symbolize what will happen when it’s clicked, most people won’t bother finding out. Even if they do, they’ll get frustrated and leave the site.

4. Necessary

Ask yourself this: is this icon really necessary? Can I get by with just using text?

Michael Zuschlag, who blogs about UX and identifies as a human factors engineer, has a rule of thumb for deciding whether or not an icon is necessary…

“MichaelMichael Zuschlag, Human Factors Engineer:

“As rule of thumb, icons alone should only be permitted when at least two out of the following three conditions apply:

  1. Space is very limited (i.e., too small for text alone).
  2. The icons are standardized (e.g., the “X” icon on a window Close button)
  3. The icon represents an object with a strong physical analog or a visual attribute (e.g., a printer icon to access printer attributes, or a red rectangle to set a page background red).” (via UX Stack Exchange)

If two out of three of those conditions don’t apply to you, don’t use an icon.

5. Visually Appealing

Here’s the superficial element. How does your icon look? Does it fit well with the aesthetic of your site? Does it match other icons on your site?

This is especially important on mobile, where icons are more necessary and frequently used. If you choose an icon that looks unappealing, you’re hurting usability. If you structure your icons / actions in an unappealing way, you’re hurting usability.

Steven Hoober of 4ourth Mobile shared this screenshot from YouTube…

YouTube's Mobile Navigation
Image Source

Steven claims that on mobile, it’s best practice to…

  1. Show the primary action (e.g. watching videos) in the middle of the page.
  2. Show a couple secondary actions (e.g. search) in the top, right-hand corner.
  3. Hide the remaining actions behind a menu icon.

(Luke Wroblewski also wrote in detail about what’s important enough to be visible on mobile here.)

Of course, how you structure icons and actions influences visual appeal. For example, if you don’t hide your tertiary actions, the site will be considered cluttered.

As an aside, be careful transitioning mobile icons to desktop sites. Aurora cautions against assuming desktop icons are as easy to find as mobile icons…

“AuroraAurora Bedford, Nielsen Norman Group:

“In general, icons tend to be more salient on mobile phones compared to desktop websites and applications. It is easy to assume that because an icon (for example, the “hamburger”) works on a mobile design, it will translate well to desktop. Alas, this is not the case!

On mobile screens, there are fewer elements that compete for users’ attention (simply because the screen is smaller and can fit less information). In contrast, desktop screens are bigger and usually contain a lot of content; users may easily miss some of it depending on what part of the screen attracts their attention.

If the only elements currently in view on a phone consist of a logo, a headline, a single image, and a menu icon, that menu icon stands a much greater chance of being noticed by a user than the same icon presented in the upper corner of a desktop site full of attractive content.” (via NN/g)

Remember, one of the key benefits of using icons is that they look nice. But some just don’t, whether it’s because of the icon itself or the way it’s structured among other actions.

How to Test the 5 Elements of a Useful Icon

When Hannah Alvarez of UserTesting tested icons, she found some surprising results…

“HannahHannah Alvarez, UserTesting:

“In our study, we found that for icons with labels, users were able to correctly predict what would happen when they tapped the icon 88% of the time.

For icons without labels, this number dropped to 60%.

And for unlabeled icons that are unique to the app, users correctly predicted what would happen when they tapped the icon only 34% of the time.” (via UserTesting)

What’s true in Hannah’s study is not necessarily what will be true for you and your site, however. It’s best that you ask your own icon questions and run your own icon usability tests.

1. Is it easy to find?

This element should be tested in-context.

First, ask yourself the following questions and conduct heuristic analysis:

  • Does it blend in with other icons?
  • Is it where it’s expected to be?
  • Is it being overshadowed by other page elements?

Now, you can conduct time-to-locate tests. For these tests, you’ll get ask people who are unfamiliar with your site or app to perform a specific action. All you have to do is measure:

  1. How long it takes them to find the right icon for the action.
  2. How often they choose the right icon on their first attempt.

Example: Gmail

Here’s what my Gmail looks like…

Gmail Icons

Despite the fact that I have used it daily for years, the icons still trip me up from time to time. For example, I routinely mistake the “Back” icon to the far left for the “Reply” icon. Also, I accidentally report people for spam when I want to archive all the time.

The icons aren’t hidden, but finding the icon that will perform the action I’d like to take on the first try still isn’t second nature to me.

2. Is it identifiable?

This element should be tested out of context and without labels.

First, ask yourself the following questions and conduct heuristic analysis:

  • Can people identify the symbol?
  • How quickly?
  • On mobile?

Now, you can ask other people the same questions. Show them an icon and ask them to explain what they think it is (e.g. a trash can, a calendar). If guesses and the truth line up, you’re good to go. If not, pick a new icon and start again.

Don’t include labels in the test, even if your icons will have labels on your site. If they don’t speak for themselves, it’s best that you choose a new icon.

Example: Floppy Disc Icons

Above, a teenager didn’t know what a clipboard was.

Now, imagine how likely it is that he knows what a floppy disc is. He knows that a floppy disc icon can be used to save progress, but he is not able to actually identify it as a floppy disc.

3. Is it easy to comprehend?

This element should be tested out of context and with some contextual clues (e.g. the type of site / app). Note that you can have identification without comprehension and comprehension without identification.

First, ask yourself the following questions and conduct heuristic analysis:

  • Do visitors know what will happen when they click?
  • Do they associate the icon with any other action?
  • Is the icon familiar?
  • How does the icon come across on mobile? iOS vs. Android?

Now, ask other people what they believe will happen if an icon is clicked. For this, you’ll need to provide some information about the site, like I did at the beginning of the article.

You can also conduct an A/B test to find the icon with the best scent. The icon position and label should stay the same in both variations. If you notice that visitors click the icon and the quickly return to the previous page, you know that icon has poor scent.

Example: Etsy

Take a look at these two icons from Etsy…

Etsy's HamburgerEtsy's Heart

What would you guess that they do? Is the top icon a menu or a list? Is the heart a like or a “save to wishlist”? They’re multi-use icons, which create confusion.

4. Is it necessary?

This element should be tested in-context.

First, ask yourself the following questions and conduct heuristic analysis:

  • Is it the right time to use an icon?
  • Can you use text instead?
  • Is the icon actually helping your visitors?

Now, you can conduct an A/B test. Leave the position the same. In variation one, use an icon. In variation two, use plain text. This will show you whether the icon is even necessary in the first place. Does it actually improve usability or does it just look nice?

Example: James Foster’s Hamburger Experiment

Here’s the first variation (icon)…

Variation 1
Image Source

Here’s the second variation (text)…

Variation 2
Image Source

Which do you think won?

The text only version results in a 20% increase in conversions. In this case, the icon was not necessary. In fact, it was actually hurting usability.

5. Is it visually appealing?

This element should be tested out of context.

First, ask yourself the following questions and conduct heuristic analysis:

  • Does it look good?
  • Does it look good with the other icons on the site?

Now, have people score your icons on a scale of 1-5. Alternatively, you can simply ask them to choose their favorite and explain why. What do they like? What don’t they like? Use this insights to design visually appealing icons.

What Goes Wrong in Icon Usability Testing

There are three common mistakes optimizers make when conducting icon usability testing…

1. You Create Imaginary Associations

When phrasing tasks, it’s easy to subconsciously prime participants to create associations that don’t exist. For example, a participant is primed to associate a heart with biology due to the way the question was worded.

This invalidates your results, of course.

To avoid this, simply conduct tests multiple times and phrase the tasks differently each time (try using synonyms) to ensure the wording doesn’t influence responses.

2. You Don’t Consider Development Stage

Icon testing looks very different at each stage of the development process.

In the earliest stage, you should be testing whether or not icons is a good idea. At this stage, you’re focused on out of context tests. Can people identify and comprehend these icons? How easily?

Once you’ve started development, you should be running multiple tests. Start with out of context visual appeal tests. Then move to in-context tests that gauge how easy it is to find your icons.

After development, it’s time to focus on optimization. Start running A/B tests (size, position, symbol, order, etc.) on your icons to make marginal improvements over time. It’s a good idea to continue testing the five elements. How do the scores improve over time? If they don’t, go back to the drawing board. You can also conduct standard usability testing during this time to cement your findings.

None of the tests discussed are designed as one-and-done experiments. They’re iterative and you should continue conducting them to slowly perfect your icon (and label) and its position on your site.

3. You Blindly Use Standard Usability Tests

As mentioned, standard usability testing can be helpful cementing your findings in the final development stage. However, it’s quite high-level, which can be problematic when testing a minor element of usability.

For example, let’s say you run a standard usability test to figure out if your icon is effective.

The first participant doesn’t even notice the icon. She was distracted by another element of your site.

The second participant stays a while longer, browsing the site normally. She doesn’t interact with your icon, either.

Now, you have no idea why. Did she even see it? Was it ugly? Was it confusing? Was it unnecessary? Was it hard to identify? You don’t know because there are too many factors involved.

Standard usability tests should support your other findings.

Conclusion

Do icons improve usability? It depends on how extensively you’ve tested the five elements of a useful icon during the development stages.

Ask yourself the following questions before using an icon:

  1. Is it easy to find?

  2. Is it identifiable?

  3. Is it easy to comprehend?

  4. Is it necessary?

  5. Is it visually appealing?

Then, go out there and actually test it. Because your opinion is of little value on such a subjective subject. What matters, after all, is whether the user can use it.

Join the Conversation Add Your Comment

  1. As touched upon here slightly, I think just as important as using easily recognized symbols/icons is not to use too many. My testing with using too many icons hurts than it helps and I attribute this exclusively to visual overload.

    I usually use a row of 3 icons on any long style landing page, and never multiple rows in one area. Second to this, don’t use more than 3 sets of 3 icons on any one landing page as this is just overkill.

    I also love icons with text on them as mentioned here, especially on mobile.

    Reply
    1. Shanelle Mullin

      Thanks for your insight, Corey. Have you run any formal A/B tests? I’d love to see the results.

  2. [In my limited experience] Users tend to respond quite favorably to icons, until they are required to interact with them. The icons used for a primary task that were skeuomorphic encouraged interaction and enhanced recognition. Other, less essential tasks, when depicted as icons, led to uncertainty and frustration, even when the user guessed correctly.
    It is my evaluation that icons are best used for essential, primary tasks and that when icons are used for secondary features, they lack the desired result.
    I agree with Corey; limited use (3-5) seems best.

    Reply
    1. Shanelle Mullin

      Interesting thoughts, Paul. Thanks for sharing your experience. Any chance you’ve run a test or two with results you can share? (Always worth asking.)

Leave a Reply

Your email address will not be published. Required fields are marked *

Current article:

How to Conduct Icon Usability Testing (and Do Icons Even Improve Usability?)