When I look back at the most important work I have done with my current company or any of the over 300 other websites that I have worked with, by far the most important work I do is in changing how groups think and operate. So much of optimization is about asking people to go past their comfort levels and their inherent biases and asking them to act rationally. At least 90% of the time and effort I put in is in dealing with these much harder and deceptive factors in success of a program.
In many ways you have to deal with the core causes of the inefficiencies of an organization. Once an organization reaches a meaningful size, usually more than 10 people or so, there are all sorts of internal political and alignment issues that naturally spring up. While it is incredibly easy to get caught up in the discussion of how to test and what to test, those are just actions taken within the symptoms of the much larger diseases. By focusing on how to deal with the larger issues the other actions easily come into place.
I want to present a few tactics to change how groups act. By dealing with these core inefficiencies you will limit the damage they can do to results and help free your program to dramatically increase results.
Like all disciplines, the things you don’t do are far important than the things you do – especially when those actions would make yourself or others around you happy. Your job is to be the doctor and to tell people to eat their vegetables and exercise, even if they mentally just want cotton candy and reality tv. Your job as an optimizer is to change the conversation, usually subtly, to force the people around you to eat veggies.
Tactic #1: Stop talking about right and wrong
So much of what people perceive about optimization is about wanting to test out an idea they have and see if it is better than what is currently there. As an optimizer it is really easy to try and get ahead of that thought process and want to push, “we should test that!” This is where you have jumped into the right or wrong path and what you should avoid. By picking up on that conversation, no matter how well reasoned you think their concept is or how much you agree, you are enabling them to get to a point where either they were right or they were wrong.
This leads validation-based testing programs, which are the least efficient programs out there. This is part of the reason overly focusing on a hypothesis is so damaging (and a 4 letter word in my world) – because it naturally leads to a people thinking they were right or they were wrong.
Yes, you will get winners (on average about 10% of the time) but you are crippling the possible scale of impact and most definitely the likelihood of an outcome.
There is no part of business where you get a gold star for being “right” nor do you get a red mark on a report card for being “wrong”. Why then would you enable people to think that just because they were “right” that you got the best outcome for the business? Why would you let people think that being wrong was a bad thing?
What you should do:
Always – and I do mean always – move the conversation towards what are the feasible outcomes. That one idea that the person has, it may be better, but is it the best option? What else could you do with that space? What are the assumptions that are built into that statement? Are you even sure the thing they want to change is what matters most?
By taking the conversation and moving it towards the higher level questions and not allowing testing to get dragged down to a right and wrong level, you are enabling people to push past their perceptions and to quickly view things in terms of figuring out the best answer, not just validating a single opinion.
The head of marketing believes that the current landing pages need a more direct CTA.
Most people’s first instinct is to want to keep their boss, or their boss’s boss, happy. These are the people who control salaries and promotions. The key here is that all ideas, no matter who or where they come from, get the same treatment. Instead of focusing on how easy it is to test that, you should talk about what are all the possible alternatives that you could make the CTA? Direct could be one, but what is the range? And how do we know the CTA is the most influential part? How could you measure the elements of the page to figure out what matters most and where to spend your resources?
Take these concepts and use those to suggest 2-3 different testing paths you could go down. If you are forced to only focus on the CTA, then ask for multiple inputs on what should go there. Also ask for off the wall examples (we are currently testing “help us help you” and recently had a winner that mentioned rainbows and unicorns) and see what the pool you can put together can be. Ask that same head of marketing to review the list and to kill their least favorite. Then test them all and go with the winner.
Tactic #2: Bring the entire team into talk about a focus area
Everyone is the hero of their own story. We all think this one great thing we have come up with will save the world or improve everything dramatically. We all overvalue our own role in things and undervalue the power of randomness in actual outcomes. The good news is that as an optimizer you get to leverage the power of stochasticity instead of being a slave to it.
Can you spot a good idea when you hear it? There are ideas you think sound good, and there are ideas you think sound bad, but there is no actual way to measure the value of the idea without context. This is why all test ideas are fungible.
What you should do:
It is important to gather as many ideas as possible and to enable a true marketplace of ideas. Take that idea that you are talking about and bring a small diverse group together to get all their ideas. This stops too much focus on a singular area.
Some of those people might think it is a problem, some might like it, but all the larger the range of ideas (the beta) the better the test. In fact, as you talk to the team, only discuss the beta of the ideas and not the specifics of any single idea. Let the people around you worry about their specific ideas. Your job as an optimizer is not to find the perfect idea, it is to run the best test possible. It has become a running joke at my workplace how often I discuss the beta of test ideas.
By bringing everyone together you have a range of outcomes and the test is not dependant on the value of any single one. By focusing on the range of the outcomes and the spread, you are increasing the likelihood of scale and of a positive outcome.
Remember that request from your head of marketing? Gather a group of people from across marketing and other disciplines. Ask them to think about the landing pages and to think about what is feasible to change. Maybe come prepared with a few examples of other pages.
Once everyone starts brainstorming, have them narrow down their changes into the 4 main types:
- Real Estate
If you get a large enough pool, focus on real estate first and come up with as many ways you could arrange the lego pieces of the page. That is your first test. Your follow-up tests are all the other broken apart sections that you just discussed. You will also want to discuss what actions could you take to better understand the page (inclusion/exclusion and MVTs work great for this). By going through the exercise you can build a really easy and large testing series, increase the odds of getting a better outcome, and show that it is not about if any one idea is right or wrong but about finding the best answer from the large pool.
Tactic #3: Start every conversation with what is the measure of success
You constantly get ideas brought up for you, from that great CTA to you need to change the tone and message of a page. People want to “connect” with users more or improve usability. Others believe you need a better “user experience” or that you need to improve the bounce rate. All of those sound great, I mean, who would argue about making a better user experience? What exactly is the definition of a “better user experience”?
The problem of course is that those are all means to an end and not the end in and of itself. The goal of any business is to make money. You want to accomplish this by pleasing people, but the how is tricky and more importantly what is the actual cut off that is acceptable? Is a 3% drop in “user experience” worth a 5% gain in revenue? How about a 2%? 20%?
It is easy to talk about what you want to happen, but the job of an optimizer is to focus on what did happen and what was best for the business.
What you should do:
Start all conversations with this: what is the end goal? Are we trying to improve revenue? Are we trying to improve conversion rates? Not what do you think will happen, not what you think will lead to that end goal, but what is the actual end goal. Once you have agreement on that end goal DO NOT TALK ABOUT ANYTHING ELSE!
If the things they think will happen do happen, and it is good for the business, then great. If something else happens, that means that you found something that makes you more money.
Those landing pages you have been working on, what is their goal? Not the perception goal, what is the business value? Is it conversion? Is it leads? Whatever that end goal that is all you talk about as well as all the ways that might lead to that.
Someone thinks they need to reduce bounce rate? Geat, what is the impact on RPV? What about someone that wants to connect more with users? Great, what is the impact on RPV? How about changing the branding on the page? Great, what is the impact on RPV?
Never let people trick you into talking about multiple metrics, or sub metrics, or what they want to happen. It is never about hope or want, it is about what did happen and can you exploit that information.
Tactic #4: Celebrate being “wrong”
You may have stopped talking about right and wrong, but now you have a result that wasn’t what person X picked. They are upset and asking for a bunch of details about why did it not work? Can you double check the data? Did our users change?
Essentially that person or people are now fast at work trying to come up with a path to explain away the cognitive dissonance that they are feeling. They were sure what they wanted would work, and now you’re saying that it just didn’t make the grade.
If you don’t watch out, they’ll start losing faith in testing, and you, when it comes to making them look good.
What you should do:
Talk about how great it is that you found something that worked even better. It was never about a single idea and it is never about fishing for reasons – it is about the constant search for a better outcome. While that person is caught up on why their idea didn’t work, you should be moving them towards the fact that something outperformed what they thought would happen. Not only does this mean the company will now make more money, but it means that you learned something new or re-evaluated false knowledge. That isn’t a bad thing. That’s the best thing that could happen.
The key here is that it is not about their opinion or a single person. It is about the constant future discovery of what works and what works best, which means that every time someone was wrong, we found something better. Hammering that message home and celebrating the amazing returns you get from being wrong and you will open up people’s eyes to just how little they really understand about what changes user behavior and makes your company money.
You tested out that series of landing pages and the one your head of marketing wanted came in next to last. It was still a little better than the old page but it was significantly worse than a few of the other variants. Your boss is now trying to figure out why that person was wrong and what data to show them.
Stop that conversation and talk about how much more money you are going to make. Talk about all the other places this new piece of information might get leveraged and what future tests this opens up. Talk about all the things you gain and let it be known that there is no downside, just a little ego deflation. By moving the conversation towards the good you help people connect with the outcome more and allow them to feel ownership of the new improved site.
Tactic #5: Proactively generate a roadmap of where you will be focusing
People love your results (or just want to see their ideas for change come to fruition). They present you a list of test ideas. These ideas spread the gamut from different pages, copy, presentation, function, or flow. They come from a variety of sources or just one source over and over again (usually a higher up).
This, of course, misses the point completely. Test ideas are fungible – one is just as good as any other until you know what the outcome will be for sure. It also allows people to focus on what they will work best or what they think matters instead of letting the data tell you both of those key pieces of information. Even worse, this list is almost never based on efficiency and can often lead to resource drains on low value and higher cost tests.
What you should do:
Come up with a prioritized list of site locations.
It might be your homepage, your landing pages, product pages, search results pages, and your shopping cart. Or it could just be all 3 pages of your shopping cart. The key is to understand what pages have enough traffic and enough information to be valuable. By talking about locations and not specific tests, you free people from thinking in terms of what they think will work best or what they want to see happen. You are also enabling larger test series that may or may not lead to the higher resource tests that everyone is clamoring for.
To give a reference, there are hundreds of page types across our sites and yet I only care about 7 specific pages, because those are the only ones that have proven to have enough influence and enough population to matter to the business. I keep an active list of 3-6 months worth of focus areas that we will work on, with the understanding that the list changes constantly based on the results of the current tests and where they take us.
Your marketing head now wants to turn to your product pages because they are still a little miffed about the results from the landing pages. Instead, you present the list of when you will be focusing on different pages.
You use this to encourage them to list their ideas. When you get to that page, after you complete the series on the landing page and your homepage, you will include them in the larger testing series.
This allows them to have their say while keeping focus on the most important and valuable parts of your site. It also helps people think of pages as a never-finished, evolving item, rather than something you just tweak once in awhile when the whim hits them.
Everyone you work with wants to do the right things.
In their own heads they are most likely already the master of their own domain. More than anything, optimization becomes a level of accountability for those notions and ideas that they hold most dear. People don’t like change or being wrong and will fight, sometimes consciously but most often subconsciously, when there is something that promises by its very nature to do just that.
Your job as an optimizer is to help people be better than they want to be. To allow them the freedom to get past their discomfort and their lack of understanding to instead focus their skills on what does matter most and not just what they think matters most.
It takes a large variety of tricks to get people to eat their veggies, floss, and not to give in to their assumptions. By focusing on the structure you give them and the environment that their ideas are allowed to go through, it’s easier to get them to do what is healthiest for the business, even if they don’t realize it.
You are not being malicious or sneaky by changing the conversation, you are simply taking it away from your own or anyone else’s ego.
You are using overt and covert tactics to cut through all the mental and organization BS and instead grow together towards a goal. Be a leader, change the conversation, and stop allowing people to go in other directions and the results you can achieve together will change your business forever. Give in to what feels good, to you or to your team, and you will always be just another cog that is awash in the sea of bias. The call is yours to make.