Marketers of all stripes are obsessed with tools.
Though no optimization program has ever hinged on which tool you used, there are important distinguishments between A/B test tools, from the statistics they use to the price and more.
One thing that is often either overlooked or misunderstood, is the difference between client-side and server-side testing tools.
Client-Side, Server-Side: What’s The Difference?
Here’s the basic difference between client-side and server-side testing tools:
Server-side tools are different, in that no modification occurs at the browser level. When the visitor lands on your page, a randomly picked version of your test is sent straight from your server. Yes, this means you have to involve your developers in testing, but it also gives you robustness and flexibility.
There’s a third type of tool as well, closely related to server-side capabilities. There’s only one company I know that uses this method (SiteSpect) so it’s not super common. What they use is something called the proxy server or intercept method.
Basically, the tool acts as a proxy server between the visitor’s browser and your server. It controls the site content that is presented to the visitor, so it eliminates most of the problems that occur with client-side tools, while retaining the robustness of server-side tools.
Okay, But When Should I Use Which?
Client-side tools offer ease of use, faster execution and a WYSIWYG editor. You don’t necessarily need to take up your dev team’s time to get experiments running.
That said, server-side tools have benefits as well. An article from Google claimed that server-side implementations offer more flexibility to do things such as:
- Run experiments for websites with dynamic content.
- Test non-UI changes that still have an affect on your objective. For example, a database query result set that is returned to a user.
- Integrate Google Analytics experiments with your service (e.g. Content Management provider).
- Manage experiments using your own optimization platform.
In summary, they offer more flexibility. They also tend to be more robust, and they eliminate a few of the weaknesses of client-side tools. Some of these potential weaknesses include the Flicker Effect (which we’ll talk about below), load speed, and flexibility.
Though, as another way to look at it, Nate Weiss (CTO of Conductics) says that the ‘skipping IT’ benefit of client-side tools could have a cost for some companies. He explains:
In summary, use different tools for different uses and goals.
The Flicker Effect
As mentioned above, one of the weaknesses of client-side tools can be the Flicker Effect. What is it?
Basically, it’s when, for very small amount of time, your website visitor sees the default page before the variation loads. It’s also known as a “Flash of Original Content.”
This generally happens because of an asynchronous script with the testing tool. While it has the advantage of not slowing down the page, this type of script can cause a Flicker Effect once in awhile.
This matters because the human eye detects, and we change our behavior when we know we’re part of an experiment. Though, there is the surprising (fluke) occasion that it helps a test. Take this example from an Unbounce blog post:
As David Hauser, co-founder of Grasshopper, said in the article:
Most of the time, though, Flicker Effect results in statistically significant differences in test results. That’s totally affecting the test, and can skew the outcome. The goal is to reduce the flicker to 0.0001 seconds – so people won’t see anything.
So, I mentioned that Flicker Effect is really only a problem with client-side tools. What are some ways, if there are any, that we can mitigate this?
Synchronous vs Asynchronous Scripts
Client-side tools can be load scripts in two distinct ways: synchronously and asynchronously. What’s the difference?
Synchronous loading is when a testing tool’s script loads completely before the page starts to load. It calls back to the server and waits for an answer. Put another way, the scripts are loaded sequentially starting with thetag.
For example, Optimizely uses a synchronous snippet to try to prevent the Flicker Effect and has “invested heavily in a balanced CDN system to make sure the impact on the page is minimized.”
Asynchronous is when some scripts can be loaded simultaneously. They load while the testing script runs, possibly causing a flicker effect if the original content loads a split section before the variation.
You can run client-side tools like VWO and Optimizely either way. In general, page loading speed is important, so asynchronous code is usually preferred. The batched scripts allow the pages to load faster, and you can usually prevent the Flicker Effect with a few measures.
As Paras Chopra said in a GrowthHackers thread:
Alhan Keser from WiderFunnel also expressed a similar sentiment. Unless you’re running a massive site with tons of visitors, page speed is of primary importance:
Other Tips To Prevent Flicker Effect
One way to prevent the Flicker Effect that Optimizely outlined is to add a timeout on asynchronous scripts. Their default setting is 1 second. This means that if the A/B testing script isn’t loaded in that time the A/B won’t be executed. If the visitor has a slow connection, that visitor won’t be included in the test.
Some other ways to prevent the flicker effect:
- Optimize your site for speed. Make sure everything, especially the images load fast. Move your site onto a fast web host. Use compression, optimized images and content delivery networks.
- Take your A/B testing tool script out of the Tag Manager (if you use one, like Google Tag Manager).
- If you can, do split URL testing instead.
- Pay attention to the html elements you change. If you made changes on the page itself (as opposed to doing split URL testing), the key thing to keep in mind is the amount of code that was changed. If you changed thousands of lines, you’re far more likely to see a flicker effect as opposed to changing button copy. So the scope of changes influences everything.
What Client-Side Tools Do Right
Client-side tools let marketers run tests accurately without the help of developers.
That’s not a small accomplishment. Development resources are expensive, in time and cash, so enabling testing independent of product development is efficient. Optimizely put it well:
“When it’s time to take action and make changes to your site, there is no replacement for dedicated technical resources — in most organizations, these resources are hard to come by and typically demand waiting in line.”
Sean Ellis introduced the idea of high-tempo testing as a key mechanism for growth.
The basic premise is that the faster and more efficient you can run tests, the more of them you can run, and the faster you can grow. Though their bottlenecks were mostly at the ideation and organizational levels, many optimizers find bottlenecks at the development and implementation levels. Client-side tools using visual editors solve this.
Andrew also noted that, while lots of tools provide easy-to-use visual interfaces to create treatments, the automatically created jQuery code is often pretty bad. This results in cross-browser compatibility issues and slowness. Always have a developer check and improve the code – or learn jQuery. It will really help.
Server-Side Tools: When To Use?
As I’ve mentioned, a huge benefit to client-side tools is the speed and ease-of-use for not-super technical people. So what, then, are the benefits of server-side tools? Why would companies use them?
He continues to talk about the applicability to native mobile applications:
Another topic of fewer conversations that Nate brought up to me was that, depending on the nature of your site and the tests you plan to run, it may be more respectful for your visitors to use server-side tools.
As Nate said, “consider which is more important: your convenience when setting tests up, or your visitor’s user experience (and impact on their mobile data plans).”
In addition, server-side tools broaden your capabilities for optimization beyond web-based-visual presentation issues, in a way. What they do is allow you to think a little different about optimization, more towards optimizing any of your transactional marketing efforts.
With client-side tools, the main topic of discussion is around the flicker effect. This is an issue, but it isn’t the only, or the largest, issue to be discussed with tools. Things like flexibility, control, and privacy affect your optimization as much or more than flicker, especially if you take the precautions to mitigate that risk.
Which Type is Your Tool?
Here’s a chart that quickly categorizes popular tools. Let us know if we’re missing any or getting anything wrong:
|Conductrics (both capabilities)||A/B Tasty|
|Divolution (both capabilities)||Adobe|
|SiteSpect (both capabilities)||Conductrics (both capabilities)|
|Optimizely (both capabilities)||Convert.com|
|A/B Tasty (both capabilities)||Divolution (both capabilities)|
|Optimizely (both capabilities)|
I’m simplifying things, but here’s the bird’s eye difference between server-side and client-side tools:
Client-side tools use either synchronous or asynchronous code, both of which have pros and cons regarding pagespeed, flicker effect, and loading issues.
Client-side tools are easier for marketers to use, and you can often get tests up and running faster without burning development resources. However, server-side tools offer greater flexibility and control. They also protect your privacy and security better, and for web apps, reduce the need for ‘heavy pages.’
Both tools have their use cases and don’t need to be used exclusively. However, far more important than the tool you choose is the process by which you optimize. Don’t overthink it with tools.