18 Top A/B Testing Tools Reviewed by CRO Experts

18 Top A/B Testing Tools Reviewed by CRO Experts

At this point in time, the field is flooded with solutions. Finding a proper A/B testing tool isn’t the problem anymore. Now, the problem is choosing the right one.

If you work in conversion optimization – whether at an agency, in-house, or as a consultant – you almost certainly run A/B tests.

Though you could hire someone full time to sift through and analyze the pros and cons of each tool, it’s easier to learn from the experience of others and make a decision based on that.

To help you with that, we’ve assembled a list of the most popular a/b testing tools and corresponding reviews from A/B testing experts to help you in your decision making process.

Without further ado, here are the top A/B testing tools used by experts (click below to navigate to a specific tool):

AB Tasty

AB Tasty is a reasonably priced and simple to use tool that serves as a good starting point for companies just starting in conversion optimization.

AB Tasty offers A/B testing, split testing, multivariate testing, and funnel testing capabilities. Using AB Tasty’s visual editor, you are able to quickly put together variants and tests and receive real-time reports with confidence levels on your goals. At the same time, you have full control of testing through advanced targeted scenarios, allowing you to trigger tests based on URL, geolocation, demographics, etc. You are also able to enrich the platform’s data by integrating existing data from CRMs and DMPs.

Chloe O’ Keeffe, Associate Conversion Consultant at Conversion.com, shares some pros and cons:

Chloe O’ Keeffe:

“Pros:

  • The interface is really clear to understand and launching or monitoring tests is easy. There are a lot of available configuration options, for example you can add multiple filters such as user IP address, localisation or cookie values.
  • Test set up is easy – the editor is easy to use with a useful preview mode – users with little technical experience can easily modify pages and create new designs or use the unique library of widgets – AB Tasty Engage.
  • Many integrations are available and useful for validating test results – such as Google analytics, Clicktale or Kissmetrics
  • Pricing is mid range – not as expensive as other leading competitors in the market
  • Customer service team are very responsive and quick to tackle problems if any arise

Cons:

  • The statistical significance calculator is somewhat basic – rather than a numerical value this is displayed as a bar chart
  • There can occasionally be a lag in test results which requires a response from the customer service team and can delay the testing process.”

The basic starter plan rounds out to $249/month and includes 50,000 unique tested users and carryover of unused credit.

Adobe Target

Adobe Target is a popular enterprise tool that combines targeted testing and personalization.

Adobe Target walks you through a three-step workflow where you first create the variant, then target the variant based on visitor segmentation, and lastly customize your goals and settings for the test. Based on this emphasis of targeting, it should come as no surprise that Adobe Target’s most exciting feature is automated personalization.

Target’s self-learning algorithm continually and automatically adjusts itself to make associations between visitor behavior and content in order to give you the best results possible. Real-time adaptation of content can help you better meet the needs of diverse visitors.

Here’s a review from Sean Patterson, a Digital Analytics and Optimization Consultant at Conversion.com:

Sean Patterson:

“An enterprise testing tool with a wide range of capability and customisation. It has similar features to other enterprise testing tools such as Optimizely, including personalisation and recommendation modules for more advanced optimisation.

The real power with Target, as with all Adobe Marketing products is when they are integrated through the Adobe Marketing Cloud. Segments uncovered in analytics or through their data platform can be pushed to Target, and results from Target analysed within Analytics for increased insight.

As with many Adobe products, integrations can be complex and require subscriptions to the other tools in the Marketing Cloud to be most effective. As with Analytics, additional modules for Target such as personalisation and recommendations come at a cost.”

Pricing for Adobe Target is available on request.

Apptimize

Apptimize is a testing tool focused on mobile optimization and is a perfect choice if mobile is an important platform for your business.

Apptimize offers you a full control visual editor where you’ll be able to rapidly create new variants and target them at different user segments. You’ll also be able to add dynamic variations and code blocks that support Objective-C, Swift, Java, HTML5, Xamarin, and React Native. Their randomized experiment methodology also ensures your experiments will return statistically significant results.

We sourced reviews for Apptimize from G2Crowd. Here’s an anonymous review talking about some things they liked about Apptimize:

It’s easy to set up and track tests. The code needed to implement the tests is straightforward. It integrates with Localytics.

And here’s a review discussing some disappointments:

“The real-time changes that we can deploy are acutally limited in the scope – so, not all of our original goals can be met with this tool. Also, we have to plan in advance what sections of parts of our app we would like to change, and only after that we can start using this feature, which means that there is a lag/delay between the time that we want to change something and the time when it actually becomes a reality. For example, with our sprint structure this delay can be as long as 3 – 4 weeks.”

Apptimize starts at $995/month and supports up to 100,000 monthly active users. You can also request a custom plan based on your needs.

ChangeAgain.Me

ChangeAgain.me provides a visual interface to Google Experiments for A/B Testing.

A good solution for those companies that already have Google Analytics and want to try split testing.

They bill based on the number of tests you run, not the number of visitors as other split testing solutions do.

No coding skills necessary. You can launch new tests with a point-and-click visual editor

Conductrics

Conductrics is an advanced tool that offers methods from AB Testing blended with Machine Learning to deliver optimal experiences, for each user, automatically.

You can use it as either a server-side or client-side tool, and you can pretty much customize it in anyway you’re trying to use it.

Here’s a review from Experimentation and Optimization Expert Andrew Anderson:

Andrew Anderson:

“Pros – Offers a lot of flexibility as far as ways to interact with your systems, from APIs to one tag solutions. Is at its best when it is not being used for just straight AB Testing, but instead as an active decision agent that is constantly adapting and creating experiences.

Because of the ability to select factors instead of just blindly adding them all in allows for this functionality on far smaller sites than other similar tools (Adobe Target Premium/1:1/Touch Clarity, X+1). Very robust system with very low variance and a wide variety of features. Great support, although the help menus are a bit technical for most groups.

Cons – Requires more knowledge about decisioning agents then most programs have. Interface is a little clunky with the current page interaction set-up rather annoying (though this is supposed to be fixed soon). QA is less than ideal. Data reporting can use some improvements, especially graphically. Has a bit of a learning curve when it comes to reading the Targeting report. Still requires a fair amount of traffic to reach full potential.

Overall – Conductrics is a great tool for programs that really want to tackle adaptive learning personalization or want a lot of flexibility when it comes to interactions.

It does require a lot more knowledge and more of a learning curve then very basic tools like Optimizely or VWO, but in the end can provide far better functionality and more flexibility than just about any other tool on the market. Great tool for those programs that are above the standard tools but not at a fortune 100 size.”

Pricing for Conductrics is available on request.

Convert

Convert is perfect for work for small to medium size clients and businesses doing in-house optimization. It is well loved for its exceptional customer support over live chat.

Convert operates using an intuitive drag and drop system. You can create A/B, multivariate, and split tests using the provided visual editor and break out more the more advanced style sheet editor for dynamic pages, giving you full control of your content. It also boosts a smooth onboarding process of your CMS and e-commerce platform and seamless integration with Google Analytics. Personalization is also an option with over 35 different elements you can use to build customer profiles.

Here’s Julien Le Nestour, an applied behavioral scientist and CRO consultant based in Sydney, giving his opinion:

Julien Le Nestour:

“Convert Experiments is a fantastic tool that should be considered and evaluated by any CRO agency or end-user client that also evaluates Optimizely and VWO. They offer a level of functionality that is globally on par with them for testing, but at a lower price level once you go above the free level offered by Optimizely.

The Convert Experiments team is very experienced in testing and they go to some incredible lengths to offer an unparalleled amount of support when you are building or running your tests. In my experience, they will assist you in all technical aspects and issues and solve them for you as part of their plans. You can also ask them to directly create tests for you if you don’t have anyone on your team to do it as an additional service. The only (potential) limitation in using Convert Experiments would be a more traditional approach in measuring statistical significance, but whether or not it’s an important point for you depends on your views on this subject!”

Convert’s most basic plan, Convert Plus starts at $599/month and gives you access to 400,000 unique users per month and 5 active domains for testing.

Dynamic Yield

Dynamic Yield is a powerful suite of personalization and engagement solutions.

With Dynamic Yield you’ll have access to a wide range of tools. Most importantly, you’ll be able to continuously A/B test and optimize across platforms (desktop, mobile web, apps). These tests are predictive in nature and leverage the capabilities of machine learning algorithms to continuously improve ROI.

Dynamic Yield, however, specializes in advanced omnichannel personalization solutions. You’ll be able to segment and quantify every user interaction and response and dynamically adjust your content to best suit each individual. Combine your segments with personalized notifications to get the most out of this particular tool.

According to a review by Martin Nicholson on TrustRadius, some pros of the tool are:

  • “The out of the box features are impressive and quick to implement. Overlays, messaging and content changes can be made instantly enabling tests to be carried out quickly.
  • Creation of custom actions enables us to bridge the gap between our online and offline tests by injecting code changes directly onto the page.
  • The test segmenting features are very robust, actions can be triggered by device, ip, cookie, audience type and at specific times of day allowing us to test whatever we want, when we want.’

And some cons:

  • “We’re not using all the features of the tool, there’s so much to choose from it’s sometimes overwhelming. The support team however are superb and have always been able to help.
  • More integration with GA would be useful.
  • Some of the features do require some technical knowledge to implement, again support are always on hand to help.”

Pricing for Dynamic Yield is available on request.

Google Optimize

Google Optimize is a Google tool that lets you test and tailor different variations of your website.

The main selling point of Google Optimize is its native integration with Google Analytics, allowing you to gain automatic access to rich behavioral insights. You’ll also be able to target the valuable segments you’ve already discovered using Analytics.

Here’s what Charles Farina, Director of Growth & Development at Analytics Pros has to say:

Charles Farina:

“Google Optimize is a completely free A/B and multivariate testing enterprise testing tool. With a visual editor, code editor, and native integration with Google Analytics it is the best tool for companies that are just starting with testing. Many nice-to-have features are still missing, like an image upload feature, but Google continues to roll out enhancements every week. You have to use or install Google Analytics to use Google Optimize.

I’m excited for the next few years, as Google Optimize should disrupt the testing tools market the same way Google Analytics did when it was released and started to get more and more enterprise features added.”

Try out Google Optimize for free and upgrade to Google Optimize 360 for enterprise-level website testing and personalization.

Kameleoon

Kameleoon is an advanced optimization platform that offers a variety of features.

Using Kameleoon, you’ll be able to run advanced A/B, split, and multivariate tests quickly and easily. You’ll also be able to understand key user insights with their navigation analysis tool and test different user segments with over 40 targeting criteria. Dynamic traffic allocation algorithms also make sure your traffic is optimally divided in order to shorten your decision cycle and improve return on investment.

Julien Nouet, Global Frontend & UX Manager at Lacoste, reviews it here:

Julien Nouet:

“We first started using Kameleoon to A/B test our website to make sure our clients were happy with the experience we offer. We chose Kameleoon because of its reliability for testing at high velocity and its advanced features.

The platform met our growing needs as we also now run personalization campaigns, not only to improve our website performances but to create tailored experiences for our visitors.

We appreciated:

  • Top notch anti-flicker technology
  • Strong and innovative architecture (NoSQL, Elasticsearch)
  • Works with React
  • Integrates well with our other solutions
  • Powerful targeting features
  • Ease of use of their WYSIWYG editor
  • Click tracking easy to setup
  • Test results can be filtered directly (by device for example)
  • Possibility to have a both visit or visitor-based test samples

Some cons:

  • WYSIWYG Editor a bit slow to load
  • Some elements missing in the preview recap (capping, script setup, …)
  • UX has some room for improvement
  • Kameleoon teams are innovating a lot … and the platform may be down for shorts periods of time. No biggie.

We’ve been working with the solution for more than a year now and we’ve had a great ROI. Overall, it’s a strong optimization platform both quite simple to use and feature-rich. As a side note—not immediately related to the tool but, their customer success team is worth mentioning. They are quick to respond and provide valuable strategic insights.”

Get started on Kameleoon by creating a free account and then upgrade to an enterprise plan if the tool suits your needs.

Maxymiser

Maxymiser is a powerful testing tool that helps you design and run sophisticated campaigns.

Like other tools, Maxymiser offers a visual editor that will help you set up variants and tests as well as an extensible coding environment for more complex projects.

However, Maxymiser shines through most in terms of personalization and targeting. This tool provides you with the ability to build rules directly using the visual editor to target profitable customer segments and display personalized content. Most impressively, Maxymiser automatically builds a customer profile that aggregates data on historic and in-session behaviors and visitor attributes for each unique visitor to your page.

Pricing for Maxymiser is available on request.

Monetate

Monetate is another well-loved testing tool that has a reputation for providing fantastic client services. They provide both a testing and segmentation tool as well as a personalization tool.

Using Monetate Test & Segment, you are able to perform advanced A/B tests and multivariate tests. They also have built-in measures of statistical significance and dynamic testing which adjusts traffic allocation to optimize for ROI. Monetate’s Intelligent Personalization Engine is an advanced machine learning engine that determines which actions will help you achieve a given goal. This means you can automate and optimize personalization decisions.

 

Pricing for Monetate is available on request.

Omniconvert

Omniconvert is another new player in the testing tool market. They position themselves as a sort of all in one optimization tool: pop-ups, on-page surveys, A/B testing, personalization, segmentation.

They’re pretty affordable for all they offer too, even offering a free version for under 10k visitors a month.

They’re an especially good tool for beginners, affordable with lots of features included.

Here’s a review from Justin Rondeau, Conversion Optimization Director at Digital Marketer:

Justin Rondeau:

“What I like best is that this product does more than just A/B test. Qualitative data is crucial, and the tool helps you gather this vital information. I also like the personalization aspect of the technology.

Overall, Omniconvert captures where the market is going, but still uses A/B testing as a backbone to validate results gathered by it’s other components. I think the user interface could use some tweaking, maybe work on a more logical flow for finding what you need.

I am always trying to squeeze more productivity out of a page, whether it means more sales or leads. I have been able to make strategic design changes with the testing tool and gather user data with the survey functionality.”

Omniconvert offers a variety of plans to suite any need, from a forever free introductory plan all the way up to custom solutions.

Optimizely

Optimizely is one of the most popular, WYSIWYG experimentation tools and a go-to choice when working with any size company.

At the root of it, Optimizely is focused on web experimentation and personalization. They offer easy-to-use A/B and multi-page experimentation and supports experimentation across platforms (web, mobile web, iOS, Android, etc.) and shows you results in organized formats using their user-friendly Stats Engine.

More importantly, Optimizely allows you to target your experiment variations based on ad campaign, geography, cookies, and a variety of other dimensions to help you design and optimized personalized content based on your audience segments.

Diane Leung, Associate Consultant at Conversion.com, shares her thoughts:

Diane Leung:

“Optimizely is the clearly the market leader out of the experimentation platforms on the market.

They’ve recently completely updated the platform with the launch of Optimizely X and continue to invest in expanding the reach of experimentation through tools such as server-side testing and personalisation.

Optimizely is suited to both the novice user and expert user, with features such as the visual editor enabling users with limited knowledge in coding to easily create and launch experiments. Optimizely also does the heavy lifting from a statistical point of view through their stats engine, so the difference in performance between your variations can easily be measured and identified.

As for negatives, the price of Optimizely might be prohibitive for some users that are just starting out with experimentation.

The experiment result reports are also fairly basic and lack advanced features to allow you to really deep-dive into data for specific segments of your audience. These limitations mean it’s very important to ensure all tracking is set up correctly prior to the experiment running or to integrate Optimizely with other analytics tools.”

Optimizely’s Pay As You Go is a good way to go if you’re just starting out and costs $49 for every 1,000 monthly unique visitors you test on. Even if the same visitor sees many experiments, the visitor is still only counted once.

Qubit

Qubit is a testing platform focused around personalization, making it one of the strongest segmentation tools on the market.

With Qubit, you’ll be able to run A/B tests and multivariate tests in order to measure progress and effectiveness of various personalization techniques. You’ll also get access to abandonment recovery, product recommendations, and social proof tools, making this software a great choice for e-commerce businesses.

According to a review on Trustradius by Niall Totten, some pros of using Qubit are:

  • “Great A/B test ideas based on both quantitative and qualitative evidence.
  • The Qubit consultants and engineers we have worked with have an excellent ‘can do’ attitude.
  • Excellent communication and clarity on test results.”

From the same review, the cons are:

  • “There is room for improvement on the velocity of getting new tests developed and QA’d.
  • I would like to see improvements to the exit survey interface to make it easier to flag, categorise and disseminate feedback to different areas in the business.
  • Depending on the test/experience it can add extra weight to page load times and cause ‘flicker’.”

Pricing for Qubit is available on request.

Sentient Ascend

Sentient Ascend is a new generation testing tool that merges machine learning algorithms with conversion optimization.

The concept is simple, genetic algorithms can find optimal combinations of design choices faster and with greater accuracy than humans can. By isolating the best local maximum, the machine assembles the highest performing segment and pushes forth optimal personalization solutions. Of course, this also means you can test all of your ideas at once and see evolution in action.

Here’s Guido Jansen from Euroflorist explaining some pros and cons of the tool:

Guido Jansen:

“From our trial with Sentient Ascend we can conclude that their method requires around 75% less traffic compared to a normal full factorial MVT test. This means you can either test faster or with 4 times as much variants.

One caveat is that you don’t exactly know up-front how much traffic the AI system will need so that currently is a bit of trial-and-error. Sentient is working on a solution for this.

This creates interesting opportunities for CRO.

Especially testing some more extreme lay-out combinations can work really well and it’s a great way to provide some uplift and get to your local optimum faster. But this being a bit of a black box and more vague results: you don’t know WHY certain variations work best.

So yes it will allow for more testing and more uplift, but don’t expect AI to replace well hypothesized A/B tests based on thorough user research any time soon. Combine the the two and you might have a golden combination.”

Viljo Vabrit, Managing Director at CXL Agency, wanted to point out a few issues with the tool as well:

  • “You can’t edit active tests. If there’s a small bug or something breaks due to a change in the control you either stop the experiment and create a new one or contact the support and hope they can make the change for you as soon as possible. Same also goes for the changes related to audience and URL targeting.
  • QA-ing everything is a pain. And I am not talking about going through the thousands or even millions of possible combinations. Just previewing a specific combination requires you to understand the complex query parameter logic in place.
  • The visual editor can be quite slow and crashes occasionally, sometimes with all of your changes.”

There are many articles about Sentient where you can read about their evolutionary algorithms or about specifics of the platform.

Pricing for Sentient Ascend is available on request.

SiteGainer

SiteGainer is a company that offers a full suite of conversion optimization tools, including A/B testing, multivariate testing, personalization, heat maps, popups, and surveys.

The tool is quite easy to set up and use, and creating everything from simple a/b tests to surveys, popups, and more is very intuitive.

SiteGainer offers a free trial and then pricing plans are based on monthly traffic, payment schedule, and additional services. 

SiteSpect

SiteSpect offers a server-side solution and is a great choice for more technically advanced testers. By editing HTML before it leaves the server, it is popular with self-hosting companies and security-conscious clients.

Of course, SiteSpect does offer a visual editor tool to help make setting up optimization and testing processes convenient and efficient. What makes it different is its ability to give you access to both client and server side, allowing for a comprehensive view of the entire customer experience and high performance for speed and scaling.

Here’s a review from Judah Phillips, an analytics expert, consultant, and author who wrote Building a Digital Analytics Organization, Digital Analytics Primer, and Ecommerce Analytics:

Judah Phillips:

“SiteSpect is one of the global leaders in testing and optimization. They are pioneers in the space with a decade of experience. Their patented technology provides the ability to test and measure testing results without having the change code on your website or mobile experience. The technology pattern matches patterns the customer defines and matches test content based on pattern recognition in real-time with no latency and total failover. Features exist for speeding up the performance of tests and running many different tests simultaneously. Unlike many testing vendors, SiteSpect supports all markup languages (HTML, WML, JSON, and XML), style sheets (CSS), or scripting languages (like JavaScript). Any type of static, dynamic, and rich content, including audio and video can be tested. They were the first company to offer a comprehensive mobile testing solution.

While SiteSpect used to require specialized hardware, the capability can now be deployed in the cloud or on-premises. Like other vendors they have figured out the “flicker” problem and continue to innovate mobile testing. Detractors of SiteSpect report that the product has technical complexity and isn’t something that can instantly used by a marketer without engineering support. It’s also a tool that requires people to understand what they are doing and plan for testing. The reporting interface has been criticized but is widely regarded as industry standard. Finally, it’s not a tool for people who want to spend $49 or more per month. It’s a powerful software for enterprises that are serious about testing and optimization and want technology that can accommodate for enterprise requirements under an SLA.”

Choose from a variety of packages that vary in integrations, features, and accounts depending on your scale and needs.

VWO

Visual Website Optimizer (VWO) is another big player in the A/B testing software market. It is a top choice for businesses with slightly smaller budgets and popular amongst many marketers.

One of the reasons it’s so popular is because of the variety in tools it gives you access to, which VWO separates into four categories: testing & experimentation, research & user feedback, analytics & reporting, and targeting & personalization.

With VWO you are able to create multiple variations of web pages and track various conversion goals. Like Optimizely, VWO also provides statistical analysis of your results. A favorite feature of VWO is its built-in heatmap tool that tracks visitor click behavior and tracking habits.

Notably, if personalization is important to your work, VWO offers easy tools for segmenting and delivering targeted offers.

Here’s a review of Alex Harris, Conversion Optimization, Director at Perficient:

Alex Harris:

“VWO is an excellent conversion testing platform. One of the main pros is the visual interface. In my opinion, I think the UX is more user friendly and the reports are more visually appealing. The reports and charts are easier to analyze than other testing platforms. Another good thing about VWO, is that is very easy for anyone to walk through creating a new test. It is a simple step by step process that I think makes more sense, especially to someone who is new to A/B testing.

Previously I would would have recommended another A/B testing tools if you were a developer, but VWO recently improved their code editor which in the past was really buggy and caused more headaches than needed.

The only downside that I have ran into is related to the WYSIWYG, which a good optimizer shouldn’t really use anyway.

In the past, when I didn’t use the code editor the WYSIWYG was really buggy. For example, I would create a test and if I didn’t make a copy of the code that was created, sometimes the variation would break and I would lose all my work. This may not be the case anymore since I only use the code editor now, but it was rather frustrating when I lost my work in the past.

Also in the past their support has a slow response time related to tickets created for support.”

The standard plan for VWO starts at $49/month and accommodates up to 10,000 monthly users.

Conclusion

There you have it: the top a/b testing tools, reviewed by the experts.

While each has their own pros and cons, it’s likely you’ll have to some some strategic thinking and exploration to get the solution that is right for your business and specific situation. Hopefully these reviews help you in the process, though.

Join the Conversation Add Your Comment

  1. Very useful article for people just getting into A/B testing. I think you could an extra layer to it by mentioning areas such as likelihood to encounter flickering, in-tool segmentation, and machine-led personalisation, as a lot of A/B testing tools are all now on a similar level; visual editor, basic reporting, basic personalisation, and maybe a few qualitative tools.

    Reply
  2. Great overview! I would have loved some kind of scoring/direct (feature) comparison, but realize that this is a huge job..
    (BTW I have long believed Hotjar to go into A/B-testing as well to go “full suite” on the features – it’ll be a great package if it happens. We’ll see. :-D).

    Reply
  3. Great list – but to be honest I’ve no idea how ‘Webtrends Optimize’ has been left off?
    We’ve been going longer than many of the above and have self-serve, hybrid and managed service options. We have both a WYSIWYG interface as well as the advanced UI for true devs to use.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Current article:

18 Top A/B Testing Tools Reviewed by CRO Experts