There are a lot of multivariate and A/B testing tools in the marketplace. Which one fits your company’s needs best depends on a myriad of factors, including, but not limited to: budget, number of visitors to your site, how you compile your analytics, and where you need to test. As an agency with diverse clients, who all have different customers and integrate their analytics in different ways, it’s our practice to stay platform-agnostic as much as possible. We do this by continually reviewing the top tools in the marketplace internally before recommending anything to our clients.
Our most recent review uncovered some exciting trends in the field. The floor of testing platforms has been elevated substantially, while the flagship enterprises seem to be making opinionated moves to distance themselves in very substantial ways. (Note: In the mirror weeks since conducting our audit, we are seeing some of these platforms respond to those opinionated moves by building out similar tools and closing those gaps once again.)
In our context at roboboogie, saying we are testing platform agnostic is saying we want to use whatever tool helps our customers test and improve their impact on their industries the most. Remember, it’s not about a tool, but about the impact we can have on our client’s customer base. Because, happy customers convert.
This list is not a list of which tool is the best. What this list hopes to be is a short, but detailed, description of some of the most powerful experimentation tools in the space with a touch of the best ways to use them for your testing purposes. We will be taking a look at Optimizely, Visual Website Optimizer (VWO), Convert, Adobe Target, and Google Optimize & Optimize 360.
To start, I will say they all claim to have ~99.99% uptime, some type of flicker-free testing, and their version of a “What You See Is What You Get” (WYSIWYG) drag and drop editor.
Here’s some information on the most common statistical models used for testing analytics, and the two we see from these five testing platforms. The Bayesian Statistical Model is the most common, but Convert and Optimziely believe the Frequentist Inference Model has an important place in moving along experiments quickly. Here’s an article on Bayesian vs. Frequentist statistics from Optimizely and this xkcd comic is just too funny.
Optimizely is the optimization platform where you can test “across every device, every channel, every customer touchpoint.” You can test across front-end, back-end, mobile apps, Xbox, and even a tvOS. They are the leading A/B testing tool in the world – 26 of the Fortune 100 test with Optimizely, and they have achieved this status by creating server-side and client-side testing unified on a single experimentation platform.
In Classic, draft and paused experiments were included in the snippet, creating extra load, but in X, draft and paused experiments are not included in the snippet making the load considerably smaller. By offering custom snippet creation, it allows teams to build snippet payloads that include only experiences that are relevant to the page a visitor is on. This requires adding more snippets but keeps experiments running leaner and faster. X now includes individual CSS and JS files (Yay!). Through the new pages process, X is able to offer always-on visitor tracking, so no more wishing you had added a pageview or click goal. If you have it set up, you can add it to an experiment at any time and the data will backfill. Through pages there is also page-level conditional activation for Single Page Applications (SPAs). Optimizely uses a sequential testing statistical model (using Bayesian theory).
Program Management, Optimizely’s newest product, is designed to help scale your experimentation cycle across your enterprise. The Program Management tool is exactly what it sounds like: it’s a place where you can link hypothesis, experiments, new ideas that have a rating system, development cycles, stats (with annotation options), and conversations all in one place. This keeps your team from duplicating information across platforms, documents, and teams.
Visual Website Optimizer (VWO)
The main difference between VWO and Optimizely isVWO is looking to optimize your optimization, and they do this in a few different ways. First, their preview is cookie-based rather than URL-based, making cross-site testing much easier. Testers previously had to append a query parameter to every page or set the experiment live to a cookied audience. They were one of the first to move forward with this method, but now it’s the industry standard. Their platform automatically takes screenshots of each of your variations on save, so you can use it as a quick preview to see how your changes are coming along. VWO uses the Bayesian statistical model and will load your snippet asynchronously so your site isn’t slowed. They achieve flicker-free testing by making your page hidden until it is loaded.
Convert’s About Us page says, “we’re always optimizing for a better product, a better planet, and a better way to work.” They rebranded in a radical way in the past couple of months and are taking a positive, caring, culturally-aware approach to show that they want what’s the best for your business. They are working toward gender equality in the workplace and being carbon neutral 100X over. This is a huge shift from the way they used to describe themselves, as “affordable enterprise testing that includes personalization.” For what it’s worth, I believe both descriptions still hold true. They are probably the least expensive of the 5 when you compare the number of visitors and domains allowed on one account (caveat: after you surpass about 300k visitors a month).
Convert’s customer service is consistently available through email, chat, and phone. They are quick to respond, and actually seem happy to help out! They have a small company feel, but still put out an enterprise-level product. They include all of their features in all plans except their most affordable, which only loses a few small features. You really need to be testing at least 300K visitors a month, across all clients and platforms, for their pricing structure to make sense.
Adobe Target is a pioneer in the space and they are all about channels, channels, and channels! We interact with brands, not just on one website, but through all of our channels. By having all your channels testable across all devices, Adobe enables you to target a user through an ad and then personalize everything else they see across your media. The name “Target” really speaks to the goal of their product. Adobe wants you to create and test personalized experiences served to highly targeted users or groups across all mediums. They want the whole conversion funnel to be a place you can create personalized experiences for your user.
The tool feels like it was designed more for marketers than developers, but the beauty of that is anyone can use it in a sophisticated manner. With this approach, there is no need for a marketer to create a ticket for the developers to build a test. Adobe is moving past analytics to what they call “digital intelligence” which is, “in today’s mobile, multi-device, and multi-channel world … the ability to transform digital data into real-time, actionable, customer-centric insights.” Another cool feature Adobe Target uses is real-time statistics so you can tweak your experiences across all channels whenever you see a change in data.
Google Optimize & Google Optimize 360
Google Optimize 360 offers enterprise-level website testing and personalization using your analytics audiences. No need to recreate those audiences within another platform, as Optimize 360 offers native integration with Google Analytics 360 (GA). With Optimize 360, you can also leverage the segments/audiences you have already built in GA to create personalized experiences across your site. This allows you to test into segments natively that would otherwise be 1) not be possible, or 2) would take a lot of setup time to initiate. Also, the data that you get back can be easily parsed in GA, offering you the power to dig into your data in a much deeper way than most other testing platforms. Google uses session-based targeting, meaning a user who ends a session may see a different experience the next time they arrive on your site. This being said, Optimize 360 combines a solid, basic, A/B-testing platform with seamless analysis within Google Analytics 360.
Choose the best tool for your needs
Optimizely has my vote for the best overall A/B platform as it integrates front-end, back-end, program management, analytics, and a bunch of other useful tools. However, one thing to take into account is their pricing structure. It creates a steep yearly buy-in wherein each tool adds another line to your bill, but if you need the power Optimizely offers, they are a great choice.
If back-end testing is not necessary, but you want a tool that includes a place to manage your experimentation, then VWO is the CRO platform for you. They have plans starting at 10K visitors per month, so VWO can help small businesses at a fair price, but they also offer plans that can support the largest enterprise customers. They make the job of testing easy and keep you organized.
Convert is still the underdog here, including all their tools in each plan without a pricey upgrade to their services. If you have multiple clients or your company has more than 300k visitors a month, then Convert creates a flexible product with a ton of tools available at the lowest cost per visitor at this threshold. Plus, they are trying to create a positive impact on the world around us which always gains respect from our team.
If you’re invested in Adobe Analytics Cloud, then having the ability to test across all your channels in one place makes a lot of sense. Adobe Target will make that process seamless and leverage the data in the most powerful way.
If you use the Google Suite of products like Audience, Attribution, and Analytics, then utilizing a simpler tool like Google Optimize to gain the functionality of Google Audiences will open testing avenues that a standalone tool cannot give you out of the box. Also, if you want to dig into your testing data in a native and straightforward fashion, Google Optimize 360 is one of the strongest on the market.
In the end, the marketplace for A/B testing is huge and one tool cannot meet every customer’s needs. Keep this in mind, and choose the platform that best suits your needs for each account. Good luck and happy testing!
Written by Jeremy Sell, Senior Experience Engineer