The Beginner’s Guide to A/B Testing in VWO

Here at Somebody Digital, we use a wide range of tactics when doing digital marketing for our clients.

We use inbound and onsite SEO to ensure our clients are ranking on Google, structure ad campaigns through PPC, and use a mix of organic and paid social to bolster their presence on all social media platforms.

Another important area we focus on is Conversion Rate Optimisation (CRO), through this we find ways to continue to optimise websites to increase conversions. We do this by using multiple tools for analysis to spot areas for improvement and based on this we implement A/B testing to change key areas on the site, looking at both functionality and design.

For instance, through our analysis we may see that the converting CTA is not getting enough visual attention compared to other components on the site. Something that we may do here is change the colour to show more contrast, for example changing it to have a blue background instead of a grey one:

We would do this by running an A/B test to see the impact that this change had on conversions. Through this A/B test, we’d show 50% of visitors the old CTA and 50% the new CTA. We would then track performance through click tracking, visits to converting pages and conversion rate to see the impact of the new design.

It’s a neat process, and one that can boost your conversion rates by a serious margin.

So in today’s article, we are going to show you how it all works, by walking you through the setup of a simple A/B test in one of the tools we use VWO.

1. Create an A/B Test Campaign

So let’s start with the basics. Upon logging into VWO, you’ll see one of two things.

If you’ve built some VWO tests before (or have worked with an agency who has), you’ll see a list of them under ‘Testing’ and then ‘A/B tests’ in the left hand column. As is the case in the below example, since we’ve already built some tests for this client.

On the other hand, if this is your first A/B test, you’ll see this screen instead:

Which merely tells you that no tests exist yet, and gives you more information about how to build them.

So to Create a new test, click the blue button in the upper right-hand corner labelled ‘Create’ to move to the next part of the form.

This will then bring you to the following screen:

As you can see, said screen provides options for the test name, the URL(s) it runs on, the hypothesis and conditions that need to be met for a user to be entered into said test.

Name your A/B test

We’ll start with the test name. Click the pencil icon next ‘New Campaign’.

By doing so, you’ll have the ability to type in the name for the test, you can call it whatever you like here…

Set the URL(s) you want the A/B test to run on

Then scroll down to the URL box. Here you can type the URL of the page you wish for the test to run on.

This will only be the case if you want the test to only run on one page. However, this won’t be the case in this tutorial.

Instead, we want the test to run site wide. So click the ‘Advanced Settings’ box to open up some more URL options.

This will then give you the ability to run the test on all pages whose URL starts with a certain term, ends with a certain term, contains a certain term, etc. Set the dropdown so it says ‘contains’, and then type the domain name for your site into the box.

And then add the full URL of the homepage into the preview URL box.

This will make sure the test runs site wide, as well as make sure VWO’s preview button opens the site’s home page when clicked.

Set a Hypothesis

Onto the hypothesis now. This is a new feature VWO added in a recent update, it lets you enter a hypothesis for each test to refer back to, as well as indicate the level of confidence you have in said hypothesis.

It’s a neat feature to fill out if you’re running a real test, and should be used in that situation.

However, we’re not running a real test here. We’re running a tutorial, and the hypothesis doesn’t really add anything in this case. So for the sake of this demo just type whatever you want into the boxes provided and set all the confidence scores to three.

That takes care of the main fields on this page.

But the page doesn’t end there. Nope, it also provides a few extra options at the bottom too.

2. How to set up the visual editor

These are found in the ‘More Options’ box, and basically provides you with the ability to choose the editor, choose the display size of the visual editor and choose who gets entered into the test respectively.

So start out by checking the ‘Open in code only Mode’ box. This is super important, since it lets you write your own code rather than get stuck wasting time in the visual editor

And it’s vital for pretty much any and all tests that run on sites with dynamic content, like social networks, eCommerce sites or wikis.

That’s because VWO’s visual editor is designed to replace static text content with other static text content, and is unsuited to making changes to parts of a site which are dynamically updated, whether that’s through a server side process or a client side one.

Hence if you use the visual editor to edit one of these sites or sections, your beautifully crafted A/B test variation will run into serious trouble the minute someone logs in, edits a page or posts a message to their user profile.

Or worse still, you’ll break said features altogether, making your site unusable in the process.

Trust us on that one. We made that mistake as well.

So unless you’re editing static content, or only making CSS changes, always check this box when creating a test.

Similarly, always ignore the next box. It’s not going to affect anything if you’ve checked the code only one, but it’s redundant given it only affects the visual editor in the first place.

Set user targeting

Finally, click the option to ‘Enable campaign for a specific user group’.

This adds a whole host of options for user targeting, letting you run the campaign for users using a specific browser, device, IP address or a whole host of other factors. For this demo campaign however, we’ll only be running the test for desktop and tablet users.

So click ‘Custom’ on the left-hand side, then choose to run the test when the user’s Device Type is equal to either Desktop or Tablet:

Once you’ve done that, click ‘Next’ to move onto the editor.

3. How to add custom code

This is where you have to add the necessary JavaScript and CSS code for your test. This requires some knowledge of front end web development, especially if you want to do anything fancy like add a dynamic filter system or load new content asynchronously via AJAX.

For now though, we’ll keep it simple. Type the following code in the JavaScript tab:

var test = document.createElement(‘aside’);
test.classList.add(‘fixed-aside’);
test.innerHTML = ‘This is a test’;
document.body.appendChild(test);

And the code below in the CSS one:

fixed-aside {
background: #fff;
border: 1px solid #222;
bottom: 20px;
padding: 15px;
position: fixed;
right: 20px;
}

This will add a static box to your website in the bottom right corner with the sentence ‘This is a test’ written in it. Said box will then scroll along with the page window, remaining 20px from the bottom right of the screen at all times:

Once you’ve added the code, click ‘Next’ again to move on. This will then bring you to the goals page.

4. How to set up your Goals/Metrics

Here you put the conversion in Conversion Rate Optimisation, by setting up the goals (or conversions) you want to track for each test.

For instance, if your test involved changing the contact form, you’d want to track clicks on the submit button, clicks on each form field and visits to the thank you page URL. That’d give you a rough idea of how many people might be filling out your form.

But we’re not testing a form in this demo, we’re testing a random box in the bottom right corner of the screen.

So here we will instead track visits to other pages, clicks on the new element and clicks on any and all links on the website.

Track visits to all pages

Let’s start with the former. Edit the existing goal (VWO always provides one of these by default) to set it to track visits to all pages on the website.

You can do this by changing both the title and the URL set up in the URL matches box. For the former, click the pencil icon next to the title to add a box you can type in, then put in something a bit more meaningful, like: ‘website visits’.

Then click the URL matches dropdown so it instead says URL contains, and enter your site domain in the box.

That’s it. You’ve now added a goal which tracks visits to other pages.

Next, add a new goal.

Track clicks on a specific element on the website

This time, don’t set it to track visits to a URL. Instead, set it to track clicks on an element, and set the element you want to track clicks on to ‘.fixed-aside’.

That way, it’ll track how many times your new sidebar element was clicked on it. It’s basically a lot like using document.querySelector for any JavaScript pros out there.

Finally, create a goal to track all clicks on links. You can do this by copying the format of the previous one, except with ‘.fixed-aside’ switched out for just ‘a’. Do this, and you’ll have a goal which will track clicks on any link on the website.

Here’s a the screenshot for reference, in case you need any more help setting up the tests:

So if you’re happy your test goals are set up the same way, do like on every other screen, and click the ‘Next’ button to move on.

5. How to distribute your traffic

This will then bring you to the final page. Here you can edit the test name again, if you wanted to. As well as make a few other useful changes to the test setup, like changing what percentage of users see the variation and what percentage see the control.

We’ll leave this at 50% for the control and 50% for the variation for the time being, you can also tick the box ‘Distribute Equally’ here, but you may want to mess around with it at a later time nonetheless. Like say, whenever you feel like testing two or more variations instead of having a control.

You can also set up integrations between VWO and various third party tools here too. For instance, you can send the data to Google Analytics if your site uses that for tracking. Same with Google Tag Manager.

But that’s a story for another article. We’ll explain how to properly integrate Google Analytics and VWO in a future tutorial instead.

For the time being though, just leave those boxes unchecked.

Do the same thing for the ones in the ‘Advanced Options’ settings too. These are useful in certain situations, but they’re again something for another tutorial, and not at all relevant to this demo campaign. Don’t change any of them from the default setup.

Click ‘Next’ once more to reach the final screen.

6. Finalising your A/B test Campaign

This is it. This is where you can choose whether to start the test immediately, or preview and start it later.

You’re gonna want to do the latter. Starting the test right now wouldn’t be a good given our lack of browser testing, and a very good way to potentially break your website for anything more complex than an extra element or console statement.

Don’t push it live right away, set it to preview now and start later. Then test the site via the preview link and do a full QA of the A/B test. If you entered everything correctly, you should see the following upon doing so:

Congratulations, you’ve now created your first ever A/B test in VWO. Hopefully you found it an interesting learning experience, and something that’ll lead to many CRO campaigns to come.

Thanks for reading!