Once you become immersed in testing your website it’s not hard to get caught in a matrix of never ending variables and what if scenarios. While there are many advanced tools that can help companies evaluate the usability of a website -- a few of which we’ve already covered -- ultimately there are two predominant types of testing methods that can be used: Multivariate and A/B Split Testing.
Today we examine these two testing concepts.
Defining Your Site’s Goals
Before we can effectively investigate these methods, let’s first establish what our goal is. Typically, a company wants to convert users into customers, which of course means revenue. Unless a user can actually click and purchase a product directly from the website, the process of converting user to customer can take an indirect route. Maybe there are forms a user can submit that signal interest or perhaps it’s more about building customer and brand loyalty by getting users to share information via social media. Take a moment to spell out the process by which your company hopes to convert users into customers.
Home Page → Product X → Sign up for Online Demo → Thank you page
Home Page → Product X → Purchase Product → Registration form → Thank you Page
Choosing Your Testing Tool
A/B Split Testing
A/B testing (also referred to as bucket testing) is the process of testing different versions of a specific page with only one element different between the pages. In other words, it tests the effectiveness of one landing page over another. Normally the current landing page will be used as the control, and a second page with some changes to the original will be used as the experiment.
What used to be a rather tried and true tactic of the direct mail industry, has permeated its way into website usability testing. Often companies will employ this method by offering two or more different user interfaces randomly to their audience. Evaluating what interface drives users through a site’s desired path can help guide website navigation and IA decisions. Retailers like Amazon and publishers like the BBC have been known to use this method.
The process for implementing and analyzing A/B Split Testing is fairly straightforward.
Step 1: Identify the Page to Test
When choosing a page to test, it’s recommended that you choose a page with high traffic so you can gather data faster and make a conclusion. Once chosen, decide on one element that you would like to use as your testing element.
Step 2: Choose the Conversion Page
Decide on a desired goal that you want to track, whether it’s a contact form submission, a download, a purchase, a sign-up or a time-on-site goal, etc. When tracking a form submission, purchase or a sign-up, you will want to have a unique “thank you” URL that you can use as your completion page. This is the URL you will add in the goals.
Step 3: Set Up Tracking Scripts
In order to track your experiment properly you will need to add tracking scripts to your control, test, and goal pages. Two types of scripts are needed: control and tracking. The control script makes sure that the experiment variations are switched randomly and that all variations are displayed an equal number of times, while the tracking script ensures that visits to both the test page and the conversion page are tracked. Implementing tracking code will vary between software.
Step 4: Decide on A/B Distribution
Depending on the number of tests you are running, you need to decide on what percentage of your traffic will be displayed the control page and version A, version B, etc. If you’re testing 2 pages, splitting the traffic up with 50% for each page is the easiest way. It is recommended that you use only one variable (i.e. your normal page, plus a page with one thing changed).
Step 5: Analyze Results
Without analyzing the results of your test, your efforts don’t mean much. Devise a plan to implement changes to your website based upon your evaluation. Many software testing solutions provide reporting features that allow you to see which variation was more successful.
Multivariate testing is the process of testing more than one component of a website at the same time in a live environment. In essence, multivariate testing lets you take conduct several A/B tests on one page at the same time.
This graphic shows how each user is shown different elements on a page in multivariate testing. (Courtesy of ProBlogDesign.com)
While the only limitations for multivariate testing are the amount of time it will take to gain sufficient data to come to a reasonable conclusion, let it be known that the more components you add to a test, the longer and more data you will need to complete a test.