Woman in Yellow Coat Standing Holding Book during Daytime
If you do your homework, you'll feel smarter about A/B testing. PHOTO: Pexels

Let’s face it. Everyone loves to be the smart one in a given instance.

But A/B testing can sometimes leave people scratching their heads, feeling anything but smart — especially about planning resources efficiently.

So what can you do? Start with your homework. Understand what is being tested and account for how the teams are potentially impacted.

A/B Testing Basics

A/B or split testing involves comparing two versions of a web page to see which one performs better.   

The difference between the pages is a web element that is of interest to a marketer — anything from the shape of a button to text or an image.  

  • A is usually a control for the test, the original layout or media element
  • B is the layout that contains the changed element

The test works by showing the two variants to similar visitors at the same time. The one that gives a better conversion rate is the preferred choice.

Getting Started With A/B Testing

Marketers should consider the following ideas before launching a test session.

Make Sure There Is Enough Traffic to Achieve the Required Sample Size

An A/B test requires a certain level of site traffic or app use to reach a needed sample size for a test.  But some organizations launch tests without checking whether the traffic levels will provide significant results within the testing time. 

Determine a sample size estimate for your test.  There are a number of ways to calculate this scientifically. Online sample size calculators, such as those from Wingify and Optimizely, simplify the process.   

The estimate can then be compared to the current website traffic or app usage.  If the difference between the estimate and the media is large, more activity — more marketing for a site or enticing customers to use an app — is required to achieve the target sample size.

Moreover, you can use real-time reports in an analytics solution to confirm how frequent traffic is contributing to your A/B test pages.  The activity can confirm the   duration planned for a test and alert if the decreases in activity will hinder attracting the target observations.   

You can also use analytics reports to monitor if your audience is from the right persona — demographic, location and interest can help ensure that you are not reaching an alternative population to who the changes were meant to benefit.

Group Media Elements Into Ideas You Want to Test

Testing every single element undermines what an A/B test is meant to accomplish.  Instead, test one idea against the current production design, with each idea featuring a group of element changes meant to address a specific design concern.  

For example, a test can be of an app or mobile page with combined element changes for the most visible section of a screen, as opposed to just changing a button.  

Review What Influences Teams (Make Sure the Stock Room Is Ready, Too)

At first blush an A/B test does not sound like a high effort activity, but like any digital changes, it can trigger a lot of technical implementation “behind the scenes” of a live site or app. 

With each task, marketers should map out roles and ask, “Who else is impacted with this test?”   

If you are testing pages that involve products on an e-commerce site, make sure that your backend operations to ship that product are ready.  The last thing you want to do is run an A/B test while creating new headaches in order fulfillment.

Overall, you should understand what has to happen to manage the tasks well.

Testing has become an increasing part in verifying if a webpage or app change will improve a customer experience. With sales at stake, organizations should leverage testing as a way to bring teams together to improve the delivery of those improvements, too.