What is A/B Testing?

An A/B test, also known as a split test, is a method of testing variations by showing them to an audience simultaneously, evaluating their reaction, and calculating the results to choose the best variation (A or B).

This method is used to compare ads, website page variations, or individual elements like headlines, backgrounds, and design styles. This provides a quantitative assessment rather than a subjective one based on taste. Here is how it looks for a website: 10,000 people click on an ad. A random half of the audience is shown one version of the site, while the other 5,000 visitors see the second version.

Example of Evaluating Results

Let's compare two duplicate versions of a landing page: A and B.

Variation A yielded a 5% conversion rate from visitor to lead. The cost per lead (CPL) is 1,000 rubles.
Variation B yielded a 1% conversion rate. The cost per lead is 5,000 rubles.

In this case, Variation A is better, and we should choose and adopt it. And even if we, our friends, colleagues, family, loved ones, or even certain clients don't like this version based on personal taste or other reasons, the statistics say otherwise. This approach allows you not to rely on taste or judge by your own preferences, but to choose a truly viable, effective, and economically justified option—the one most likely to lead to success. Stop arguing about the button color, background, or headline phrasing—just test it!

When fishing, the bait must taste good to the fish, not the fisherman!

Rules of A/B Testing

It is essential to ensure a statistically significant number of repetitions (traffic/views).
It is important to get a sufficient gap in the results between the compared variations to confidently choose the winner.
The test must be parallel and run simultaneously so that the results are not affected by seasonality or the time of display.
Traffic must be uniform across all variations. It would be counterproductive if one version received search traffic while another got social media traffic.
The core offering and topic of the tested variations must be identical. If you try to compare two variations for different services or products, you won't understand why one performed better—whether the product itself is in higher demand or if the design made the difference.

To understand how many repetitions are needed and how big the gap in results should be to declare a winner, there are specific formulas and calculators available online. Look them up. However, there is a simpler method: run an A/A test before the A/B test. Let the variation compete against itself. Once the results level out (which won't happen immediately), use that traffic volume as your benchmark.

The Idea Behind A/B Testing

The core idea of this method relies on the fact that the viewer sees the variation for the first time and doesn't know about the other versions. Viewers do not communicate, consult, or share opinions with one another.

This is exactly the scenario on promotional marketing websites, where 98% of visitors are new. Very few return a second time. They either get interested and contact you, or they close the site, forget about it, and leave. We cannot afford to hand out different versions of printed materials to attendees sitting next to each other in a conference room. They might notice the different branding options and wonder "why is there such a mishmash," as one printing client humorously put it. In the online world, the situation is completely different.

When A/B Testing is Not Needed

A split test is an effective tool for choosing the best variation. But remember that the advertising budget must also be increased proportionally.

If you decide to test a business hypothesis, build a landing page, set up ads, and want to run a trial budget of 40,000 rubles, you will need to allocate twice as much budget to test two page variations. For 3 or 4 variations in an A/B/C or A/B/C/D test, the budget increases accordingly. Additionally, preparing multiple versions of a website requires much more work than building just one.

Read new articles:

All Topics