The Benefits of Multivariate Testing Over Traditional A/B Split Testing
We all know it’s easier to convert more of the visitors you already have on your website than it is to go out and get more traffic – it’s what we call Conversion Marketing. If you’re already running A/B split tests on your website’s pages, paid ad campaigns, and landing pages, congratulations – you’re doing the right things right. If your web marketing plan calls for, and implements, multivariate testing then you’ve moved on to the next level and are very likely enjoying the benefits of multiple variable conversion testing.
Multivariate testing provides us a way to statistically evaluate the immediate effect that certain variables (small changes to your website) have on any measurable goals of your website, as well as the effect that these variables exert upon one another (interactions).
To illustrate the power of multivariate testing, here’s an example:
As many of us know, “trust logos” can have a significant impact on the performance of a website. Let’s assume, for example, that in an A/B split test we tested the condition of A) having a particular logo against B) not having that logo using an appropriate sample size. Let us further assume that we found a statistically significant higher overall conversion rate for scenario A.
Another variable that we might consider is background color. For example, we can compare a blue background with a navy background, and find that the navy background color provides us yet another incremental gain in our overall conversion rate.
Logically, we would want to implement the two conditions that performed the best – having the logo on and using a navy background. When we do, however, we might notice that our overall conversion rate plummeted! What happened?
Well, the two individual winning conditions, when used together, did NOT constitute a winning combination. For whatever reason (and that could be something as seemingly small as even a slight clashing of colors) using the logo with a navy background provided unfavorable results even though using the logo or navy background alone gave us the boost we were looking for.
Design of Experiments (DOE – also known as fractional factorial designs) can help us to identify the main effects and the interactions of an array of variables on a control signal (or in this case, conversions). Let’s continue with the example we just ran, and explore the various combinations of this two-level factorial design:
- Variable setting/combination 1: Logo ON, Background BLUE
- Variable setting 2: Logo OFF, Background BLUE
- Variable setting 3: Logo ON, Background NAVY
- Variable setting 4: Logo OFF, Background NAVY
By measuring the overall conversion rate (or average page views per visit, or virtually any metric you may be tracking), we can find which of these combinations works best together. Now, this particular method calls for 4 tests to evaluate just 2 variables, and you’re probably thinking ahead. Three variables would require 8 test runs (2^3) – and what if I wanted to test, say 7 variables? That’s 128 individual tests I’d have to run!
The good news is that tools such as Orthogonal Array Testing Strategy (OATS) and fractional factorial designs can help us set up our test runs in a way that reduces the number of runs while allowing us to obtain statistically significant estimates of all main and interaction effects. When implemented correctly, our test of three independent (two-state) variables using OATS would require not 8 but 6 test runs, and at the expense of confounding the main effects with our two-factor interactions, if we wanted to try 7 variables at once we could cut our test set from 128 to just 8 runs. It’s certainly possible to try more variables – just remember that the more variables you introduce the more difficult the tests will be to keep track of, the more your results will be diluted, and the longer you’ll have to wait to introduce any other changes to your site (while these tests are being run everything else on your website must remain constant).
Multivariate testing and DOE can be a powerful way to understand how changes you make to your pages affect the goals of your website, and a good website should always be testing submit pages, funnel pages, landing pages, and product pages to make sure you’re making the best of the traffic you already have.
