A/B and Multivariate Test tips

It goes without saying that when you want to make a change to your website, that it is a great idea to introduce the new changes via an A/B or Multivariate test. The tests will let you know quantitatively whether or not the changes that were made helped or hindered your web visitors and your business.

When design your test there are a few things to consider. Should one do an A/B test or a multivariate test?
How many combinations should the multivariate test include?

As a rule of thumb A/B/n test are good for testing one layout versus another (even though it is best to only vary one variable in a true A/B/n test) or when you don't have many successful conversions that will occur in a one month time period.

When doing a first test it is often best to do an A/B test to get a quick win and convince others in your firm of the value in testing changes to your website.

If your firm has already seen the light then you may want to move to multivariate testing. One thing to be careful of when running a multivariate tests is the number of combinations that are created when you design your test. You want to make sure that your site will be able to produce 100 successful conversions per combination in a month. So that means that if your site currently gets 700 people a month filling out your lead form and you want to test some new options that you shouldn't have more than 7 combinations in your test. Having 100 conversions per combination helps with the statistical significance of the test and having it complete in a month enables you to implement changes without waiting forever for result. If you can get results in 3 weeks as a results of a large number of success that is great, but when design a test a good rule of thumb is 100 successful conversion per combination in a month.

Too many people create tests that have too many combinations as a result of testing too many small changes. Focus on bigger items and fewer of them.