Google Website Optimizer

Google Website Optimizer Hint: Beware of Small Numbers, Don't Rush to Judgment

Whenever we’re about to launch a new experiment, I always give clients the same speech. It goes something like this:

“You probably won’t be able to resist logging into GWO and checking the results every day. But always bear in mind that experiments cannot yield reliable results until they have run long enough to gather sufficient data. Early results are often surprising — even alarming — but you must resist jumping to conclusions. Let the experiment run its course.”

Now I have a confession to make: I sometimes have a very hard timing taking my own advice. A recent example comes to mind.

We were running an A/B test of a lead-generation form for a Financial Institution. Our revised (“B”) version quickly took the lead over the original. Within a week or so, it was showing an improvement of about 200%! We were astonished, and of course we started congratulating ourselves on how clever we were.

Luckily, we didn’t share our internal boasting with the client. Because over the next few weeks, we watched in dismay as our lead fell to 150%… to 100%… to 50%… and eventually settled at around 20%. Still respectable, but imagine how bad we’d have looked if we had told the clients we’d given them a 200% lift!

In other cases, we’ve seen GWO declare a winner, only to “undeclared” it the next day. You can’t rely entirely on what the tool is telling you.

So when can you confidently say the experiment has run its course? I’d suggest a good rule of thumb is, when all three of these criteria are met:

  • The experiment has run for at least two weeks (preferably longer)
  • Each version of your page has 100 or more conversions, and
  • Google declares a winner

Don’t trust what small numbers tell you. Be patient… difficult as it is.

Cardinal Path

Share
Published by
Cardinal Path

Recent Posts

Optimizing user experiences with Digital Experience Analytics (DXA) platforms

As consumers become increasingly digitally savvy, and more and more brand touchpoints take place online,…

2 weeks ago

Enabling Value-Based Bidding with Google Tightlock

Marketers are on a constant journey to optimize the efficiency of paid search advertising. In…

4 weeks ago

Resolving “Unassigned” Traffic in GA4

Unassigned traffic in Google Analytics 4 (GA4) can be frustrating for data analysts to deal…

1 month ago

This website uses cookies.