Does Data Speak Louder Than Opinions?

Do You See What I See?
As an analytics consultant, it’s important – strike that – mission critical, to make sure you understand how your customer uses data. Sometimes we have a tendency to assume our clients look at and interpret data the same way we do. Nothing could be further from the truth. All customers look at data differently, and uncover insights that we might consider surprising or unexpected. We owe it to our customers to ask the right questions and best understand not only what’s being measured and reported on, but how that data is being consumed and interpreted.

reCAPTCHA = evil
A recent experience highlighted the importance of understanding how a customer looks at data.

I received a request from a customer asking me to QA some Google Analytics tracking code that had been deployed to a page. In order to test this scenario, I needed to go to the page, fill out a form and submit. If all went well in my testing, I would see a hit sent to Google Analytics via my HTTP header monitoring tool.

Seems straightforward and easy enough right? Sure… except that I wasn’t able to submit the form. Huh??? Yeah you read that right – I wasn’t able to complete and submit the form. Not once, not twice, but 30 times I tried and failed! What in the darkest depths of Middle Earth (Yes, I loved The Hobbit, and watched it recently, as you might guess), would cause this issue, you ask? The culprit was that most dreaded of all online phenomena – the reCAPTCHA!

First, let me state my complete and utter disdain for reCAPTCHA. I hate it, and consider it amongst the darkest of evils on this planet. Why do I hate reCAPTCHA so much? That’s easy 🙂 I certainly understand the benefits gained by reCAPTCHA. It blocks spam form submissions and only allows legitimate human submissions. Yadda yadda yadda 🙂 I’ve heard it all. Still hate it! Some of those alphanumeric combinations are so out of this world that no human could ever read them.

reCAPTCHA is a marketer’s worst nightmare. Web forms are sort of like Cookie Monster “More cookies…more cookies!”. All they want is for users to fill them out. Cookie Monster is cute, and simply wants more cookies. Sure, he leaves a mess of crumbs behind, but I’ve never heard him complain about the quality of the cookies he consumes.

reCAPTCHA takes an innocent, unassuming, cute and cuddly puppet like Cookie Monster and turns him into…the Soup Nazi! For those of you who didn’t catch the reference, or were deprived of the wonders of Seinfeld, check out this link to learn more. The Soup Nazi makes you stand in line (quietly), and only gives you a very measured amount of soup. Don’t you dare look at him the wrong way, or “No Soup for you” will echo in your ear drums. The mere presence of the Soup Nazi strikes fear into the hearts of the most brave of people, and renders them the likeness of jello – jittery and paranoid. reCAPTCHA has the same impact on web forms by making conversion significantly more complicated, and frustrating the user beyond compare.

reCAPTCHA sits at the forefront of the classic battle between IT and Marketing. Both sides have valid arguments as to it’s usefulness, and as much as I dislike it, reCAPTCHA obviously does filter out spam. At what cost though? Are legitimate customers jumping ship out of sheer frustration? Let’s see what the data has to say…

Data’s Turn to Talk
When I was unable to complete the form, I started thinking how this must be affecting other users. Surely, this must be giving other users the same headache it gave me. Luckily we were tracking all sorts of details about the form, so the answer was in the data. Time to dive in 🙂

In this type of analysis, it would be good to look at:

• Percentage of users running into reCAPTCHA errors
• How many users are receiving multiple reCAPTCHA errors?

Here’s what I found: (all data referenced below is for a period of one month)

What stands out from looking at the data above?
1. There were more reCAPTCHA errors (270,969) than submissions (241,803)!!! That means the problem is very widespread, and means that a majority of people are running into this problem. In fact many are hitting multiple reCAPTCHA errors.
2. 76% of form submissions resulted in reCAPTCHA errors!!! This was nothing short of shocking for me. 76%??? That’s insanely high! I happily wrote an email to my customer, full of excitement at what I’d found. I received a prompt reply thanking me for the analysis. The marketing team was also flabbergasted by these results and understandably wanted the reCAPTCHA removed.

Wait…A Different Perspective?
A few days later I received a call from an IT manager in the same organization. First he asked to verify how I had come up with this data. Upon confirming its validity, he also thanked me and said this data was immensely useful for them. He then went on to point out that their reCAPTCHA error rates are inline with industry standards and he wanted to make this a monthly report that he could trend with the intent of making sure the rate isn’t going down. Not going down??? What? You see he was in IT, and his focus was to make sure the reCAPTCHA was doing what it’s designed to do. The harder it is to read those ridiculous patterns, the more effectively it must be working. I thought they would be motivated to remove the reCAPTCHA, and instead they want to make it harder!!!

Always understand how your client will use data. It’s of critical importance and helps you provide meaningful insight, and ensure that data is actually being used to impact the business.

Thoughts, comments? I’d love to hear your perspective!