User Testing of Websites: Leave Your Agenda at Home

I recently reviewed some videos of user testing sessions. I was shocked to see the moderator repeatedly break one of the most fundamental (and common-sense) rules of website usability testing: Don’t lead subjects through the tasks.

In this case, the moderator consistently gave the subjects hints. For example, he’d ask, “How would you sort products by price?” A sensible question, perhaps. Except that prominent on the page was a menu labeled “Sort by…” And one of the menu options was “Price”.

How could anyone not complete this task?

In assigning tasks, the moderator should have avoided the wording used in the navigation and menus. For example, he could have asked, “Assume you want the cheapest one possible. Can you get this page to rank items according to how much they cost?”

Leading your subjects is bad. Equally bad — and less often discussed — is the reverse: making tasks overly difficult by phrasing them in a confusing manner.

I once knew a web usability specialist who seemed to believe his job was to create as thick a report as possible. The more “errors” he uncovered, the more satisfied he was… even if the “errors” were only in his own mind!

He consistently phrased tasks in such a confusing manner, that subjects had no idea what they were supposed to do. Then when they failed to complete the task within a reasonable timeframe, he’d mark it down as a failing of the website. Clients were left scrambling to correct “problems” that simply didn’t exist.

When conducting user testing sessions, remember it’s not your job to prove the website is easy to use. Nor is it your job to prove it’s a usability disaster.

Assign realistic tasks in a clear, yet non-leading manner. And leave your agenda at home.