Never make management decisions for a website based on opinions. There is often a Jekyll and Hyde difference between what people say and what they do.
SIMS is a hugely popular simulation game. They wanted to improve sign up for optional registration for those who had purchased the game. At one stage they tested two sign up pages, one of which was 40 percent more successful. Now, 40 percent is a huge, massive difference.
I have shown hundreds of web professionals the two pages and asked them to choose what they thought was the most effective page. About 80 percent of people go for version B. This maps well to the voting that occurred on the site Which Test Won? (Well worth visiting if you're interested in this sort of stuff.) The people who go to Which Test Won? tend to be in the business of managing websites.
The problem is that it was Version A that was actually 40 percent more successful. And this is far from a unique occurrence. To our eyes, Version B looks better, but the cold hard data shows that Version A massively outperforms it.
Our world is changing radically. The age of intuition, gut instinct, opinion, and natural creativity is on the wane. These are romantic notions and we love to believe in them, but when the data comes in they are being increasingly proven not just to be wrong, but horribly wrong.
The worst way to design a website is to get five smart people in a room drinking lattes and posting post-it notes. The longer you leave them the worse the website becomes. The next worst way is to get 10 customers in a room drinking lattes and giving their opinions on the new design. That model is really, truly broken.
The relentless march of cold, hard data can be seen everywhere. There is huge resistance to this evidence, but resistance is futile. Take poker. "Phil Hellmuth Jr. may be the world's most decorated gambler," TIME wrote in July 2010. "But last year it all began to fall apart. Hellmuth, 45, lost money and failed to make the final table of even one tournament for the first time in more than a decade.
"Was it his cards? No, Hellmuth says, pacing the floor of his suite at New York City's Plaza Hotel. He blames the new breed of math nerd, those guys using a mountain of sortable data from the millions of hands played online to dominate the game. "The reason I won 11 bracelets is my ability to read opponents," he explains. "These new guys are focused on the math. And they are changing everything."
The world of human behavior is much more observable and measurable today. Computers can crunch data like never before and we spend more and more time in the online world, a perfect place to be measured and observed.
I voted for Version B, the page that was much less successful. It's scary. How can our opinions be so wrong so much of the time? But that's not really the question. Rather, the question is: How can we use the data to make better decisions?