The Magic Behind Amazon's 2.7 Billion Dollar Question

By Jared M. Spool Mar 18, 2009

Since its release in 2007, the last volume of the Harry Potter series, Harry Potter and the Deathly Hallows, has garnered 3,286 reviews from Amazon.com customers. While response has been overwhelmingly positive for the book, several hundred Amazon customers rated the book as mediocre or worse.

Because of a very subtle yet clever feature, Amazon makes the best of both the positive and negative reviews easy to find. And that feature, based on our calculations, is responsible for more than $2,700,000,000 of new revenue for Amazon every year. Not bad for what is essentially a simple question: "Was this review helpful to you?"

The Problem with Chronology

Amazon had reviews from the very first day. It's always been a feature that customers love. (Many non-customers talk about how they check out the reviews on Amazon first, then buy the product someplace else.)

Initially, the review system was purely chronological. The designers didn't account for users entering hundreds or thousands of reviews.

Interestingly, only a fringe portion of the audience writes reviews. For example, while Harry Potter and the Deathly Hallows has more than 3,000 reviews, our calculations indicate Amazon sold more than 4,000,000 copies of the book. That's 0.075% or only one out of every 1,300 purchasers that took the time to write a review.

For small numbers, chronology works just fine. However, it quickly becomes unmanageable. (For example, anyone who discovers an established blog may feel they've come in at the middle of a conversation, since only the most recent topics are presented first. It seems as if the writer assumed the readers had read everything from the beginning.)

The problem came with the eleventh review. Since the product page only showed ten on the first page, the eleventh pushed the earliest review onto a different page. This worked fine as long as every new review was better than the existing ones.

But that wasn't happening. Newer reviews often had a what-he-said vibe to them, echoing the sentiments of the well-written reviews, while, at the same time pushing them out of the reader's view.

Adding Editorial Perspective

Amazon needed a way to make the best reviews bubble to the top. The obvious approach would be an editorial team to select the best reviews for highlighting.

However, the cost would be exorbitant. While Amazon already has a team that checks reviews to eliminate spam and off-topic comments (you never see unwanted promotions in their reviews), it's a lot more work to rate each review and select the best.

Doing so would also go against Amazon's philosophy of letting the market decide what's good and bad, which makes Amazon a trusted agent. If customers perceived Amazon was selecting reviews for items that provided better profits or manufacturer kick-backs, it could damage the credibility the site.

So the team had to come up with an approach that scaled well but didn't hurt credibility. In true Amazon tradition, why not ask the customers what they thought?

The Elegance of the Question

Even though the question is one of Amazon's most important features, it's impressive how understated its introduction was. One day, for many of Amazon's users (but not all), it just showed up. It was one of many changes occurring on the site that month. Most users didn't notice it for weeks and few understood how it was going to change the value of the reviews. (It's not even clear that the team, when they launched it, truly understood what was about to happen.)