Google switched over to a new search indexing system this week. Aptly named Caffeine, the architecture is said to be faster than previous technology, and provides 50% fresher results.
Old Search vs. New Search
Regardless of how it might seem, Google doesn't search the entire Web when answering queries. Instead, the engine sifts through its own index, which was previously updated in batches. Unfortunately, this method prevented individual pages from being available until their batch was completed, leading to nasty delays between when Google found or updated a page, and when was available.
Under Caffeine, which takes up somewhere around 100 million GB of storage in one database, Google can crawl the Web in smaller portions and update its index on a continuous basis. The framework enables Google search to analyze hundreds of thousands of pages each second, and adds new information to the index at the reported rate of hundreds of thousands of gigabytes per day.
"With Caffeine, we analyze the web in small portions and update our search index on a continuous basis, globally," said Google Software Engineer, Carrie Grimes. As we find new pages, or new information on existing pages, we can add these straight to the index. That means you can find fresher information than ever before — no matter when or where it was published."
"We've built Caffeine with the future in mind," continued Grimes. "Not only is it fresher, it's a robust foundation that makes it possible for us to build an even faster and comprehensive search engine that scales with the growth of information online."
What's Going to Happen to Bing?
Speaking of the future, a fresh indexing system sounds like bad news for rivals like Microsoft and Yahoo. Martin McNulty of Forward3D (formerly Trafficbroker), for example, said the new framework threatens to put Bing "in the shade."
However, Microsoft has made a handful of fairly significant moves in the last few months (see: the integration of Foursquare; a new design; their partnership with Yahoo) and their numbers aren't looking half bad. According to Hitwise, the number of searches on Bing related to automotive, health, shopping, and travel rose 95%, 105%, 100%, and 71%, respectively, compared to the same month a year ago.
Moreover, Bing Webmaster Tools are on the way. Built to run on Silverlight 4.0, the tools aim to help webmasters better analyze important trends through connected visual data about their websites. The "no-cost tool set" features an Index Explorer, which enables users to actually look at crawling and index data, as well as the option to submit both 'priority' URLs and those to be blocked from showing up on SERPs.
And so while Google may be tuned in on the growth front, Microsoft is staying busy with what sounds like could be a significant move in the search marketing world. Looks like these two giants could stand to take notes from each other.