2014-24-JULY-SPIDERBOT.jpgContent producers spend a lot of time worrying about Google's search algorithms. But maybe it's time to think less about how frequently Google crawls your site -- and more about the potential damage from evil Googlebot imposters, who assume Googlebot’s identity to gain privileged access to websites and online information.

According to new research released today by Incapsula, a web security firm, millions of these “evil twins” are used for distributed denial-of-service (DDoS) attacks, hacking, spam, content theft and other shady activities on a daily basis.

Marc Gaffan, Incapsula’s co-founder and Chief Business Officer, shared a disturbing statistic. "For every 25 Googlebots that visit your site, you will also be visited by a fake Googlebot," he said.

Why worry? Because more than 23 percent of these fake Googlebots are designed to wreak havoc on your website.

Bite Me!

In case you've been worrying more about content than creepy, crawly things, here's a primer. A web crawler — or "spider" — is an Internet bot that systematically crawls the World Wide Web, typically for the purpose of Web indexing.

Among the innumerable creatures roaming the web, Incapsula contends, "few are as intriguing as Googlebot – a web crawler that facilitates knowledge exchange between billions of humans, influencing our perceptions, preferences and imaginations in more ways than we can even comprehend."

Googlebots crawl the web to discover new and updated pages to be added to that ever so important Google index.

Incapsula observed more than 400 million search engine visits to 10,000 sites, resulting in more than 2.19 billion page crawls over a 30-day period. It found:

  • Googlebot’s average visit rate per website is 187 visits per day
  • Google’s average crawl rate is four pages per visit
  • Google doesn’t crawl popular websites any more than smaller websites
  • On average, a website will be visited by 187 Googlebots a day
  • Content-heavy and frequently updated websites, including big forums, news sites and high-scale e-shops with a wide array of frequently updated products, are more thoroughly crawled
  • Googlebots crawl more pages than all other search engines combined

2014-24-July-share-of-page-crawls.jpg