Getting your site crawled by the major search engines is central component to search engine optimization – without it, there’s no way your site will be indexed and ranked and no one will find you.
Search engine spiders, though, are known to go a little crazy from time to time…Bing’s reputation in this area especially caught my interest.
It appears from Rusty Brick’s analysis that all search engine spiders do this but the majority of complaints come from Bing’s MSNBot. So what’s the problem if a search engine spider excessively crawls my site?
I want that correct?
You absolutely want search engines to crawl your site but if it’s excessive, it can affect your host server and resources. Too much demand on your server can lead to your site crashing – just think about news reports you hear about websites crashing because too many people tried to visit them at one time.
Well search engine spiders are site visitors as well, technically speaking.
One webmaster on this discussion thread complains of the MSNBot spider specifically. Starting around September 17th, his site went from 300-600 page views per day from the MSNBot to 2500-3000. Another webmaster says that in the first half of October, the MSNBot catalogued over 17,000 pages on a 1,400 page site!
Apparently Bing’s MSNBot is the worst offender – over 80% of complaints of excessive site crawling involve them.
So if you seem to be having problems with your server, this could be the source.