IPv4 Fremnet Logo

Good code yeh?

Damn search engines · Jun 5, 11:32 by Shannon Wynter

Someone please, feel free to tell me if I’m doing something wrong on my site that leads some engines to believe that they need to re-visit my site so damn frequently.

Seriously, in 5 days, just how many times do you need to index my site?

Here’s the run down.

In less then 5 days I have been hit by the following engines

Engine Hits Robots Bandwidth
BaiDuSpider 460 6 7.52mb
MSNBot 323 95 2.26mb
Yahoo Slurp 314 28 1.49 MB
Googlebot 50 6 285.04 KB

By far the best behaved bot in that list is Google, even Yahoo is better… but why does BaiDuSpider (1) think that I need 460 hits in 5 days, and (2) use over 3 times the bandwidth of MSNBot (which uses almost twice as much as Yahoo to do a very similar job)

Is there some flag I should be setting to tell them to shove off? the content on this site doesn’t change that often, it’s not twitter!

I’ve decided to drop BaiDuSpider at the firewall – I justify this in 2 ways.

  1. It appears to be ignoring my robots.txt
  2. I don’t cater to the same audience they do.

The list of addresses I’m blocking:


Spam no more - rel=nofollow is active here, spamming my comments will not help your page rank.

  Textile help
---== Copyright Shannon Wynter - All rights reserved - All wrongs avenged ==---