Damn search engines · Jun 5, 11:32 by Shannon Wynter

Someone please, feel free to tell me if I’m doing something wrong on my site that leads some engines to believe that they need to re-visit my site so damn frequently.

Seriously, in 5 days, just how many times do you need to index my site?

Here’s the run down.

In less then 5 days I have been hit by the following engines

By far the best behaved bot in that list is Google, even Yahoo is better… but why does BaiDuSpider (1) think that I need 460 hits in 5 days, and (2) use over 3 times the bandwidth of MSNBot (which uses almost twice as much as Yahoo to do a very similar job)

Is there some flag I should be setting to tell them to shove off? the content on this site doesn’t change that often, it’s not twitter!

I’ve decided to drop BaiDuSpider at the firewall – I justify this in 2 ways.

1. It appears to be ignoring my robots.txt
2. I don’t cater to the same audience they do.

The list of addresses I’m blocking:
119.63.193.129
119.63.193.130
119.63.193.231
119.63.193.33
119.63.193.41
119.63.193.55
119.63.193.56
119.63.193.62
119.63.193.63
119.63.193.70
119.63.194.87
119.63.194.91
122.152.128.13
122.152.128.21
122.152.128.48
122.152.128.49
122.152.128.70
122.152.129.53
122.152.129.54
122.152.140.146
123.125.64.38
123.125.64.49
123.125.64.51
123.125.66.100
123.125.66.101
123.125.66.102
123.125.66.103
123.125.66.104
123.125.66.105
123.125.66.106
123.125.66.107
123.125.66.108
123.125.66.109
123.125.66.110
123.125.66.111
123.125.66.112
123.125.66.113
123.125.66.114
123.125.66.115
123.125.66.116
123.125.66.117
123.125.66.118
123.125.66.57
123.125.66.58
123.125.66.59
123.125.66.60
123.125.66.61
123.125.66.62
123.125.66.63
123.125.66.64
123.125.66.65
123.125.66.67
123.125.66.69
123.125.66.70
123.125.66.71
123.125.66.72
123.125.66.73
123.125.66.74
123.125.66.75
123.125.66.76
123.125.66.79
123.125.66.80
123.125.66.81
123.125.66.82
123.125.66.83
123.125.66.85
123.125.66.86
123.125.66.87
123.125.66.88
123.125.66.89
123.125.66.90
123.125.66.92
123.125.66.93
123.125.66.95
123.125.66.96
123.125.66.97
123.125.66.98
123.125.66.99
220.181.32.19
220.181.32.22
220.181.32.64
220.181.32.68
220.181.32.77
220.181.50.219
220.181.7.103
220.181.7.29
220.181.7.48
220.181.7.97
61.135.163.102
61.135.168.15
61.135.168.172
61.135.168.39
61.135.168.49
61.135.168.54
61.135.168.82
92.48.126.214

Comments