cancel
Showing results for 
Search instead for 
Did you mean: 

Blocking bots and other dubious stuff

megapixel
Grafter
Posts: 27
Thanks: 1
Registered: ‎30-07-2007

Blocking bots and other dubious stuff

The access log for my website shows that it is being visited by lots of search 'bots'. It didn't use to happen, but I've assumed that the activity is probably inevitable and nothing to worry about, which I didn't until now. Now it would appear that other websites of a more dubious nature are also being bounced or re-directed off my site. Perhaps this too is inevitable and nothing to worry about, but I was wondering if there was any way of better protecting my website, and blocking or filtering out these 'visits'? Thanks in advance for any help.
2 REPLIES 2
Peter_Vaughan
Grafter
Posts: 14,469
Registered: ‎30-07-2007

Re: Blocking bots and other dubious stuff

You can only block search bots that conform to the rules e.g. google, yahoo! MSN etc. The majority of 'unwanted' bots won't so there is no way to block them reliably.
You need to create a robots.txt file which the search bots will read to get info on what they are allowed to search through and add to the search engines.
http://www.robotstxt.org/orig.html
And to check what he robots.txt file does use http://www.sxw.org.uk/computing/robots/check.html
megapixel
Grafter
Posts: 27
Thanks: 1
Registered: ‎30-07-2007

Re: Blocking bots and other dubious stuff

Thank you Peter. I'll follow up those links.