Showing results for 
Search instead for 
Did you mean: 

Search Bots


Search Bots

Having a look at my domain stats...

I could see there were several search bots that were scanning my pages.

Is there anyway to stop them from doing this?

Community Veteran
Posts: 6,983
Thanks: 8
Registered: 10-04-2007

Search Bots

Most people want it that way so that there site gets noticed by the search engines :?

There is a HTML code though that you can insert into the header to stop the bots looking beyond that page.
I am looking through the massive book I have on HTML trying to find it at the moment.
If I do come across it again I'll post back here but if anyone else knows it then feel free to post as well.

Ok found it
Try <META NAME="ROBOTS" CONTENT="nofollow,noindex"> that should stop the robots doing anything with it. Cool
Community Veteran
Posts: 4,729
Registered: 04-04-2007

Search Bots

Add the follwoing file in your htdocs directory, or document root.

User-agent: *

Disallow: /

This will tell all search bots or robots that you do not wish to have your site scanned.