Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Blocking bots and other dubious stuff
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Plusnet Community
- :
- Forum
- :
- Help with my Plusnet services
- :
- Broadband
- :
- Blocking bots and other dubious stuff
Blocking bots and other dubious stuff
04-03-2008 9:34 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
The access log for my website shows that it is being visited by lots of search 'bots'. It didn't use to happen, but I've assumed that the activity is probably inevitable and nothing to worry about, which I didn't until now. Now it would appear that other websites of a more dubious nature are also being bounced or re-directed off my site. Perhaps this too is inevitable and nothing to worry about, but I was wondering if there was any way of better protecting my website, and blocking or filtering out these 'visits'? Thanks in advance for any help.
Message 1 of 3
(923 Views)
2 REPLIES 2
Re: Blocking bots and other dubious stuff
04-03-2008 9:38 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
You can only block search bots that conform to the rules e.g. google, yahoo! MSN etc. The majority of 'unwanted' bots won't so there is no way to block them reliably.
You need to create a robots.txt file which the search bots will read to get info on what they are allowed to search through and add to the search engines.
http://www.robotstxt.org/orig.html
And to check what he robots.txt file does use http://www.sxw.org.uk/computing/robots/check.html
You need to create a robots.txt file which the search bots will read to get info on what they are allowed to search through and add to the search engines.
http://www.robotstxt.org/orig.html
And to check what he robots.txt file does use http://www.sxw.org.uk/computing/robots/check.html
Message 2 of 3
(353 Views)
Re: Blocking bots and other dubious stuff
04-03-2008 9:40 AM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
Thank you Peter. I'll follow up those links.
Message 3 of 3
(353 Views)
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Plusnet Community
- :
- Forum
- :
- Help with my Plusnet services
- :
- Broadband
- :
- Blocking bots and other dubious stuff