Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Damn Webcrawlers!
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Plusnet Community
- :
- Forum
- :
- Help with my Plusnet services
- :
- Everything else
- :
- Damn Webcrawlers!
Damn Webcrawlers!
26-10-2014 5:49 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
My websites aren't widely relevant so I was a bit surprised to get a sharp complaint from PN saying that I had exceeded my traffic allowance by 7Mb (wow! - that must have really melted the servers). After a bit of webstat studying I noticed a steady rise in downloads from some commercial IP addresses and decided to add a robots file to the sites, and Lo and Behold my traffic has dropped back down to around a twentieth of the previous peak.
I've never bothered about this before, but I imagine that the increase in spiders and search engine activity must be becoming a substantial part of web traffic.
I've never bothered about this before, but I imagine that the increase in spiders and search engine activity must be becoming a substantial part of web traffic.
Democracy - 3 wolves and a lamb voting about what to have for lunch!
Message 1 of 2
(896 Views)
1 REPLY 1
Re: Damn Webcrawlers!
26-10-2014 8:13 PM
- Mark as New
- Bookmark
- Subscribe
- Subscribe to RSS Feed
- Highlight
- Report to Moderator
It is surprising how quickly all the traffic adds up from the crawlers.
It is even worse if you host some large pictures or files and link to them from the main pages (especially .pdf's) - I've had huge bandwidth consumed overnight before from errant crawlers.
It is even worse if you host some large pictures or files and link to them from the main pages (especially .pdf's) - I've had huge bandwidth consumed overnight before from errant crawlers.
Message 2 of 2
(109 Views)
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page