Showing results for 
Search instead for 
Did you mean: 

Damn Webcrawlers!

Posts: 668
Thanks: 163
Fixes: 1
Registered: ‎31-07-2007

Damn Webcrawlers!

My websites aren't widely relevant so I was a bit surprised to get a sharp complaint from PN saying that I had exceeded my traffic allowance by 7Mb (wow! - that must have really melted the servers). After a bit of webstat studying I noticed a steady rise in downloads from some commercial IP addresses and decided to add a robots file to the sites, and Lo and Behold my traffic has dropped back down to around a twentieth of the previous peak.
I've never bothered about this before, but I imagine that the increase in spiders and search engine activity must be becoming a substantial part of web traffic.
Democracy - 3 wolves and a lamb voting about what to have for lunch!
Rising Star
Posts: 3,432
Thanks: 23
Registered: ‎18-01-2013

Re: Damn Webcrawlers!

It is surprising how quickly all the traffic adds up from the crawlers.
It is even worse if you host some large pictures or files and link to them from the main pages (especially .pdf's) - I've had huge bandwidth consumed overnight before from errant crawlers.