My websites aren't widely relevant so I was a bit surprised to get a sharp complaint from PN saying that I had exceeded my traffic allowance by 7Mb (wow! - that must have really melted the servers). After a bit of webstat studying I noticed a steady rise in downloads from some commercial IP addresses and decided to add a robots file to the sites, and Lo and Behold my traffic has dropped back down to around a twentieth of the previous peak. I've never bothered about this before, but I imagine that the increase in spiders and search engine activity must be becoming a substantial part of web traffic.
Democracy - 3 wolves and a lamb voting about what to have for lunch!
It is surprising how quickly all the traffic adds up from the crawlers. It is even worse if you host some large pictures or files and link to them from the main pages (especially .pdf's) - I've had huge bandwidth consumed overnight before from errant crawlers.