Prevention of the exceeding of allotted bandwidth.

[#1]

I have recently been taken to task by PlusNet for the fact that my two domains have exceeded my allotted bandwidth. As a result my domain content was archived and I had three weeks of domain loss before the re-instatement of the sites took place.

Now, whereas I can control the web servers of the four subdomains which I host in my office, I have no such control of the PlusNet webservers (possiboly quite understandably). So strictly, can I be held to account (and my sites be archived) if I cannot control access to my domains?

Perhaps someone could explain:
- am I able to setup deny lists for my sites? If so where?
  (I apparently have no access to Apache's .conf files, and I accept that it is "verboten"!);
- does access denial in ".htaccess" (by domain or IP) work in the root, and other directories of the domains?

I would like to get rid of these domain archiving incidents, so any advice will appreciated.

(Incidentally many of the offending IP addreses belong to Microsoft.com, in Ireland, USA (Redmond), and Japan.)

Thanks in advance.

Kenneth

[#2]

---

@kaspencer wrote:

I would like to get rid of these domain archiving incidents, so any advice will appreciated.

---

Have you considered moving your domains to a professional hosting platform with greater bandwidth and probably more accessible controls?

[#3]

@baldrick Thanks for the reply. I'm waiting for the move by Plusnet.
But any effective advice within the limited control which I may have of the domains will still be welcome.

[#4]

Without knowing the contents of your site it's difficult to give any meaningful advice.

So here's some generic advice. Keep documents and large images off site using links. Use 'robots.txt' to limit web-crawlers. https://www.cloudflare.com/learning/bots/what-is-robots-txt/ & other info sources.

[#5]

I use robots.txt to deter unwanted bots though how much bandwidth it saves is questionable.

[#6]

The usual bandwidth killer is high resolution images.  Such should be made more suitable for website usage.  Alternatively host the images elsewhere (OneDrive) and provide links to them from your web pages.

[#7]

The OP could insert a simple redirect in the index file to web space on his/her office system.

[#8]

@Champnet @Townman 

Thanks for the replies.

It doesn't seem to me that the excess bandwidth is due to Hi-Res images.

A huge proortion of the excess is due to enormouslly massive searches for non-existent files (wp admin and php admin files) generating huge log files which I have been told all contribute to bandwidth consumption. There are also a significant number of whole-site downloads.For example on 20250825 the log file for Atworth.org.uk was 975k. One of the PlusNet support staff told me that log files are part of bandwidth usage.

Almost none of the image files are hi-res anyway, and none are very large.

On the four subdomains sites which I run on my own office server I can stop these in their tracks using my Apache2 configuration and .htaccess files, I cannot do that on the domains hosted on PlusNet as I do not control the server.

Apparently this issue won't exist for me as the replacement web hosting offers unlimited bandwidth.

Thanks for the comments ...

Kenneth Spencer.

[#9]

IIRC and you know what you are doing, I believe there is or you can place a .htaccess file in the root directory via FTP., akin to creating the robots.txt file.

It is worth a try!

[#10]

@Townman 

Thanks.

Actually I already have an .htaccess file in the root directory.

I suspect that it may not be enabled, as on my server in my office I have to reload or restart Apache before any new changes are put into effect. So it's difficult to know when they might be activated, as I can neither reload nor restart the PlusNet web server..
It also seems to me that even if you deny from host or deny from ip (in.htaccess) it may not stop a visitor from reading the file list (html, images or otherwise) or asking for any number of files but it seems to me that the directory can still be scanned for any files, and the file search ends up being listed in the log anyway, resulting in bandwidth consumption according to the number of files targetted. Sometimes the file list searched is hundreds of files.

Thus are my conclusions from reading my logs for about 2 decades, and it gets steadily worse. I'd like to know whether there is a directive we can put in the .htaccess file which simply makes an unauthorised site blind to all content, and does log the files that have been targetted in the search, but it doesn't seem that way.

Thanks. It's good that you are interested

Kenneth Spencer

[#11]

@kaspencer   I use the following in my htaccess files, does as the comment says...

#### /disallow directory browsing ####
Options All -Indexes

[#12]

---

@kaspencer wrote:

... it may not stop a visitor from reading the file list (html, images or otherwise) or asking for any number of files but it seems to me that the directory can still be scanned for any files ...

---

I am more than a tad rusty in this space; would the presence of a default.htm / index.htm file inhibit directory listing in a web browser?  There dose not need to be anything in the file, but I think it would inhibit the web browsers from returning a directory listing.

[#13]

.htaccess in root directory of F9/PN normal webspace works just fine in my experience. I used to have an enormously long .hta file with an ever-growing list of banned single IPs, blocks of IPs and even entire countries ! Lots of other URL tweaking going on as well at various times such as redirecting anything particularly large to a host elsewhere or redirects during updates for instance.

As far as I'm aware, .htaccess is generally available by default and does not, in my experience, require Apache config changes or a reboot etc. If you screw something up (such as by introducing an endless loop in the rules or whatever) so your website no longer works and your browser just complains and gives up, you can just edit the .hta via ftp and the updated .hta is then used almost instantly. However, the Apache mod_rewrite function does need to be enabled but that's done in the .hta file rather than in a .conf or whatever. Make sure the first 2 lines of your .hta are:

[code]
RewriteEngine on
RewriteBase /
[/code]

I believe that it's entirely possible to have multiple .htaccess files. There is most likely a default .hta somewhere in Apache, then the .hta in the root directory, then any .hta in sub-directories etc. I believe the last .hta found takes precedence but I've only ever used a single somewhat lengthy .hta in the root directory. PN probably hate me bigtime because it no doubt took far longer to process the .hta on every attempt than it would have to serve the content being abused

I'm obviously not sure exactly what you are actually trying to do but I used to display a small B&W .gif image instead of any requested content. Things like "banned IP". "restricted access" or "no hot-linking" etc.  My .hta was therefore something like this:

[code]
# Options +FollowSymlinks
# Options -FollowSymlinks
# Options +Indexes
# ErrorDocument 404 /404filename.htm
RewriteEngine on
RewriteBase /

# *** Always display warnings regardless of any other rules ***
RewriteRule ^WARNINGS/(.*) - [L]

# *** IPs banned completely for whatever reason ***
RewriteCond %{REMOTE_ADDR} xxx.xxx.xxx.xxx [OR]
RewriteCond %{REMOTE_ADDR} yyy.yyy.yyy.yyy [OR]
RewriteCond %{REMOTE_ADDR} zzz.zzz.zzz.zzz
RewriteRule .* WARNINGS/banned\.gif [L]

# *** IPs getting tweaked images for whatever reason ***       
RewriteCond %{REMOTE_ADDR} xxx.xxx.xxx.xxx [OR]
RewriteCond %{REMOTE_ADDR} yyy.yyy.yyy.yyy [OR]
RewriteCond %{REMOTE_ADDR} ^78\.10\.163\.[0-9]{1,3} [OR]
RewriteCond %{REMOTE_ADDR} ^5\.173\.[0-9]{1,3}\.[0-9]{1,3} [OR]
RewriteCond %{REMOTE_ADDR} zzz.zzz.zzz.zzz
RewriteRule .* WARNINGS/bandwidth\ warning\.gif [L]

# *** Sites banned completely for hotlinking ***
RewriteCond %{HTTP_REFERER} usergroup\.plus\.net [NC,OR]
RewriteCond %{HTTP_REFERER} xxxxxxxxxx\.com [NC,OR]
RewriteCond %{HTTP_REFERER} yyyyyyyyyy\.co\.uk [NC,OR]
RewriteCond %{HTTP_REFERER} zzzzzzzzzz\.something [NC]
RewriteRule .* WARNINGS/do\ not\ steal\ bandwidth\.gif [L]
[/code]

The first 4 commented out lines don't always work on all Apache implementations. They're very useful but sometimes they will work, sometimes they don't. The next 2 lines are absolutely necessary or nothing you enter in .hta will ever work ! I had loads of other stuff as well but it's just the more obvious things shown above by way of a quick example. The [L] means stop processing .hta and exit if the conditions has matched and rewrite has taken place.

Don't forget that .hta is processed on every access which means even after any rewriting has taken place. It's therefore very easy to set up a loop or hang if you're not careful. That's why I had the first rule to always display warnings before even thinking about doing anything else. The first time .hta is processed, the requested URL is tweaked accordingly if necessary. The second time .hta is processed, the relevant warning action will always be taken. Otherwise, .hta is just going to do the rewrite again because it will find the same IP or referrer match again regardless of the content now being requested.

I'm by no means anything even remotely close to being any kind of expert on this ! Just several decades of mostly trial and error plus plenty of hints, advice or pointers from users who (allegedly) know more than me. I think .hta is a very powerful tool *IFF* you can master it (and also regex) sufficiently well enough to accomplish whatever it is that you want to achieve. My only other thoughts are:

1) Make sure directory listings are disabled by one means of another if only by ensuring that every directory and sub-directory has an index.htm file even if it does nothing much. If there is no index.htm found in a directory then browsers/bots will generally see the entire directory contents listed out for further abuse.

2) You can't do anything much about abusers/bots repeatedly trying to access a lengthy list of mostly non-existent files. But each attempt will only result in a #404 being served if you do nothing in the .hta or elsewhere to limit them. Always ensure that your #404 page or any blocking action you may take in .hta uses minimal bandwidth. My (often very rude :P) warnings only consumed just a few bytes.

3) Make sure you have a robots.txt file in the root directory. Not just for (hopefully) preventing specific bots/agents if needs be but to (hopefully) prevent scanning of directories and sub-directories that you don't actually want to be scanned and potentially be made public. I had a "disallow" entry for every directory and subdirectory that contained anything I definitely wouldn't want to see listed in search engines various.

4) I feel your pain TBH ... PN really are a total PITA with the draconian actions taken for (often) minor breaches of bandwidth usage. Especially when it's not actually that easy to keep on top of your bandwidth usage or prevent the ever-growing problems caused by the ever-increasing number of abusers in any case. I know you keep a close eye on webstats just like me but when these problems arise, you only get to see the problem after the event anyway by which time it's often too late. All my historic stuff is hosted elsewhere now after my one and only major run-in with PN archiving my webspace and taking a dog's age to deal with restoring access. You *WILL* be so much happier and feel so much less hassled/abused when your site is hosted elsewhere

In most cases >99% of my (now unlimited) bandwidth is actually consumed by abusers running though a lengthy list of possible files they'd like to find for reasons various. Also, like you, I've noted that M$oft network IPs being by far the worst offenders for abuse in general. It seems that if you happen to send a link to someone who uses outlook and/or office365 then you subsequently get plagued by M$oft servers trying to obtain every possible scrap of data from your site/server based on data in that link. Not just once or twice but repeatedly for days/weeks/months. Numerous attempts by numerous different IPs all downloading the very same stuff (and scraping everything else they can along the way) over and over and over again for absolutely ages. If that had happened when I was using my webspace then it obviously would have blown the bandwidth limit with monotonous regularity.



BTW, apologies if any of the above falls into "teaching granny to suck eggs" or "as published by the Ministry of Stating The Bleedin' Obvious" category and all that. Hopefully there's something that may be of some help to you though If you have any specific questions or whatever than I'll try to help if/when I can ... but it could well turn out to be a distinct case of the blind leading the blind I'm afraid :lol:

[#14]

Yeah, I know it's stupid o'clock but I thought I'd do some playing around with .hta files on F9 webspace

This /might/ perhaps be a bit naughty but I've just tried it on Ye Olde F9 webspace without any obvious issues. Assuming you have a list of known abusive IPs or blocks of IPs and are happy to keep updating it as new ones appear then it may be worth a try as an alternative to just serving a blank page, a tiny graphic or the usual #404 etc.


[code]
# Options +FollowSymlinks
# Options -FollowSymlinks
# Options +Indexes
# Options -Indexes
# ErrorDocument 404 /404filename.htm
RewriteEngine on
RewriteBase /

# *** Redirect Abusive IPs To Somewhere-That-Does-Not-Exist ***
RewriteCond %{REMOTE_ADDR} xxx\.xxx\.xxx\.xxx [OR]
RewriteCond %{REMOTE_ADDR} yyy\.yyy\.yyy\.yyy [OR]
RewriteCond %{REMOTE_ADDR} ^78\.10\.163\.[0-9]{1,3} [OR]
RewriteCond %{REMOTE_ADDR} ^5\.173\.[0-9]{1,3}\.[0-9]{1,3} [OR]
RewriteCond %{REMOTE_ADDR} zzz\.zzz\.zzz\.zzz
RewriteRule ^.* http://www.somewhere-that-does-not-exist [R,L]
[/code]


Possible reasonable candidates for www.somewhere-that-does-not-exist:

1) An undefined sub-domain eg blackhole.your-domain.co.uk

2) Localhost eg 127.0.0.1

3) Some other domain/website under your control where bandwidth isn't a significant issue eg a free hosting service somewhere.


I can't see what the logfiles actually look like for a day or so but it ought to show the original request coming from a problem IP getting a 301/302 redirect response. If the requested file actually exists then it will not be served and nor will a #404 page be served if the file doesn't exist.

However, bear in mind that whatever you do, it might not actually resolve the problem in reality due to the way PN monitor bandwidth.  Unfortunately, PN use data from the radius server I believe rather than using the Apache server logs that we get. It is therefore entirely possible for PN to determine that your allocated bandwidth has been exceeded but the Apache raw logfiles or webalizer data shows that everything is just about OK. I've also had regular suspicions over the years that the raw logfiles supplied often seem to be a bit light on actual traffic. Traffic that I know happened. Traffic that PN would no doubt be using in their bandwidth analysis. The redirect response data may well be counted by PN in any case of course and it could even potentially be in excess of that for just serving a blank page or whatever !

Once you have a .htaccess file in place and apparently working as expected, all you can do is try the various options available for dealing with abusive requests and see which method (if any) results in a reduction in bandwidth used. Have fun ... and keep looking forward to that shiny new server coming sometime real soon(ish)

[#15]

PS: You could also try replacing the RewriteRule for redirecting to www.somewhere-that-does-not-exist with " RewriteRule ^.* - [F] " which should just generate a #403 response instead. This also appears to work as expected on Ye Olde F9 webspace. Probably not that much different to a #404 in terms of bandwidth used though.