Hi There,
I am the Engineer who has been working on the CriticalPath trial since its inception and I would like to hold up my hand and apologies for the mail problems experienced yesterday. I know how key e-mail is to our customers and the impact that something like this can have on people, and know that no amount of words will make up for what has happened.
I also hope to provide a more technical explanation for why we need to introduce anti-spam solutions like CriticalPath and the events that led to where we are now.
It maybe best to start with the service that you, as customers, receive from us as far as combating the constant and ever increasing deluge of spam. We constantly receive requests for improvement in the area of tagging and deleting spam and it is this battle to improve that drives the work we do behind the scenes.
It is also worth considering how much email is now spam. We have been improving our in-house filtering over time and for quite a while now we have blocked about 60% of all mail as 100% guaranteed spam. Added to this we then tag mail as spam when we are less certain about it and let you the customers manage it.
During the day yesterday, before the problems were identified, the criticalpath solution was collecting data on the messages it was processing. The spam detection rate on that system is more accurate than our own (one of the main reasons why this work is being done) and the reports are surprising. The averages over the full day show a consistent figure of about 90% of all the mail we receive being spam (and that will not include everything as nothing is 100% accurate). A similar trial on our internal corporate mail, which has fewer domains on it, puts that figure at 97%.
It is to pull out this 10% of mail that we have been working to improve, the trial yesterday being one part of that.
Although a full incident report will be provided in the next day or so, the problems were caused by a change in the ACL (Access Control List) on our systems to allow them to integrate with the criticalpath solution for load testing. These changes resulted in mail that would normally have been allowed through being included in the 60% of mail we reject. As soon as this problem was identified the changes were rolled back.
This is not an excuse, there are many things that me personally and business will take from this but it is core to the way we run our company that rather than hide behind a front-line of PR staff we stand up and engage our user community - because without that we would not be PlusNet.
Josh Berry
System Engineer
PlusNet