Evening all,

As most of you will undoubtedly be aware, this week has seen a few problems with the email platform. This post like a few others that I've made recently, is intended to provide a little more information on our continued commitment to ensuring that we are doing everything possible in order to provide a first class email platform that is resilient, scalable and most importantly reliable.

Background


Some of you will already be aware that we have recently installed a brand new mail storage platform and today started moving customers across to the new system. Further details regarding this work can be seen here. If that's a little too much to take in then make sure you check out the Service Status announcement here as the work will involve a small interruption to your service.

In addition to this, we've also completely rebuilt our mxcore mail delivery platform and almost doubled the number of servers responsible for handling mail collection.

The above changes should certainly be welcomed and will go a long way towards safeguarding the future reliability of the email platform. There is however one core contributor to the problems we have seen that this work doesn't address. That problem lies firmly at the feet of everyone's worst enemy... Spam!

Spam!

Spam is an issue on the rise. The amount of spam flying around the Internet is increasing; 2006 was the worst year for it so far; and this trend is expected to continue. It's clear just how significant a problem this presents by looking at the information gathered from our own systems. The following represents a snapshot of the volumes of email we were processing in the run up to Christmas:



We can clearly see that the volume of mail being processed by the platform increased by about 55% in the 2 weeks before Christmas alone!!!

Such increases in spam puts massive strain on the mail platform...

• We need extra resources to process the mail as it arrives. Without enough processing power mails begin to queue which results in delays for customers.
• Email that's awaiting delivery or awaiting collection has to be stored somewhere. This is what the storage platform is for. With increases like those exhibited above you can expect this to start running out pretty quickly unless you're effectively policing spam.
• All those GB's of mail have to get to us somehow so of course we have to scale the transit connections to the rest of the Internet to cope with the additional traffic.

So what are we doing about this? Well, to begin with we're changing the way in which customers mailboxes are set up when they create an account. By default we have always provided a 'catch-all' email address. What this means is that email is delivered to you irrespective as to what comes before the '@' sign. This makes you very susceptible to spam as emails are often sent to random aliases. This is something that we see crop up all too often in these forums so it will be nice to see the back of it. We're looking to make these changes in the 3rd week of April and I posted a bit of a teaser back in January containing more information that can be seen here.

In light of the most recent spate of mail timeouts we've also taken the decision to make some configuration changes to the way in which our mail delivery servers handle spam. Before explaining these changes it's helpful to have an understanding of how the delivery servers handle your email.

Mail Delivery

When you send an email, the email program or web application you are using sends your message to a local mail delivery server (SMTP Server). The SMTP server then forwards the mail onto the server responsible for collecting and delivering messages for the person you're sending the email to. For emails sent to our customers this is our mxcore platform.

Now consider what would happen if our mxcore servers were unavailable or out of service. It's important that the email isn't lost so it makes sense to have a backup server. These servers are what form our mxlast platform.

Legitimate email rarely needs to pass through our mxlast platform as it is very uncommon for the mxcore servers to be unavailable.

You wouldn't have thought this looking at the volume of email that passes through the server though:



The reason for this is that most of this email is spam. One trick that spammers often use is to directly target the secondary mail server of a domain (in our case the mxlasts). The reason they do this is because these servers are far less likely to have as stringent spam scanning as the primary server meaning the spammers email is more likely to get successfully delivered.

Our SPAM filtering platform

We have already added spam filtering to the mxlast platform as part of our recent mail delivery server upgrade. Whilst this helped to increase spam detection rates, it didn't help reduce the previously mentioned processing and storage problems. This is because the servers still have to deliver the SPAM to the customer.

Our SPAM filtering is a 'multi-layered' process. This basically means that we use a number of different SPAM filtering mechanisms before emails are delivered to the customer. If an email isn't identified as spam then it will be checked by the next process which will pass it on to the next process etc. If any of these processes identify the email as SPAM then the tag [-SPAM-] is appended to the subject line of the email and delivered to the customer.

Proposed Changes

On April 12th we will be making some changes to the way that our mxlast platform handles the mail that is sent to it. Details regarding this work have been posted as a Planned Maintenance Announcement here.

Each email will be checked against one of our SPAM filtering processes and if determined to be SPAM will not be accepted by the platform. This will greatly reduce the volume of email our servers have to process as there will no longer be the overhead associated with delivering these junk emails to customers.

Some customers have aired concern in the past about legitimate email they have received in the past that has been marked incorrectly as [-SPAM-]. It is worth mentioning at this point that this does *not* mean that we will now be dropping this email without the customer knowing.

These mistakes are the result of our Bayesian filter which is a completely separate process to the one we will be using to refuse email on the mxlasts. Because we use this process now as part of our spam filter we have been able to make sure we are confident as possible that no email will be refused incorrectly.

What does this mean?

Well most noticeably we hope it means that anyone plagued by large volumes of spam should see a marked reduction in the amount of junk email they receive.

We have estimated that these changes will reduce mail volumes on the mxlast platform by around 60-70%. That in itself is about 20-30% of the total volume of email passing through our platform. Whilst making SPAM less of a nuisance for customers we are also helping protect the integrity of the mail platform and reduce the potential for problems similar to those witnessed earlier this week.

We are aware that we have many customers who prefer to manage their own email by running their own SMTP server. We would like to assure these customers that the changes to the mxlast platform will not affect them and they will continue to manage their own email as normal.

Feel free to discuss these plans and ask any questions you may have and I'll be more than happy to try and source the answers.

This worries me a bit. Some of my legitimate mail is still being labelled SPAM for silly reasons like having a subject like "How are you?" or being sent from South America.


Will we know that mail has been dropped with or without a way to retrieve the dropped mail? I suspect once dropped it'll be gone forever.

Second question is, will the dropped mail be bounced back to the sender?

I agree something drastic needs doing urgently, though I don't receive any spam. I keep my PlusNet address strictly private for close friends and family and use gmail or yahoo for all other purposes.  My PN mail traffic is very small, could the amount of traffic per account be considered before mail is dropped?

Both Gmail and Yahoo's filters work magnificently (Yahoo's seems slightly better than Google's). I rarely see false positives in either and the very small volume of spam mail that squeezes through (when spammers come up with another way to elude filters) is dealt with very quickly. I don't believe that PN's filters are anywhere near as sophisticated yet.

By the way I think this community forum is a great way to safeguard customers' user names and their email address. Well done !

This is only mail that hits mxlast first.  Genuine mail will hit the mxcore (primary) servers unless they are unavailable.  As Bob explained, Spammers use a technique of purposely hitting the mxlast (secondary) servers as secondary servers generally have less spam protection on them - not in our case. 

We will certainly be erring on the side of caution with this to minimise any chance of false positives.  All bases have been considered.


On your first question, correct.  We'll be rejecting the mail at the delivery stage, so that the mail will not be accepted by the platform.  This will result in a bounce back to the sender in most cases.


Thanks :-)

Trouble with this is that spammers rarely use their own address and use real spoof addresses, I have been subject to many returns like that. Most annoying when one did not send them. Just increased my spam.

Yep... that is a common problem and one that cannot really be resolved.  It's all down to the design of E-Mail.  This should benefit a lot of customers, though, in terms of reducing the amount of inbound spam from spammers.  But yes, if the address spoofed is genuine, and not the spammers, they will get the bounce.

Anon,
Is your email a straightforward email account, or is it via a mail-forwarding website hosting account?
In the latter case, I was able to filter at the hosting site (Just the Name) to just allow genuine addresses through, e.g. specifiednames@myhostname.co.uk, while spam names e.g.  spamnames@myhostname.co.uk gets routed to the host's black hole bin.

Is there any real point in bouncing spam, because most of it is on fake email addresses, otherwise they could be chased and closed down. If this assumption is correct, why not just dump it?

Hi there,

We do have measures in place to reduce back scatter. With this piece of work though the mxlasts refuse to accept the email. It is therefore up to the sending mail server to determine whether or not to requeue, drop or bounce the message.

The events of the last couple of days makes this headline look more than a bit sick.

PN seem to be a disaster waiting to happen, every week.

I switched off the new Spam checking on my email account before it came into force, because I was concerned that it would misidentify some good emails as Spam, and then I would not see them (I use Outlook) until I logged into Webmail and checked the Spam folder.
I was therefore surprised that the spam sharply stopped today - in fact I haven't had a single one.  Is this because the spammers have been targeting your mxlast server directly, and you are now rejecting the spam emails there?  If so, that is somewhat of a relief.  Is it not likely, however, that the spammers will now switch to using the primary servers?

Further to the above, the spam is back - as of 9pm on one of my email addresses.

We host our own SMTP server and back in the good old days when PN could not cope with the amount of mail, all the spammers mail came direct to our server, which allowed us to tag them and setup filtering. When the new servers came online the spammers had a birthday because all there mail was accepted by the mxlast, which left us unbable to active filter the lowlifes. We do this for a number of clients and they noticed an increase in spam within days of the new servers coming online. This left us with having to write filters and rely on spamhaus etc to reject most of it?

Since the new email platform - Squirrel Mail - came on line I am finding that some of my emails that are NOT spam are being labelled as such and are therefore being routed to the Spam folder.

There doesn't seem to be any provision in the Squirrel Mail system to notify the filters that certain emails are not spam and others that get through, are.

Does the old system, whereby you could forward an email to the 'thisisspam@usename.plus.com' or 'thisisnotspam@usename.plus.com' still work? If is does maybe the wrongly labelled emails can be resolved, but its still a time consuming process.

I have started using Googlemail for some of my emails. Their system is very easy to use and I get it to collect my emails from my Plusnet mailboxes. It filters out even more spam. The beauty of their system is that you can select multiple emails and with one click alter the spam or not spam setting.

Maybe someone from PLUSNET should take a look and implement a similar simply system on the Squirrel Mail platform.

DAVE

I'd put that down to coincidence as the fundamental configuration of the spam filter is the same now as it was before we moved to SquirrelMail.

Nope, there's not. There is a plugin though for SquirrelMail that would probably achieve this as per the Usergroup thread here.

Yes, we still provide this functionality. More info here.

As mentioned above, there are filters and modifications that would allow this and it's definitely something we want to look at doing at some point. The problem at the moment is the fact that we're a little loathe to alter any of the underlying SquirrelMail code. After all, this contributed to some of the problems we had with the @Mail implementation. Bear in mind as well that I don't think a 100% formal decision has been made on SquirrelMail and the future of the Webmail platform. I'd be surprised if we made any significant changes until the next release of SquirrelMail is available. Suggestions have been put forward though.