cancel
Showing results for 
Search instead for 
Did you mean: 

Spam - and how PN can crack it.

Tony_W
Grafter
Posts: 745
Registered: 11-08-2007

Spam - and how PN can crack it.


There are two extra forms of protection - see http://newsvote.bbc.co.uk/1/hi/technology/7027451.stm.
1. Sender Policy Network (SPF) has been around for some time (years), but is still to be taken up by PN.
It would be useful for PN to implement this - both as an additional method of screening emails, and for recipients of PN emails to verify that they actually came from PN customers.
2. DomainKeys Identified Mail (DKIM) is new-ish and is likely to be taken up by Yahoo Mail worldwide within weeks.
They both require extra processing by the email servers to check that the emails are genuine, but look to be 'the future' in spam protection.
Companies can implement both methods at the same time and there are some very big players who are advocating these strategies (Microsoft, Yahoo, Amazon and EBay).
You can vote for PN to implement SPF at http://usergroup.plus.net/pugit/view.php?id=173 where the latest post (10th August) says:
Quote
At this stage we can't commit to do this work. We need more evidence to suggest that this is an item that many customers would benefit from if we are to prioritise this item.

Perhaps PN could also lead the way in implementing DKIM - it would be something that would get them extra publicity, and the lower spam levels could be a draw for new customers.
17 REPLIES
alkatraz
Dabbler
Posts: 24
Registered: 22-06-2007

Re: Spam - and how PN can crack it.

Personally I have to say I'm quite annoyed with Q&A over SPF.
Just The Name is a completely separate domain name registration system owned by PlusNet. It does not support SPF TXT records. Because I was aware of the number of people requesting the ability to add SPF to their domains I WROTE THE CODE TO DO IT! This was in the early part of 2006. Q&A are well aware that the code exists to implement SPF on JTN and because the JTN code is fairly simple, confirming it won't break JTN (which it definitely wouldn't) would take very little time.
But for over a year, absolutely nothing has been done. The code is there, ready to go, but no-one can be arsed to put it in place.
Tony_W
Grafter
Posts: 745
Registered: 11-08-2007

Re: Spam - and how PN can crack it.


I am surprised/shocked/saddened to learn that PN have been so reticent about SPF if they have indeed been aware of the existence of the sample code for so long.
According to Wikipedia SPF was implemented in 2004 by, amongst others, Amazon, AOL, EBay, Google, GMX, Hotmail, Microsoft, and W3C.
It also states that:
Quote
Receivers checking SPF can reject messages from unauthorized machines before receiving the body of the message

Surely that must help with the bandwidth used. It should also lower the current spam processing load with less bayesian filtering and fewer blacklist lookups.
The DKIM system is being vigorously proposed - why doesn't PN get some extra kudos from implementing it early? They say that they are worried about the Christmas spam rush - perhaps they can help do something about it…
Community Gaffer
Community Gaffer
Posts: 17,644
Thanks: 636
Fixes: 162
Registered: 05-04-2007

Re: Spam - and how PN can crack it.

Hmm SPF is something that has been looking into previously but TBH I haven't heard much of DKIM myself. I'll ask around and see if I can get an answer on here for you.
If this post resolved your issue please click the 'This fixed my problem' button
 Chris Parr
 Plusnet Staff
Ianwild
Grafter
Posts: 3,835
Registered: 05-04-2007

Re: Spam - and how PN can crack it.

It's not a case of us being reticent about any of these solutions. We are very open minded about this, but there is no point in us spending development time on things that don't have much benefit. What does SPF actually solve in the current environment in terms of inbound spam? Given that at last count less than 5% of domains had SPF records, and of those many have the 'ALL' wildcard set (Meaning there is effectively no SPF!), dropping email from non SPF compliant domains is impractical. Using SPF records for spam scoring has some value, but not too much.
When it comes to allowing customers to set their own SPF, I think that's a fair request, but it will make no difference to the amount of inbound spam arriving in our customers mailbox. We have always planned to allow this on the next re-write of our domain control system, but at this point in time we simply can't justify giving this priority over other work in progress.
I wasn't aware that any code had been written for JTN alkatraz - could you drop me an email (iwild@plus.net) with details of who you have been working with on that one and where you got to and I will pick it up. It's my understanding though that early this year we re-implemented most of the back-end code powering JTN so I'm not sure if what you have done will still be valid. Again there, I'm presuming this was code to allow the setting of SPF in domains rather than effective spam reduction for inbound mail?
DKIM has more promise for me (It doesn't break mail forwarding and multiple RFCs for a start!), but it still doesn't solve most of the problem at this point in time. I think that's worth more investigation certainly, and I will look into this further myself.
What I would add is that the current plan is to outsource spam management(And all the surrounding systems) to a third party who can provide the expertise in this field needed to operate an effective solution. As part of that work we will have the opportunity to review all of our tools for managing spam, and I will make sure the debate on these items is had. To date, I would argue strongly that we not have ignored any viable and useful technologies for reducing spam significantly but as I say, we do have an open mind about what needs to be done and we are certainly not ignoring the requests we are getting.
Regards,
Ian
alkatraz
Dabbler
Posts: 24
Registered: 22-06-2007

Re: Spam - and how PN can crack it.

Perhaps I ought to add that I was working for PlusNet and one of three people responsible for JTN at the time.
I gave the code to Q&A who kept saying they'd look at it but never did, and after I left one of the remaining people looking after JTN bugged them about it and still there's no sign of anything being done with it.
aaronbennett
Dabbler
Posts: 15
Registered: 04-10-2007

Re: Spam - and how PN can crack it.

Quote from: alkatraz
Personally I have to say I'm quite annoyed with Q&A over SPF.
But for over a year, absolutely nothing has been done. The code is there, ready to go, but no-one can be arsed to put it in place.

Alkatraz, you know that I'm aware of your code modification, and discussed it with you as you wrote it. It has been sat with me over the past year, and I assure you it is not through indolence that it hasn't been implemented. Also note that your code has not been tested or reviewed, and that this was a personal undertaking of yours outside of the Parbin development plan.
We acquired several subsidiary brands after the Metronet takeover; and we have been consolidating and stabalising them all in turn / improving hardware / improving administration / improving efficiency / improving reliability etc.
Currently we are migrating all Pay-as-you-Host customers across to a new cluster of servers based on Plesk hosting software (which allows custom TXT records incidentally).
Although we would like to improve Just-the-Name as soon as possible, and there are many ideas and proposals already on the table, we have had to prioritise our workload so that we address all the existing problems we first inherited. When Just-the-Name is redeveloped, it is likely that the entire domain control system will be scrapped and re-written to allow for much greater functionality than before. Yes, I will be pushing for SPF at that time, but considering the actual programming language will be changed for the revamp, it is unlikely I will be passing your code to QA.
Again, none of these comments have any bearing on Plusnet's own DCS, which is its own entity and not related to JTN.
Community Gaffer
Community Gaffer
Posts: 13,161
Thanks: 932
Fixes: 77
Registered: 04-04-2007

Re: Spam - and how PN can crack it.

Just seen that Ian's already replied, but I'll post my views anyway seeing as I took the time to write them! Wink
Quote from: Tony
1. Sender Policy Network (SPF) has been around for some time (years), but is still to be taken up by PN.
It would be useful for PN to implement this - both as an additional method of screening emails, and for recipients of PN emails to verify that they actually came from PN customers.

What screening would we do? It certainly wouldn't be wise to block mail from domains without SPF due to the number of hosts not using it. Neither would is be wise to mark the mail as spam. I suppose it could be used for spam scoring but it's not the accuracy of the spam filter that's in debate here is it?
The second point you make is very valid and would at the least prevent address forgery and customers suffering where spammers have used their address in the From field. What about sending from PlusNet addresses if you're using a third party relay server though? It would be good for customers to define their own SPF records but I don't think there'd be any visible reduction in the volumes of our own customers spam if we were to do this (although to be clear, I'm not against the idea).
The same points apply to a certain extent to DKIM.
Quote
Personally I have to say I'm quite annoyed with Q&A over SPF.
Just The Name is a completely separate domain name registration system owned by PlusNet. It does not support SPF TXT records. Because I was aware of the number of people requesting the ability to add SPF to their domains I WROTE THE CODE TO DO IT! This was in the early part of 2006. Q&A are well aware that the code exists to implement SPF on JTN and because the JTN code is fairly simple, confirming it won't break JTN (which it definitely wouldn't) would take very little time.
But for over a year, absolutely nothing has been done. The code is there, ready to go, but no-one can be arsed to put it in place.

I must admit that this is news to me but the code being sat there isn't enough for it to be implemented, as I'm sure you're only too aware alkatraz. It would still need to go through the regular avenues and the Just The Name platform isn't high on the list of current priorities I'm afraid as far as adding additional functionality is concerned. I'll certainly ask around though.
For the PlusNet platform the code does not exist and to implement either of these technologies would inolve considerable development resource.
TBH, I think we're at the point where we need to consider the outsourcing things like spam management (or perhaps email as a whole?) - That's where I think we need to be heading.
I'm not sure that many customers would even see the benefits should we ever decide to implement SPF or DKIM ourselves.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

alkatraz
Dabbler
Posts: 24
Registered: 22-06-2007

Re: Spam - and how PN can crack it.

Quote from: aaronbennett

Alkatraz, you know that I'm aware of your code modification, and discussed it with you as you wrote it. It has been sat with me over the past year, and I assure you it is not through indolence that it hasn't been implemented.
etc..

Aaron, who stole your speech processors and replaced them with those of corporate management? That post sounded like a press release(!).
The reason I had expected that someone might do something about it is not because it's my code or something that is urgent, but purely because the changes required are almost zero.
The current system takes user input from an HTML form, writes it out to a file and then those files are periodically parsed and used to overwrite the zone files for each domain. The updated code I created does basically only two things:
1. Adds an "SPF" field to the HTML form the customers get
2. Adds to the parsing script the functionality to pick up the extra stored record and add it to the zone file.
An experienced Perl developer would have been able to duplicate it in 45 to 90 minutes depending on how well they know the current code.
Given the absolutely TINY amount of change to code, the tiny amount of time it would take to test (if testing is even necessary given that the sub-system is tiny) and the fact that it would give domain customers (who are with JTN solely for the purpose of domain hosting) something they've been asking for without having to go to a different hosting company; I really was expecting someone to add it in.
Cost = Near 0
Benefit = Much greater than 0
0 < Much greater than 0

Ian, obviously (as above) speak to Aaron about it.
Tony_W
Grafter
Posts: 745
Registered: 11-08-2007

Re: Spam - and how PN can crack it.


From Bob at 13:03
Quote
What screening would we do? It certainly wouldn't be wise to block mail from domains without SPF due to the number of hosts not using it.

From the BBC website link shown in the first post:
Quote
DKIM lets honest e-mail senders prove they sent a message by encrypting a two-part signature, or key, in a selected part of the mail.
The e-mail provider, such as Yahoo, puts an encrypted private key into the e-mail when it is sent.
It is linked to a public key held by the internet's domain name system - the phonebook of the internet.
The mail server which receives the e-mail checks to ensure that the private and public keys match, proving that the message has come from a genuine sender.

Emails purporting to be from domains that DO use DKIM (e.g. Yahoo) can be checked and those without the correct embedded code can then be rejected, since those emails are definitely not from those domains.
Perhaps it would be better to outsource the email system if you feel that there is little benefit in PN putting in the time and effort. Maybe the other company would have the expertise and time to implement these things.
However, from what alkatraz has said, certainly (at least) the outgoing mail SPF implementation sounds simple enough. Perhaps we wouldn't get quite so many 'Mail undeliverable' returns that clog up your servers and our inboxes when our address are spoofed.
Also, AOL and Yahoo would be less likely to block our legitimate business replies to our customers.
Community Veteran
Posts: 1,621
Thanks: 64
Fixes: 2
Registered: 17-06-2007

Re: Spam - and how PN can crack it.

Quote from: Tony

Quote
DKIM lets honest e-mail senders prove they sent a message by encrypting a two-part signature, or key, in a selected part of the mail.
The e-mail provider, such as Yahoo, puts an encrypted private key into the e-mail when it is sent.
It is linked to a public key held by the internet's domain name system - the phonebook of the internet.
The mail server which receives the e-mail checks to ensure that the private and public keys match, proving that the message has come from a genuine sender.

Emails purporting to be from domains that DO use DKIM (e.g. Yahoo) can be checked and those without the correct embedded code can then be rejected, since those emails are definitely not from those domains.

So how does DKIM work when my server sitting at home sends mail out using 4 different domains. All my outgoing email goes via Plusnet so whose key(s) are used.  Even if I wasn't using my own email server how would it handle me using different "personalities" inside Forte Agent. Plusnet must be forwarding email for thousands of domains - how would it all work?
aaronbennett
Dabbler
Posts: 15
Registered: 04-10-2007

Re: Spam - and how PN can crack it.

Quote from: alkatraz
Aaron, who stole your speech processors and replaced them with those of corporate management? That post sounded like a press release(!).

Corporate management ripped out my sense of identity and bound me to adhere to a diplomatic style of public communication so as not to harm the brand, I have no free will and am presently being whipped into conformity. Don't be silly, I've always written exactly the same way.
The maths you present are over simplified and take absolutely nothing into consideration about what has been happening behind the scenes at the Parbin desk. Trust me I'm a pragmatist not a suit and would push the code change if it was valuable and so easy to do.
JTN is not as you say a hosting company. It is primarily a domain registration agent and only a small percentage of customers use a custom zone file on our getsurfed nameservers. Out of those an even fewer number (<5) have raised lack of SPF as a concern to the support desk over the year.
Thanks for looking into this in the past, but like I say, it's something I'm looking into as part of the JTN overhaul, in my personal opinion it's not something to be forced to the top of the immediate workstack based on the feedback I have received so far.
Tony_W
Grafter
Posts: 745
Registered: 11-08-2007

Re: Spam - and how PN can crack it.

SteveA
That quote was from the BBC website and was concerned with how received email from a domain that uses DKIM can be determined as being genuine.
I do not run my own server, but if you wished to set up the system for your server, I suppose that you would have to lodge a key with the DNS system.
I do not know for sure, but as it is the originating domain that requires the key, I do not think it matters who forwards your email - the private key would be embedded at source. Possibly, email clients will come out that will allow this to happen, should the system become widepread.
At the moment though, emails would only be rejected if they came from a domain which HAD implemented DKIM, but which the keys did not match.
dratddestroyer
Grafter
Posts: 164
Registered: 27-09-2007

Re: Spam - and how PN can crack it.

To the plusnet bods...
I have my own server in Docklands. Runs RHEL and does my mail and web. I don't give out my plusnet email address because i believe PlusNet's email service is far too slow and unreliable.
Using SQLGrey, Spamassasin with lots of plugins and ClamAV  I get one spam a week from thousands of attempts to send spam.
The best bit it is all open source, free software (Except the RHEL but you could use CentOS) that could be easily implimented on one box in a day as a test and then once it is working scale it up.
But Plusnet won't do that....
MikeWhitehead
Grafter
Posts: 748
Registered: 19-08-2007

Re: Spam - and how PN can crack it.

That isn't how business works in relation to live servers. You never take testing in one day and push it live the next, you would just be asking for trouble.
You mention you get one spam a week, but that's to your own personal mail server to a non PlusNet e-mail address. PlusNet has a completely different problem, where their customers addresses have been harvested and are used far more severely than your one will.
Fair enough you may not get much spam, but you only have one e-mail address to contend with. A lot more people have a larger problem with it (aren't so careful with their address, etc), and PlusNet serves tens of thousands (hundreds of thousands I think?) of customers. Even if it was only 1 spam per customer a day, that would be 100,000 spam emails a day they need to deal with. The problem, however, is much larger.
I'd much rather nothing was done for now and they properly tested things rather than go for the "Oh, lets botch these few things together and hope for the best" approach, which would almost certainly raise problems (and make fault resolution take even longer since they haven't tested and got used to all the aspects of each application).