cancel
Showing results for 
Search instead for 
Did you mean: 

Webmail Incident Report

Webmail Incident Report

Webmail Incident Report

The Webmail Incident Report is re-produced below. This will be distributed to customers via our newsletter soon.

Web mail Incident Report

Our customers will be aware of a recent serious security issue affecting our Webmail service. This document contains the results of our investigation and what actions we are now taking to minimise the resulting impact for our customers. We are extremely sorry for the inconvenience and upset this may have caused our customers. We hope this report will provide some reassurance to our customers about how seriously we are taking these events.

Contents

1. Incident Summary 2. Breakdown of events 3. Our response and plans going forward

1.Incident Summary

Further details, explanations and answers to questions we have been asked are below the summary section.

a.What happened?

A malicious attack was performed against vulnerability in our implementation of our Webmail software which allowed the attackers to: - Take a copy of our webmail database, which contained the email addresses of customers who have used Webmail. This also included email addresses contained in customers' online address books and addresses customers had sent to and received from, using our Webmail service. Additionally the database contained some email addresses from our previous Webmail system. - Change code in a web page, served by one of the six webmail servers, to present a pop-up page which, for a period of time, carried a Trojan payload. Only customers without up-to-date Windows software and inadequate anti-virus software would have the Trojan affect their PCs.

b.When did it happen?

Our investigations have shown that the exploit was initiated at around 17:30 on the evening of Friday the 4th May, 2007. Customers started receiving spam on the evening of Sunday, 13th May 2007.

c.Why did it happen?

A vulnerability within our implementation of Webmail code in our portal was discovered and used by malicious attackers. Our subsequent investigations found a number of vulnerabilities with our implementation of the Atmail application, including the vulnerability which had been exploited. This led to the decision we took to stop using the software entirely.

d. Who is suspected of this attack?

This has been the subject of a criminal investigation, which means we are not in a position to share all of the details which we are aware of. However, the timing of the attack and the sophistication of the exploit indicates a considerable amount of planning and expertise. The code was written in Russian and was of high quality.

e.Was this preventable?

With commercial hacking of this nature it just isn't possible to be 100% protected. However, from this incident we have identified a number of processes and procedures that need improving, so that we can minimise the risk of future incidents and reduce the impact of any attacks. We have also increased the frequency of our scanning process to help us identify attempted intrusions in the future.

f. What have we done to resolve the problem and prevent something similar in the future?

Since the issue came to light the entire team at PlusNet has focused on the security of our network and customer data, and email system improvements. In order to resolve the issue and limit the impact on our customers we have: - Undertaken a complete external security audit and rebuilt aspects of our platform that we felt didn't meet stringent security best practices - Created a dedicated PlusNet security team which is formally responsible for all aspects of data and software security on our platform - Pledged to continue to proactive identify and contact customers who exhibit signs of having a malware infection - Improved general security for all customers, such as the introduction of strong passwords, and additional safeguards against further attacks We are also: - Developing a method to allow customers to change from their current username-based email address. - Improving our anti-spam platform to allow more options to have mails which are identified as spam to be separated, reviewed via our website, and/or deleted without them being delivered into the main mailbox - Introducing more improvements to our help and support pages to help customers avoid spam - Researching longer term options to upgrade our Squirrelmail webmail service to a higher specification that integrates with our portal and other communication tools We'll be contacting all our customers over the coming weeks to let them know about the improvements. You can also discuss proposals in the customer forums e.g. http://portal.plus.net

2. Breakdown of events

This section of the document is structured into a timeline of events, split into four key periods: 4th May - 8th May, 9th May - 12th May, 13th May - 16th May and 17th May - 21st May. We have included details of both what we knew at the time and what came to light afterwards. We very much hope this will answer everyone's questions in a direct and factual way, but would ask that customers help us, if they see areas which require further explanation, by commenting on the Blog version of this article. We will be happy to update these details on the basis of comments where we agree that this could be clearer or more informative.

a. Friday 4th May- Tuesday 8th May

Our first indication of an issue came from two tickets picked up on Sunday 6th May by our support team. These were flagged on a handover report by a Customer Support Centre (CSC) shift manager, but within the CSC at that time we were unable to replicate the problem. Initially, as both customers were using the same anti-virus software, the issue was thought to be related to a recent McAfee update which was misreporting a problem. By Bank Holiday Monday (7th May), we had received around 10 reports of the same issue but had still been unable to replicate the problem internally. The significance of the reports had not been recognised at this stage, and in our overall volume of calls and tickets these 10 reports were a tiny percentage of contacts received. It has since transpired that due to the nature of the load balancing system in use on our Webmail platform that connections were 'Persistent'. "This means that each computer is connected to the same server on each attempt, hence not being able to hit the affected server. This resulted in tests from within our CSC all hitting an unaffected server and as such, replicating this issue from our test machines proved to be very difficult. In the early hours of Wednesday 9th May, we identified the real source of the reported problem and were able to begin taking responsive action.

b. Wednesday 9th May - Saturday 12th May

On the morning of Wednesday 9th May we formed an incident response team made up of staff from Networks, Development and Customer Support and a priority problem (42694) was raised. A number of HTML files had been modified on the server 'WM04', which was one of our six public Webmail servers located in London. Each of these machines was a Linux server, operating the Apache Web server software and the Atmail Webmail code. The files which were changed had additional code added to them which generated an 'IFRAME' pop-up containing a link to a video on a Russian website which, when played, activated the Trojan. At this stage we were focussed on understanding the exploit which had resulted in customers potentially being exposed to a malicious webpage. We were not aware that customer email addresses had been obtained illegally at this point. As the significance of the problem was understood we decided to take our Webmail service off-line to check the other servers for the same compromise. A Service Status notice was posted and our Webmail servers were taken offline at 16h00 on Wednesday 9th May 2007. Once we had removed the affected server, we did run a thorough check for the same problem as had affected WM04 across the platform and no compromise was found on the other servers. The underlying problem was resolved by taking the WM04 server out of action so we returned the Webmail platform to service. From Wednesday afternoon, we identified customers with the Trojan data signature on their line; blocked the Trojans from getting config updates and we also looked for changes in customer SMTP traffic patterns. Using this data our CSC began calling customers who exhibited symptoms of the Trojan to offer advice on remedial action.

c. Sunday 13th May - Tuesday 15th May

While we were continuing to address the Trojan issue, a new issue (42837) was raised on the evening of Sunday 13th May. This followed numerous reports of spam to mailboxes which had never received spam before. This immediately took priority over the Trojan investigation due to the number of customers affected but the two issues were worked in parallel. On initial investigation by our on-call engineers we were unable to locate the specific cause of this and none of our monitoring showed a notable increase in overall mail volume on our platform (less than 2% difference to the same period the previous week). By this time the first influx of spam seemed to have stopped. At this stage we believed that we had mitigated the original Webmail security flaw and successfully dealt with the underlying issue. It however appeared likely those responsible for injecting the Trojan on WM04 were also behind the spam outbreak. We did think it would be unlikely for anyone to have run the type of query needed to extract this information without triggering certain alarm systems we have in place. (These are targeted at looking for unusual or large database queries). Following further investigation, at 11h00 on Monday 14th May we managed to replicate a query that could have been used to do this without triggering our monitoring. As such we began cross-checking all of the affected email addresses in order to confirm that a Webmail breach could account for all of the spam sent to email addresses that was being reported. We did eventually find every email address that was reported as receiving these spams within the Webmail system. These were stored in a number of different areas such as address book contacts, account identities and the logs of received and sent mail which were stored within the Webmail system. We issued an announcement alerting customers of our findings and continued with our full scale investigation into the breach. At that stage we decided to leave Webmail running, but took the decision to lock down logins to Webmail and our website to UK only IP addresses to reduce the risk of speculative follow-up attacks. At this stage, we had engineers on site in London who were performing integrity checks and server forensics across the entire platform. We believed that one server (WM04) had been compromised using a vulnerability in part of the Atmail code, which was coupled with some security-related weaknesses on our own server configuration. Once the affected server had been recovered for further forensic investigation, we found a malicious file disguised to look like an image file. This image file contained PHP code that allowed an attacker to run commands on our webmail server. Using this code they were able to transmit tables from the Webmail database to a remote location, and make changes to other HTML files so that visitors were presented a pop-up page containing a Trojan. After a full sweep we also found other files of a similar nature dated from the 4th May 2007 on WM04. The attacker gained information from the Webmail database from several tables which included customer email addresses, entries in customer address books, email addresses for mail sent by customers via Webmail, email addresses for mail received by customers via Webmail. Additionally when we first implemented the Atmail solution in 2004, we imported customers email addresses contact details into the Webmail database from our main databases. We did this to make the solution more convenient for users by setting up their email account for them initially. Customers who joined us after this time only had data created in the Webmail database the first time they logged in. We advised Calacode, authors of Atmail of the incident and they provided excellent support and have since published hotfixes to the code. The incident was confined to the Webmail database and personal data including payment details were not obtained. Our Networks team worked through the night on Monday/Tuesday performing further security checks and server hardening where necessary. This included turning off Website Wizard and our WiFi portal applications, which were not running the latest version of PHP and for which a quick upgrade was not possible. Overnight we also performed penetration testing (Using Nessus and other tools) across our entire platform. In the afternoon of Tuesday 15th May we emailed those broadband customers who were exhibiting signs of being infected by a Trojan virus. We sent an additional email on Wednesday 16th May to dial-up customers who may have logged into WM04. Shortly after this we made a detailed Service Status to customers to explain the progress we had made in our investigation. At this time we also felt we had enough evidence of malicious intent to notify the Police, and an incident was raised.

d. Wednesday 16th May - 21st May

Our network engineers and software developers continued working throughout the night on Tuesday/Wednesday. At this stage we were planning to rebuild our Webmail service using brand new servers configured to use the PHP version of Atmail, rather than our original Perl-based version. At the time we believed this would prove more secure, and our priority was to restore a working Webmail service. As well as building servers, work continued to focus on network hardening throughout night. By Wednesday morning we had identified further potential vulnerabilities within the newest available versions of the Atmail Webmail system and decided to take our existing Webmail out of service and we commenced work on a replacement system using Open Source code. During this time we were continuing to harden every aspect of our platform security we could, and were working closely with the BT security team to ensure that our network was as secure as technically possible. This included work on areas such as the improvement of our scripts that detect abnormal sessions activity and the commencement of work to allow stronger customer passwords. By Thursday afternoon we had completed the majority of these improvements and our attention turned to restoring a working Webmail service, using alternative Webmail software. By Friday, a temporary Webmail solution based upon the SquirrelMail software had been built and we were in the final stages of security, penetration and functionality testing of the new servers and software. Our replacement Webmail service was released on Saturday evening. 3. Our Response and plans going forward Throughout the Webmail issue, we have been keen to listen to our customers, and we hope this document answers many of our customers' questions about the incident. As soon as we were aware of the severity of the situation we swung into action and we feel we can be proud of our response to the issue. Many members of our Networks, Development and Customer Support Teams gave up their social and family lives and gave their own time to work on implementing the things that we identified as needing to be done urgently. We are determined that out of these issues will come a change in our company culture We want to be regarded as an on-line business where security is placed at the forefront of everything we build. We are currently in negotiation with a number of potential partners who we will work closely with to ensure that our network and the software running on it always remains as secure as it is possible to be and that this is regularly audited by a third party. We are also reviewing processes to ensure that diagnosis, communication and resolution can be swifter in the future. We have had literally hundreds of suggestions and requests from customers regarding ways to reduce spam and we are grateful for them all. Listed below are the current items we are able to commit to at this stage, timelines for which will be provided as soon as possible. Other customer suggestions are currently being reviewed and prioritised. We have been grateful for the response and support expressed by so many of our customers and we want to repay that by ensuring that we deliver as much of what is requested as soon as we possibly can. From the suggestions we have received from customers so far we will implement the following: 1) The ability to change your username 2) Get a free .uk domain name 3) SSL encrypted connections for POP3 and IMAP email and FTP 4) Improvements to the 'Manage my mailbox' tool 5) The ability to blackhole username@username.plus.com 6) Publishing our spam detection rates on our website for comparison with other ISPs 7) Publishing our new Privacy Policy and Data Retention policies on our portal. We will contact all our customers with firm timescales for the above improvements. --- Again, apologies for the lateness of this one tonight - It's been another late one for us all. Regards, Ian Wild

0 Thanks
18 Comments
915 Views
18 Comments
Community Veteran
A useful report; thanks Ian.
Community Veteran
N.B. the link to this from the homepage of the community website doesn't work: the link points to http://blah/05/24/webmail-blah, rather than /05/23/.
Not applicable
One cannot help thinking that many of these new security measures should have been in place in the first place - they are best practice after all. It is not as if these risks are unheard of or unprecedented - apparently a similar breach took place at Supanet. I feel that in addition to tightened security Plusnet should be paying financial compensation to customers who have had their clean email addresses handed over to spammers.
Not applicable
Why were penetration tests not carried out by a third party BEFORE this incident? It is very worrying that you only noticed AFTER customers had started to receive spam. This is gross negligience. Myself and my colleagues are very unhappy that our company email addresses have got into the hands of spammers, and our client address books too. I think it is now time to move. When the whole broadband service along with the phone lines went down last year, because PlusNet were relying on one power source, we thought 'what a bunch of amateurs'. This confirms it.
sh
Not applicable
While your reaction to this event from a technical perspective seems to have been adequate, it all strikes me as a perfect example of shutting the stable door after the horse has bolted. If we believe everything you say you have achieved in terms of making your system more secure in the relatively short space of time since the atack, how much simpler and better for everyone would it have been if you'd done a proper security audit of your systems before rolling them out? How can you possibly have thought it acceptable to have a large IT organistation of any sort, let alone an ISP with the customer base that PlusNet has without a dedicated security manager beforehand. This is simply amateurish. In the 4 years I have been a PlusNet customer, I have certainly never been charged for an amateur service and am extremely upset that poor quality, inadequately tested software and absence of management have resulted in my email address details, and those of friends, relatives, colleagues and clients being made available to the spammers. Through carefully managing who email addresses are made available to, my wife and daughter have never received a single bit of spam. Over the last couple of weeks, they have been receiving a dozen or so messages each day with the sort of content that nobody, especially children, should be exposed to. I have "fixed" this by deleting their original mailboxes and creating a new email address for them each, which is a damned nuisance - it would be unthinkable for a phone company to make a customer change their number to avoid nuisance callers as a result of the company's mistake and we get a lot more email than phone calls these days. This also simply shifts additional spam to my catch-all mailbox. Given that your spam filter regularly marks valid messages as spam, I simply don't trust your new service not to dump important messages - it seems to have a particular problem with order-confirmation emails from online retailers who use "sales@domain" type addresses. More concerning for me is that the spammers also seem to be using the harvested email addresses to "spoof" the "from" fields for their filth which they send to other harvested addresses. The IT savvy realise what's happenning, but it took some time to explain to a less clued-up contact that it really wasn't me trying to sell "Wonder Dik". Thanks for that. As for communication with your affected customers, I happen to think this has been pretty poor. This report was originally promised several days ago and is still pretty well buried in the website. Given the seriousness of the incident and the prominence it has had (were you aware that this has was #1 "Technology" story on the BBC news site yesterday?) this report should be linked to from a prominent position on your homepage. Why you couldn't also have emailed affected customers a link as soon as it was available, I really don't understand. In the email communication we have had, we were advised to chnge our password, but not told whether that meant our individual mailbox passwords as well as our overall login. You also told us that the hackers had gained access to the webmail system but not credit card details, but have not said whether the contents of emails themselves were made available or if "only" address books etc. were targeted. I would also like to know how your square the statement above, "Throughout the Webmail issue, we have been keen to listen to our customers" with the email request from Phil Webb on 17/05/07, "I would be most grateful if, during the next few days, you could avoid contacting us..." - surely that's actually a sign you were keen for your customers to shut up and let you decide what was important. All in all, a pretty sorry episode, really. On a personal level, I feel sorry for the guys who have worked through the nights and into their weekends to fix this, but if they have had to do this in their own time, that's an indication of PlusNet being a bad employer expecting unreasonable committment from employess, not something to shout about.
Not applicable
I still don't feel that I really understand how the hackers managed to breach security, you say : "A vulnerability within our implementation of Webmail code in our portal was discovered and used by malicious attackers". Is there any chance you could expand on the exact nature of the vulnerability and how it allowed hackers to exploit it? After all you have plugged that hole now and greater awareness of the problem might be useful to others exposed to such vulnerability.
Grafter
No one was forced to work to solve this - If you think PlusNet is some big corporate employer who would behave like that, I believe you have it wrong. We work as a team here and everyone wanted to pull together and do everything they could - That's a great thing about working here - especially when things go wrong. While we asked people not to contact support directly about this we've been keen to keep people in the loop, and hear their comments in forums etc - If everyone with something to say about this had called support, any customers with other routine problems would have had no chance of getting through. We will be putting the report on our website and sending a link to it to all of our customers. We have nothing to hide here and we don't want to avoid telling people what happened. That said, we still can't go into all of the details because that makes us more vulnerable and is just asking for trouble in some ways. Finally - If you find our spam filters marking mail as spam, please ensure you let us know as per the instructions at http://www.plus.net/support/security/spam/spam_protection_guide.shtml?supportb=index_t5spamprotectionguide#5 On the other comments, I wouldn't like to say more than we have - We did carry out regular security checks on our core network and as a general rule we operate and resilliant network environment. There is no excuse for the compromise against webmail service mind you, and I'm not trying to make one. Ian
Not applicable
I appreciate that for reasons of security and the on going criminal investigation there are details which are not disclosed in the report. I am concerned, however, that details of the attacker's level of access to the platform remain vague. The report says: "A number of HTML files had been modified on the server...", and "The attacker gained information from the Webmail database....", presumably using the file which "contained PHP code that allowed an attacker to run commands", taken together this strongly suggests that the attacker(s) had gained privileged access to the server's filing system. If this is the case they obvious had the facility to scan and steal ANY data that was on that server, including the contents of e-mails. This incident was brought to the attention of force9/plusnet by the actions of the attacker, i.e. they chose to reveal their activity. Given the attacker's ability to manipulate the server, and possibly cover their tracks, how can we be certain that the server was not 'silently compromised' prior to this alert, or indeed that any of the other servers have remained untouched? Something does not add up. The report acknowledges that the attacker was not just an average script-kiddie and that significant planning had been put into the attack. This seems like a great deal of effort to just propagate a virus and generate nuisance e-mails! Pete
Community Gaffer
"This seems like a great deal of effort to just propagate a virus and generate nuisance e-mails!" Not really! I'm sure they sell lists of email addresses for quite a bit, and sell the use of their zombie networks for distributing spam. It's all about the cash.
Not applicable
[...] information about the changes we are making for our customers, announced in the webmail incident report, are now available. These deliverables [...]
Not applicable
[...] These activities are additional features as well as the deliverables in the Webmail Incident Report [...]
Not applicable
I'm a bit confused as to why the contact email addresses provided to plusnet for account management purposes were imported into the webmail system. This seems like unnecessary replication of confidential customer details to potential insecure locations for no benefit. There was completely no reason for this. As a result of this action, the email address I gave to plusnet confidentially for communication with plusnet has now been given away to spammers. Can I suggest in future that in order to keep confidential customer details confidential (as require by the data Protection Act) plus net doesn't mix data stored as part of a customer's confidential account details with data stored as part of the process of operating a customer's account, such as webmail messages and address books. I guess we should just count ourselves lucky that plus net didn't decide to replicate the credit card details onto the webmail database. Please in future think about the consequences of unnecessary replication of data, and what it means to keep data confidential or secure.
Grafter
There was a benefit for most customers, in that we automatically set their from email address for them when they used webmail. We were asked to do this by a number of customers, and it was a replication of how our previous webmail system had worked. At the time the implications never occurred to us, which we obviously regret. In the future, following the implementation of very strict new data retention policy that we will commit to and follow to the letter, this will be something we would completely avoid. Ian
mh
Not applicable
This whole episode has been a disaster for me, no thanks to plus.net. An important email address which I had taken very strenuous efforts to keep spam away from is now receiving a continual stream of spam. I wrote to plus.net to ask for financial compensation, and they have refused, and in their haste they even replied by sending me a letter addressed to another customer instead of to myself, which is sadly typical of the overall degree of care they give to their customers these days. There is now no way to undo the spam, only the option of filtering, which carries with it an unacceptably high risk of false positives. For an independent opinion on the incident, you need look no further than the following article on The Register. It is of course rightfully critical of plus.net. http://www.theregister.co.uk/2007/05/15/plusnet_spam_attack_may/ By the way, I have "referrals" on my plus.net account because in better times I recommended the service to others. But I certainly have no intention of doing that any more, and when I finally get round to sorting out some things so as to be in a position to migrate to another ISP, I'll do my best to take my with me also those whom I referred to plus.net in the past.
Grafter
I'm sorry to hear how you have been affected by this. We are not offering financial compensation, but we certainly don't wish to lose custom and are making ever effort we can to build back confidence in our platform by doing the right things all of the time. Regards, Ian
Not applicable
[...] on the webmail incident report, we pledged to provide answers to any remaining queries which the original report didn’t address. [...]
Not applicable
This is totally unaceptable. Your responses to customers concerns are answered with an attitude that is rude and dissmisive. You offer no compensation while people are having to spend time sorting a problem that Plusnet created. I have just tried to contact your Customer Services dept only to be put through to an automated system telling me there is a 20 min wait. You then send me 6 emails to tell me that you would like me to fill in a survey. I suggest you read your own forum and you will see the level of feeling without the need for a survey! The situation is now getting worse with more rubbish e-mails each day. I sent in a request for compensation as a reply to my bill, I have yet to get a responce.
Newbie
OK, its 2 years on (2009). Where are we with the intention to provide "3) SSL encrypted connections for POP3 and IMAP email and FTP"?Huh