Webmail Incident Report
The Webmail Incident Report is re-produced below. This will be distributed to customers via our newsletter soon.
Web mail Incident Report
Our customers will be aware of a recent serious security issue affecting our Webmail service. This document contains the results of our investigation and what actions we are now taking to minimise the resulting impact for our customers. We are extremely sorry for the inconvenience and upset this may have caused our customers. We hope this report will provide some reassurance to our customers about how seriously we are taking these events.
1. Incident Summary 2. Breakdown of events 3. Our response and plans going forward
Further details, explanations and answers to questions we have been asked are below the summary section.
A malicious attack was performed against vulnerability in our implementation of our Webmail software which allowed the attackers to: - Take a copy of our webmail database, which contained the email addresses of customers who have used Webmail. This also included email addresses contained in customers' online address books and addresses customers had sent to and received from, using our Webmail service. Additionally the database contained some email addresses from our previous Webmail system. - Change code in a web page, served by one of the six webmail servers, to present a pop-up page which, for a period of time, carried a Trojan payload. Only customers without up-to-date Windows software and inadequate anti-virus software would have the Trojan affect their PCs.
Our investigations have shown that the exploit was initiated at around 17:30 on the evening of Friday the 4th May, 2007. Customers started receiving spam on the evening of Sunday, 13th May 2007.
A vulnerability within our implementation of Webmail code in our portal was discovered and used by malicious attackers. Our subsequent investigations found a number of vulnerabilities with our implementation of the Atmail application, including the vulnerability which had been exploited. This led to the decision we took to stop using the software entirely.
This has been the subject of a criminal investigation, which means we are not in a position to share all of the details which we are aware of. However, the timing of the attack and the sophistication of the exploit indicates a considerable amount of planning and expertise. The code was written in Russian and was of high quality.
With commercial hacking of this nature it just isn't possible to be 100% protected. However, from this incident we have identified a number of processes and procedures that need improving, so that we can minimise the risk of future incidents and reduce the impact of any attacks. We have also increased the frequency of our scanning process to help us identify attempted intrusions in the future.
Since the issue came to light the entire team at PlusNet has focused on the security of our network and customer data, and email system improvements. In order to resolve the issue and limit the impact on our customers we have: - Undertaken a complete external security audit and rebuilt aspects of our platform that we felt didn't meet stringent security best practices - Created a dedicated PlusNet security team which is formally responsible for all aspects of data and software security on our platform - Pledged to continue to proactive identify and contact customers who exhibit signs of having a malware infection - Improved general security for all customers, such as the introduction of strong passwords, and additional safeguards against further attacks We are also: - Developing a method to allow customers to change from their current username-based email address. - Improving our anti-spam platform to allow more options to have mails which are identified as spam to be separated, reviewed via our website, and/or deleted without them being delivered into the main mailbox - Introducing more improvements to our help and support pages to help customers avoid spam - Researching longer term options to upgrade our Squirrelmail webmail service to a higher specification that integrates with our portal and other communication tools We'll be contacting all our customers over the coming weeks to let them know about the improvements. You can also discuss proposals in the customer forums e.g. http://portal.plus.net
This section of the document is structured into a timeline of events, split into four key periods: 4th May - 8th May, 9th May - 12th May, 13th May - 16th May and 17th May - 21st May. We have included details of both what we knew at the time and what came to light afterwards. We very much hope this will answer everyone's questions in a direct and factual way, but would ask that customers help us, if they see areas which require further explanation, by commenting on the Blog version of this article. We will be happy to update these details on the basis of comments where we agree that this could be clearer or more informative.
a. Friday 4th May- Tuesday 8th May
Our first indication of an issue came from two tickets picked up on Sunday 6th May by our support team. These were flagged on a handover report by a Customer Support Centre (CSC) shift manager, but within the CSC at that time we were unable to replicate the problem. Initially, as both customers were using the same anti-virus software, the issue was thought to be related to a recent McAfee update which was misreporting a problem. By Bank Holiday Monday (7th May), we had received around 10 reports of the same issue but had still been unable to replicate the problem internally. The significance of the reports had not been recognised at this stage, and in our overall volume of calls and tickets these 10 reports were a tiny percentage of contacts received. It has since transpired that due to the nature of the load balancing system in use on our Webmail platform that connections were 'Persistent'. "This means that each computer is connected to the same server on each attempt, hence not being able to hit the affected server. This resulted in tests from within our CSC all hitting an unaffected server and as such, replicating this issue from our test machines proved to be very difficult. In the early hours of Wednesday 9th May, we identified the real source of the reported problem and were able to begin taking responsive action.
b. Wednesday 9th May - Saturday 12th May
On the morning of Wednesday 9th May we formed an incident response team made up of staff from Networks, Development and Customer Support and a priority problem (42694) was raised. A number of HTML files had been modified on the server 'WM04', which was one of our six public Webmail servers located in London. Each of these machines was a Linux server, operating the Apache Web server software and the Atmail Webmail code. The files which were changed had additional code added to them which generated an 'IFRAME' pop-up containing a link to a video on a Russian website which, when played, activated the Trojan. At this stage we were focussed on understanding the exploit which had resulted in customers potentially being exposed to a malicious webpage. We were not aware that customer email addresses had been obtained illegally at this point. As the significance of the problem was understood we decided to take our Webmail service off-line to check the other servers for the same compromise. A Service Status notice was posted and our Webmail servers were taken offline at 16h00 on Wednesday 9th May 2007. Once we had removed the affected server, we did run a thorough check for the same problem as had affected WM04 across the platform and no compromise was found on the other servers. The underlying problem was resolved by taking the WM04 server out of action so we returned the Webmail platform to service. From Wednesday afternoon, we identified customers with the Trojan data signature on their line; blocked the Trojans from getting config updates and we also looked for changes in customer SMTP traffic patterns. Using this data our CSC began calling customers who exhibited symptoms of the Trojan to offer advice on remedial action.
c. Sunday 13th May - Tuesday 15th May
While we were continuing to address the Trojan issue, a new issue (42837) was raised on the evening of Sunday 13th May. This followed numerous reports of spam to mailboxes which had never received spam before. This immediately took priority over the Trojan investigation due to the number of customers affected but the two issues were worked in parallel. On initial investigation by our on-call engineers we were unable to locate the specific cause of this and none of our monitoring showed a notable increase in overall mail volume on our platform (less than 2% difference to the same period the previous week). By this time the first influx of spam seemed to have stopped. At this stage we believed that we had mitigated the original Webmail security flaw and successfully dealt with the underlying issue. It however appeared likely those responsible for injecting the Trojan on WM04 were also behind the spam outbreak. We did think it would be unlikely for anyone to have run the type of query needed to extract this information without triggering certain alarm systems we have in place. (These are targeted at looking for unusual or large database queries). Following further investigation, at 11h00 on Monday 14th May we managed to replicate a query that could have been used to do this without triggering our monitoring. As such we began cross-checking all of the affected email addresses in order to confirm that a Webmail breach could account for all of the spam sent to email addresses that was being reported. We did eventually find every email address that was reported as receiving these spams within the Webmail system. These were stored in a number of different areas such as address book contacts, account identities and the logs of received and sent mail which were stored within the Webmail system. We issued an announcement alerting customers of our findings and continued with our full scale investigation into the breach. At that stage we decided to leave Webmail running, but took the decision to lock down logins to Webmail and our website to UK only IP addresses to reduce the risk of speculative follow-up attacks. At this stage, we had engineers on site in London who were performing integrity checks and server forensics across the entire platform. We believed that one server (WM04) had been compromised using a vulnerability in part of the Atmail code, which was coupled with some security-related weaknesses on our own server configuration. Once the affected server had been recovered for further forensic investigation, we found a malicious file disguised to look like an image file. This image file contained PHP code that allowed an attacker to run commands on our webmail server. Using this code they were able to transmit tables from the Webmail database to a remote location, and make changes to other HTML files so that visitors were presented a pop-up page containing a Trojan. After a full sweep we also found other files of a similar nature dated from the 4th May 2007 on WM04. The attacker gained information from the Webmail database from several tables which included customer email addresses, entries in customer address books, email addresses for mail sent by customers via Webmail, email addresses for mail received by customers via Webmail. Additionally when we first implemented the Atmail solution in 2004, we imported customers email addresses contact details into the Webmail database from our main databases. We did this to make the solution more convenient for users by setting up their email account for them initially. Customers who joined us after this time only had data created in the Webmail database the first time they logged in. We advised Calacode, authors of Atmail of the incident and they provided excellent support and have since published hotfixes to the code. The incident was confined to the Webmail database and personal data including payment details were not obtained. Our Networks team worked through the night on Monday/Tuesday performing further security checks and server hardening where necessary. This included turning off Website Wizard and our WiFi portal applications, which were not running the latest version of PHP and for which a quick upgrade was not possible. Overnight we also performed penetration testing (Using Nessus and other tools) across our entire platform. In the afternoon of Tuesday 15th May we emailed those broadband customers who were exhibiting signs of being infected by a Trojan virus. We sent an additional email on Wednesday 16th May to dial-up customers who may have logged into WM04. Shortly after this we made a detailed Service Status to customers to explain the progress we had made in our investigation. At this time we also felt we had enough evidence of malicious intent to notify the Police, and an incident was raised.
d. Wednesday 16th May - 21st May
You must be a registered user to add a comment here. If you've already registered, please log in. If you haven't registered yet, please register and log in.