Datacentre Outage - NEW

[#1]

At approximately 01:00am this morning our primary datacentre suffered a complete power outage. This meant that most of our internal systems, portals, and email were unavailable. Our network engineers worked to restore the primary services, and internal systems and portal access are now available. We are continuing to experience problems with email services, and are working on restoring mail access as our priority. We will provide a further update as soon as possible, and would like to thank you for your patience whilst we resolve these issues. Kind Regards, Mand Beckett Customer Support Please note that we don't usually post service status notices on to the Community Site, however as the service status feed isn't updating correctly to reflect the problems we are posting this here. Any further updates will be made on our usual service status post which can be found here:- http://usertools.plus.net/status/archive/ As the service status threads are working correctly this post has been removed from the front page of the Community Site.

[#2]

Thanks for the report Bob. I've certainly been in the position of having to "think on my feet" during a rollout and I extend some sympathy to Josh. I think the important thing to remember is that a mistake was made, has been identified, and processes have been altered to prevent a reoccurance of a similar problem. Obviously it would have been better to prevent the problem in the first place. However, I'm firmly of the belief that we are all human, and mistakes are inevitable. I applaud the openness that Plusnet have shown throughout the incident, and the openness they now show in publishing the incident report for everyone to see and critique. I hope lessons will be learned. Barry

[#3]

Thank you for a very open report - it certainly explains the mail behaviour I have seen. I trust Josh will not be overly censured - the man who never made a mistake...never made an improvement either....and is probably a liar as well! Best wishes......Trevor

[#4]

The trouble is that these types of incident keep happening with PlusNet. The comments on 'The Register' are of the 'laughing stock' variety and, given the history, that's hardly surprising. Forgive my cynicism but I detect 'weasel words' in the penultimate paragraph. The sort of thing that we have come to expect from politicians. "Unfortunately due to the nature of the error and the volume of mail involved, it is genuinely impossible for us to do this." Which is it, the nature of the error or the volume? Should we infer that had the volume been smaller recovery would have been possible and, if so, why was it impossible with the actual volume? "Customers should be sure that were there a practical way for us to achieve this, we would have gone to any lengths to make it so." What is the significance of the word 'practical' here? Are we to infer that there was a way 'to achieve this' but it was not 'practical'.... in other words too great a hassle? I note that in one thread a member of staff at PlusNet points to an external e-mail provider. Is it PlusNet policy to advise customers to seek their e-mail services elsewhere? Chris

[#5]

Thanks Chris. We all share a frustration about this happening and would agree it's unsurprising that the register has used it as an opportunity to have a dig. This incident reset the clock for us, and now we can only go back and build up another period of stability... The report is very honest, and I'd say the opposite of weasel words have been used. With technology, most things are generally possible, but not everything is practical. We receive 70 emails a second, and that sheer volume means we can't log all of the details of every email and what happened to it. For these purposes, we could perhaps have found a way to send an email to the sender of every from address that hit our platform on the 22nd August, but we have no way of knowing whether that was a faked spam address or whether the mail itself was actually dropped as a result of the problem. Even if that were appropriate solution (which it isn't, because the vast majority of the email addresses there would be faked, mailing lists, or otherwise unhelpful), you would be talking about sending 10s of Millions of emails, which is impractical to say the least. I hope that explains why we didn't just say it was impossible though! We don't have a policy on what services staff recommend to customers. We expect staff to be open and honest with customers, and to help them as much as possible on that basis. If it was appropriate for a customers needs to use a different email service, I don't think anyone here should get into trouble for telling a customer that. Ian

[#6]

I am very encouraged by the honesty of this report. Anyone in the IT world knows how difficult it is to stick to procedures when involved a 'live' incident. It is in stark contrast to the frustrating and worrying secrecy of Onetel, who obviously had a problem with their email service for over 10 days, but wouldn't give any explanation or prognosis either at the time ar afterwards. Keep up the honest approach - it really boosts my confidence in PlusNet.

[#7]

The first I heard of this loss of emails was today when I received the PlusNet Newsletter. It may explain why I never received notification that an item I had ordered online was out of stock, so I wasted time chasing it up later. Perhaps, as soon as the issue was resolved, you could have sent all customers an email warning us that emails which should have been received at PlusNet between x and y hours may have been lost. I would certainly have found it helpful.