Update on this morning's outage

[#1]

@Jelv
If that was aimed at me then I am not sure  what you mean.
When a UPS kicks in, my understanding is that it provides output to a unit (computer) and isolates it from the mains. No power would have got back to the building's mains supply.

[#2]

Tony
If they have to work on a completely isolated building or system, they will most certainly NOT want any voltage floating about regardless of the source..
Anyway, we could be having a post mortem about this for weeks on end, just like the last email problem, so I politely suggest that we stop any conspiracy theories on this.. however, tin foil hats and black helicopters are available on request. 

[#3]

Can I book one of the hats - these rays are getting to me?

[#4]

Sorry but I still don't understand how the ups didn't keep your servers running until they shutdown cleanly.I have a £15 ups which closes my computer down in the event of a power-outage without any problems.

[#5]

donj53 and everyone else - from what I understand, what happened with PlusNet is more akin to you pulling the power cable out of the back of your PC, rather than you experiencing a power cut. No matter how good a UPS is, it won't provide help in that scenario.

[#6]

Quote from: donj53

Sorry but I still don't understand how the ups didn't keep your servers running until they shutdown cleanly.I have a £15 ups which closes my computer down in the event of a power-outage without any problems.

Because, as already stated, the contractors effectively pressed the Emergency Power Off (EPO) button. Most datacentres have large UPS's which supply PDUs in the server cabinets. The EPO button effectively switches off these UPS's. Most datacentres do not have individual UPS's for each individual server these days, mainly due to the additional environmental impact these have.

[#7]

Perhaps I can clarify the situation in relation to my experience in an industrial environment.
In a scenario where there is any potential risk to the staff due to powered equipment it is mandatory that a total shutdown capability is provided. In my industry this was referred to as the full emergency stop and was normally a large red button. If this was hit it took all the power off regardless as to whether it would potentially cause equipment problems or damage. This was different to a controlled emergency stop where the system remained powered but stopped as quickly as possible. Interestingly the controlled emergency stop normally stopped things quicker than the full emergency stop where all power was removed and the brakes  automatically applied.
In the case we are talking about the design is intended to remove all power from the equipment and hence make it safe if something has gone very badly wrong. The assumption is that this would only be used in a dire emergency and not if a poorly trained or inexperienced sub contractor thought it was a local isolation switch (my supposition - may not be correct)

[#8]

Just think for a minute - there's a real fire in a room full of computers and the fire brigade arrive. They kill the power and all the UPS's take over, so they have to delay tackling the fire while they go round each individual machine turning them off one by one, then they can start to tackle the fire. Realistic? No! There has to be a master emergency switch that kills everything instantly.
I thought Mark's post made it pretty clear that Plusnet are as unhappy as we all are and that there were going to be some "interesting" discussions between Plusnet and the management of the data centre, not least as to how there could be obviously planned work (contractors are not working on site at 1 am without it being planned) without Plusnet being notified. Another question I bet is being asked is how the contractors didn't know that it was a 24 hour operation and that they couldn't just turn the power off, and lastly how come contractors were not being supervised in a way that would have allowed someone to step in and prevent the "human error"?
The action taken by Plusnet to get things up and running again is to be applauded. The one area where they failed (which they have admitted) was the poor communication. The fact that if the main systems are down there is no way a service status announcement can be made is pathetic. Five years ago I was advocating that there needed to be a service status system which was totally independent of all other Plusnet systems - surely they could now put a server for this in another BT site which could be updated using by a dialup modem in an emergency.

[#9]

Quote from: jelv

[...] surely they could now put a server for this in another BT site which could be updated using by a dialup modem in an emergency.

That's what I always thought servicestatus.net was going to become. Maybe PN will take another look at that now they've experienced a scenario in which it would have been useful.

[#10]

Have you looked on there? The first mention on there was at 09:53:38!
http://www.servicestatus.net/index/

[#11]

Quote

The fact that if the main systems are down there is no way a service status announcement can be made is pathetic.

Totally agreed.

Quote

Maybe PN will take another look at that now they've experienced a scenario in which it would have been useful.

We most certainly will.

[#12]

Quote from: Be3G

That's what I always thought servicestatus.net was going to become. Maybe PN will take another look at that now they've experienced a scenario in which it would have been useful.

Already being looked at. The intention is to host the (currently in development) new Service Status tool at servicestatus.net.

[#13]

Isn't it ironic that when faced with connectivity issues we call PN to be greeted by a recorded message telling us to go online for help.
Whereas when PN have a problem, you can't find out online, you have to phone up instead.
Made me chuckle a bit anyways.

[#14]

While I understand the reason why hitting the Big Red Button caused the problem (Need to put a sign on that to say "This is not a Toaster, please don't turn it on and off")  , I thought that PlusNet had more than one Data Centre in case of a disaster?  - or have I got that wrong?

[#15]

They do - if you read Mark's original post in this thread, you'll see him mentioning it. The problem was that PlusNet weren't aware anything was going to happen, so they had no-one prepared to switch operations to that second data centre.