Firstly guys, apologies for the hassle this is causing you all. The cause of the problem and the resulting issues it created meant that this wasn't a simple case of failing between data centres. In fact a number of services were failed over following the first outage (yesterday afternoon), however there was database corruption and other such issues which meant that manual intervention was required to get some things back up and running.
To my knowledge we're talking about a localised power failure combined with a faulty contactor at one of the two Sheffield data centres we have, however I will confirm this once I have the full picture. We're still running off back-up power and part of the problem recovery is dependent on replacement hardware that isn't expected until later on today.
There are still a number of residual problems we're working to resolve. I'm personally aware of the following customer-facing elements...

• Service Status/Usertools RSS feed is broken
  
• Access to both new and old Webmail platforms is problematic and the latter may return database errors when trying to log in
  
• The Community Site may be slow to respond or time-out when trying to log in
  
• Webstats hasn't run for yesterday
  
• Missing pictures/problems uploading to Community
  
• Inbound email delays
  
• Faz-2-Email is down
  
• Delayed billing for anybody due to have a payment taken/processed today
  

.
If anybody is having persistent/intermittent problems with any other services as of 10.00am this morning then please do shout up so I can make sure nothing is being overlooked.

Quote from: kdiment

In my innocence I thought that if Sheffield went down then services would still be available from London, and vice versa, and that was the main advantage of operating from two widely-spaced locations. Clearly, if that was the intention, then it doesn't work. Or are there major failures at both sites at the same time?

As Colin's explained, most services are split across two separate data centres in Sheffield. Failover did take place but the circumstances surrounding the problem have led to it being far more intrusive (more details to follow).

Quote from: Tigger

Well, according to the service status page, which I can now access again, there's nothing wrong!

I'm going to get something posted there shortly now we're able to.

Quote from: Routefinder

Was great to see Adam the off duty PlusNet guy to post............................but hang on mo, what about those still working in the hothouse.  As far as I can tell no one has posted to explain officially what the heck has gone wrong and when a full resolution will be achieved

Adam was on call as it happens so wasn't technically off-duty as such. As mentioned above, I'll provide further details about what happened in due course...

Quote from: dogday911

I did phone the support centre earlier this afternoon and was told that there were system problems.  But I'd rather not continually phone them, wouldn't it be better to have some sort of notice posted about the problem.

Due to the nature of the problem we were unable to post a portal announcement at the time (although I will be doing shortly). If you use Twitter then I'd suggest following us there as we posted a few announcements via that medium during the thick of the problem.

Yet again the need for a status site where postings can be made independently of the portal and the Sheffield data centres has been proved!
Usertools has a widget which shows the status from the member centre. If a website was set up in London which had two panels, one showing the status panel and a second which had a place for urgent messages when the portal or Sheffield data centres were down it would not need any updating when everything was normal.

Quote from: Bob

Service Status/Usertools RSS feed is broken

That should be back... if you could simulate these problems again at some point when I'm actually available to take a look at what it's doing (rather than at 5AM :P) then I'll fix it so it doesn't break in this way again...
To be honest, I think as soon as someone posted a status message, it would have automatically fixed itself anyway.
Colin

Still unable to log-in to webmail here, in case that helps!!?

I'm getting timeout and failure to load after entering login details on the "old system"
The beta system I can login to (very very slowly) and get the mail page, but that then times out having failed to load anything into the page framework
.

My access to my Webmail is STILL so slow and unreliable as to be unusable.
As others here have stated, I am sure the PN staff are doing their best to sort things out and I, too, have no interest in kicking people when they are down. However, thinking in such terms is inappropriate. Why do I state "inappropriate"? I do so because the key issue here is, is PN an amateur or a professional outfit? If amateur, then offering our patience and sympathy while they make "best effort" is appropriate. If professional, then the fact that they are “doing their best” is not good enough and, in fact, irrelevant. A professional organisation would have done extensive Risk Management: multiple “what if” scenarios that should, undoubtedly (since it has actually happened) have included the situation they now find themselves in.

Plusnet Home webpage responding much quicker again.
Webmail is pretty hopeless, will not load, keeps giving server errors and contact service administrator messages!
On the plus side msn works perfectly well.

Quote from: cso

Quote from: Bob Service Status/Usertools RSS feed is broken That should be back... if you could simulate these problems again at some point when I'm actually available to take a look at what it's doing (rather than at 5AM :P) then I'll fix it so it doesn't break in this way again...

Thanks a lot for that Colin!
I've just posted the following to Service Status (although obviously you can now ignore the bit about our RSS feed!
[quote author="Bob Pullen"]Access to Sheffield-based services/Data Centre Power Outage (69729) - NEW
Yesterday afternoon at approximately 2.30pm we suffered a problem at one of our Sheffield data centres that was triggered by a regional power failure.
Whilst our systems are typically designed to deal with situations like this, the circumstances of this particular problem caused significant disruption.
Many customers will have had problems accessing the services we host in Sheffield. This includes, but is not limited to, our internal systems, customer email collection, customer hosted websites, access to Webmail, access to our main website and access to our Community Site and discussion forums.
Until now, the problem has also prevented us from being able to post service announcements via the usual channels. We have however been trying our best to keep customers up to date via Twitter (@plusnet) and our discussion forums.
Power was restored early evening yesterday, however due to the nature of the problem manual intervention is required to get a lot of services back up and running. This has been further compounded by a second outage that occurred at approximately 1.30am this morning with similar consequences.
We've managed to partially recover from this second outage although there are still some residual problems we're working towards fixing. Part of the problem recovery is also dependent on replacement hardware that we're not expecting until later on today.
At the time of writing, we're aware of potential issues affecting the following services (although this list may not be exhaustive).
- Slow/Problematic access to both our new and old Webmail platforms
- Inbound email delays affecting the delivery of messages to customers mailboxes
- Slow/Problematic access to our Community Site and Discussion Forums
- Missing customer Webstats
- Delayed billing for anybody due to have a payment taken/processed today
- Access to our Fax-2-Email service
- Access to our Service Status RSS feed
Please be mindful that this problem does *not* affect broadband connectivity or performance so if you have a problem with your actual connection then you should raise it via the normal channels.
We would like to apologise for the inconvenience that's being caused and will post another update later today advising of progress.
Kind regards,
Bob Pullen
Customer Support

I *think* I've fixed the problem with the RSS feed going blank permanently, as I found it was doing it again (and tracked down the cause - which I've passed on to Bob) so hopefully this is now fixed.

Another problem.
Attempting to attach a picture to a post gives this error
which states
An Error Has Occurred! Cannot access attachments upload path!

Quote from: PlusComUK

For me... Webmail still not working 13:45 Tuesday 3rd. Log-in page loads very slowly. Logging in doesn't work, never loads

The service is up but it's running on reduced hardware and is under a lot of load at present.

Quote from: Oldjim

Another problem. Attempting to attach a picture to a post gives this error which states An Error Has Occurred! Cannot access attachments upload path!

Thanks Jim, that one should now be fixed.

Ok Back from the Hospital so I see that:
Webmail
https://webmail.plus.net/ - Still off line
Pay As You Host
https://payh.plus.net/ - Is Now Back Up
Dyslexia Health Education Association Forum
http://www.dhea-forum.org.uk/ - Is Now Back Up 
PlusNet's main site
https://www.plus.net/ ; - Is Now Back Up
So it looks like it is just Webmail still Off Line

It isn't offline entirely Gary, but yes it's probably one of the main services that remains problematic.
Edit: How are people finding access to Community now?