I was using the system at 01:30 the only thing  I could not do was access the portal and e-mail.  I could still access the internet. 

What we really need to know have we lost any e-mail messge when the problem happened?

puddy

Plusnet are to be commended for their swift action by getting an "away team" to the 3rd-party data centre to deal with the "Human error"

I'm surprised however that "contractors" can do work around critical systems without consulting data centre staff who would or should know the implications of the work.

That issue has been raised by PlusNet with the data centre management.

Funny you should say that ... I had exactly the same thought myself at 0100 when I suddenly found mail/portal wasn't playing ball so I sent various test messages during the outage, well until around 0330 anyway.  The good news is that they all appeared when I checked this morning

I'm not suggesting that this is definitive proof of no problems at all but it seems most unlikely to me that any external mail was lost, at least for those using postini.  I couldn't imagine it being any different for non-postini users so I didn't bother to check sending mail direct to PN.  Must say that I was expecting problems as postini allegedly doesn't store messages that it can't forward immediately so I was VERY pleasantly surprised when they all dropped into my test mbox as if by magic this morning !  The only 'delay' was due to the fact that I couldn't access the mbox during the outage, there was no delay in delivering the messages to PN.

Translation - As usual Plusnet have engaged a cheapskate cowboy data centre so badly and unprofessionally managed that they can in the first place ever let the all too predicatble night-time cowboy contractors they have employed to do some dodgy on the cheap electrical work to shut everything down, presumably because only the night security guard (who has not a clue about how computers work) is actually on duty.  Even if this Big Red Off switch was initially pressed in error then surely the senior ops management at the Data Centre would immediately have realised the consequences of the disastrous error and pressed the Big Green On button to restore power within a few minutes.  Having a Big Green On button is just as important as a Big Red Off button, precisely because some low skilled British workman with an IQ of about 62 might very well press the Big Red Off button when he shouldn't to suit his own inconvenience and narrow objectives whilst not giving a damn for the main responsibilities of the data centre.

You are clearly lieing to us by saying that the outage was only 4 hours as I became unable to pick up email around 1am and had crucial email that had been sent by someone who had then gone to bed for a business meeting the following morning.  The email was not available at 1am or 2am and your servers were refusing to authentical mail delivery via Pop and there was also no access via the web portal.  This remained the case at 9.30am in the morning when I had to leave for the meeting.  The email finally came back on its own without me re-entering any passwords.

I telephoned and spoke to one of your call centre technical support staff at 1.30am.  As usual there was the typical complacence, indifference and arrogance that has always characterised Plusnet telephone technical support staff who then gave the impression this was a trivial problem, was under control and all systems would be back no later than 3am.  Whereas the reality was that these staff were so complacent that they didn't think a major ISP having its email down for 8 or 12 hours mattered and that it was more important for on call staff to be allowed to have their beauty sleep.

One hesitates to ask the other obvious questions such as where is your second or even third data centres that you cut over to routing your email through in the event of a failure in the first.  What happens if the first is flooded or hit by a plane crash?  Does that mean no Plusnet email for a week?

I have been told by the current MD of Entanet reseller CCS Leeds that ensuring 24/7 resilience in pop and smtp email is a simple matter and that he has engineered a number of setups that have made this possible in all circumstances.  Any company like Plusnet that frequently lets all email go off for many hours at a time does so because it doesn't think its customers have the same importance as a major business customer and therefore hasn't bothered to invest adequately in sufficiently resilient alternative backup systems

Conclusions:-

Plusnet - a bunch of amateurs

Data Centre Company - managed by a bunch of incompetent cowboys

Lack of Second Data Centre in Event of Disaster Hitting The First - Unbelievable for an ISP of this size

Summary - As ever Plusnet prove that they are not a well managed ISP and that everyone in the company passes the buck to someone else as to why a disaster of this kind happens and the well paid directors of the company, who took the policy decisions that led to this situation, as usual apply the Mr Teflon approach by trying to evade all responisbility for their own incompetent management of the business.

Erm how did you come to that conculsion? 

From what I have read  PN had no idea about the contractors working at the 3rd party Data Centre and when they found out what was going on they sent in their network staff to deal with the situation.

The incident happened in the early hours of the morning and PN had it more or less sorted in about 4 hours. It could have been worse

Your post was quite long, did you suffer some major loss during the outage?

For all you know the "cheapskate"  data centre you quoted  could run by your beloved  Entanet ? (imagine if that was the case)


Plusnet's honesty sometimes makes them appear to look amateurish,  *All* ISP's have some operational difficulties at some time not all will admit it.

The facility that is being discussed here is a 'lights-out' facility in that it is not manned 24x7 although access is always available. The owners of this facility are Sky (although it was Easynet when we moved in), and I'd therefore suggest your 'cheapskate cowboy data centre' is complete rubbish. We are still awaiting the full incident report from them as this can take 10 days which is standard in cases like this are there are contractual and legal issues to cover. We have had a verbal account of the events but before we can comment we need the formal written report.


What would we gain by lieing [sic]? If we were going to do that we'd just say nothing at all, which is just not what we are about. We've also got over 200k customers who would be quite able to see if the events were not as stated. It's no secret, and there are Service Status posts detailing some intermittent issues with e-mail during the morning, however this was caused by the servers coming back before the network and therefore not mounting their storage correctly. This is one thing we have taken from the incident and are looking at changes to the load balancer to stop that happening again.


I've just looked at the audit trail on you account and the first time it was accessed was at 07:56 on the 21st?


About half of my team were in overnight recovering from this incident. I believe all services were restored very quickly, and within the timescales that you quote with the exception of e-mail collection which although the servers were restored there was an issue with them. The reason that we didn't get any more staff up is that we needed to cover during the day too if more issues were encountered.


Absolutely not! no e-mail was lost during this incident as simple things like e-mail delivery automatically fail-over. The things that need manual intervention are things which connect to the database. We do this because if there was automatic failover there is the potential for data loss and integrity issues and we want someone to have made the decision to failover. Take for example the scenario where our data centres become unable to communicate with each other, with automatic failover the servers in one site would fail to their local database and the servers on the over would fail to theirs this would lead to a complete mess of data being written to each database and us not knowing which was accurate.


The proof is in the pudding as they say and I'm sure they will have tested that in all circumstances. As for you comments about frequently letting e-mail go off for many hours at a time, I'll accept that it has happened more than anyone would like in the past 12 months, but I don't think it's frequent. In terms of investment, we have made significant investment in our infrastructure over the past 12 months with a large proportion of that on e-mail.


If by passing the buck, you mean reporting on the events to our customers then we're certainly guilt of that. Ultimately I am responsible for the team that support the infrastructure our facilities, and am therefore responsible for the service that we offer to all of our customers. Does an event like this cause me to review all of that, yes of course it does but in this case it was an event that could not have been planned for and is one that I've seen many times in data centres from the major ones in London to ones owned by the high street banks. These things do happen, and in my mind it's as important how you deal with them as to how it happened, and in this case my team did an outstanding job in restoring both service and servers in the affected data centre.

Phil

Phil,
Thank you for an excellent reply to the "complainant".

My comment on his "diatribe" is this..... if you are that unhappy with the service that you consider your are (not) getting, then respectfully "go and get on another bus".

Problems happen with all kinds of industry, and I am sure in your own personal life, you have problems, and need time to get them sorted out. 

If you think that you have "all the answers" before a problem has manifested itself, then perhaps you had better start your own ISP, and see how many satisfied customers you can support.

You may have been incovenienced by the loss of service, in particular, email, however, there are alternative means of communication, i.e. telephone, for verbal communication, and FAX services for paper type communication.  Both of these could have been used in your situation if you needed to contact someone (who you already admit had gone to bed) before the meeting the next day. 

To sum up..... I think your "diatribe" was way over the top, and probably caused a lot of ill feeling amongst the people concerned who were working very hard to restore the service.

also if you are going to "Stretch the incredulus" its a shame to get caught out

pwebb is to be congratulated for his generosity in giving Capvermell's diatribe so full an answer.  I would have been tempted to close the account until he gets past his teenage years.

I was amazed at Capvermell's diatribe, after venting his spleen about almost every aspect of PN's operations he is caught out by his own audit trail
(not bad for a bunch of amateurs as he puts it)


I'm sure he will be more careful about letting his imagination get the better of him in future.

The only comment I have is on the Sky/Easynet DC which you have kit in... if the DC is anything like Sky or Easynet's core network (which I suspect it may even be part of) I wouldnt touch it with a bargepole. Just my two cents... lets face it.. Sky are *not* a comms company, and therefore have no place in running a datacenter...

But I reckon PN are to be commended on the speed and hardworkingness (sorry, I cant remember the proper word for it) of their employees! Always going the extra mile, thats what I like to see

I had to attend a meeting of the Leasehold Valuation Tribunal the following day and was trying to see if a fellow leaseholder here had responded to my earlier emails and would be becoming a co-respondent.  Waking them up at 1am was only likely to achieve the desired result.

I could not collect Plusnet email just after 1am and my recollection is of calling Plusnet to enquire about the failure before I went to bed some time after 3am.  I recall speaking to one of your staff before I went to bed just after 3am.

As I post under the name Capvermell and do not have a Plusnet broadband connection or IP address then how do you know which Plusnet email account I am?  Presumably only from the content of one of the email headers I have previously posted?

As regards your data centre being a Sky/Easynet one as another poster commented Sky are not a serious player in the internet market and are focused on offering a cheap service to bolster satellite subscription.  Thus I would not expect anyone who needs a resilient service like a high street bank to be using them.

At the end of the day there are companies who are focused on things not failing and those where there are well intentioned individuals (as you no doubt yourself are) but an overall corporate process so chaotic that disaster repeatedly ensues.

The reason I am with an Entanet ISP for broadband is because their service is always up and does not go down overnight and their SMTP outbound server also never goes down for more than a handful of minutes and allows me to send email when temporarily using broadband connections of other ISPs.  No doubt I would be a Plusnet broadband customer now if I had not long ago lost confidence in Plusnet's technical reliability.

As to Plusnet email my continued use of it comes down to inertia and the fact that on the whole in the last 6 months with spam filtering etc and no long multi hour pop downloading failures things had been much better.  But if we look at 6 to 24 months ago Plusnet had a truly atrocious record for repeat mult hour failures of its Pop email servers and even wiping out customer emails.  Plusnet were getting better up to a couple of days ago on email but only after a previous truly atrocious service.  And let us not forget that the reason we were all being heavily spammed was again down to Plusnet incompetence.

At the end of the day if Plusnet is serious about its operations then it will use the kind of data centre a bank, stockbroker or Easyjet, who rely on mission critical 24 hour operation, would use and would have hardware and databases that would immediately failover to a second data centre in the events of a problem at the first.

The fact is that customers could not get email for at least 8 hours and much though Plusnet may mutter about how most equipment was back up after 4 hours if it wasn't back up in a condition where customers could get their email then it wasn't really much use.

I notice that this forum seems to mainly contain Plusnet devotees who seem to worship the company despite its many longstanding failings in technical resilience and I will take that in to account when considering if it is worth making any further posts in this forum.

Moderators Note.

I have split the off topic posts from this thread to Update on this morning's outage [Off Topic]

Please can I ask the we keep this thread on its original topic.

Jonathan
aka Chilly

I now recall that when I phoned Plusnet around 2am I did not get the member of staff who took the call to log in to my user account as I merely asked what the outage was and when it would be fixed.  They said by 3am at the latest.

Only later when calling again after 7am and the problem was still not fixed was I asked for my username and password and log in to my account.  That is why it appears to the gentleman from Plusnet as though I did not ask for an update until the morning.

1. You are utilising a FREE email service on the basis that "my continued use of it comes down to inertia and the fact that on the whole in the last 6 months with spam filtering etc and no long multi hour pop downloading failures things had been much better".

2. No emails were lost.

3. There was a period of around 8 hours where emails were unable to be collected - you were very unfortunate that it affected you. It appears that some people were "affected" for longer simply because they did not realise they needed to re-enter their password as their mail client had discarded it when login failed (that had me going for a bit).

It was exceptional circumstances. In my books a measure of a company is far more down to how problems are resolved. In this case staff were mobilised very rapidly for a stupid o'clock problem and everything brought back on-line in remarkably short time. I must admit that I was surprised by the total lack of synchronisation problems between the two sites after it came back given the nature of the problem.