Plusnet outage

[#1]

Hi guys,
It was a bit of nasty problem really.  Some known hardware bugs caused a cascade of disconnections from the BNGs, i.e. the disconnections from one BNG caused a similar impact when lots of those customers attempted to reconnect on the next BNG.  We've worked out how to lesson the impact of another failure now so it shouldn't happen again, and we're working with the vendor to get the bugs fixed.    The older AGs were fine but just got very loaded with connections which lead to a bit of poorer performance on in the later evening, but the traffic management protected the prioritised traffic as expected.  It helped that it was a Friday evening so peak time traffic is usually lower.
The platform had balanced itself by Sunday so we didn't see any issues then when we'd reach the peak period.

[#2]

Quote from: Kelly

Some known hardware bugs caused ...

Kelly,
One just has to ask why if your vendor knew about this hardware deficiency they had not forewarned PN so that the infrastructure could be made defensive of the issue...

Quote from: Kelly

We've worked out how to lesson the impact of another failure now so it shouldn't happen again, and we're working with the vendor to get the bugs fixed.

...before the event / until the vendor got their fingers out to fix their kit.
Is this yet another illustration of PlusNet's several vendor management issues?  There seem to be too many areas where PlusNet's service provision fails as a direct consequence of poor vendor performance... and there is no visibility of effective PlusNET management of third party failure.  For clarity, here is a short list of the vendor related issues I recall from the forums over recent months...
1. BTw - ordering systems
2. BTw - speed test system
3. BTw - CN20 line management via CN21 tools
4. BTOR - appointment attendance and the rest of their sins
5. Load balancers
6. Telephony system stats reporting
7. BNG - hardware faults
I would guess that is just the tip of the iceberg?  What is being done by purchasing about these supplier failures?
Kevin

[#3]

Quote from: Townman

Quote from: Kelly Some known hardware bugs caused ... Kelly, One just has to ask why if your vendor knew about this hardware deficiency they had not forewarned PN so that the infrastructure could be made defensive of the issue... Quote from: Kelly We've worked out how to lesson the impact of another failure now so it shouldn't happen again, and we're working with the vendor to get the bugs fixed. ...before the event / until the vendor got their fingers out to fix their kit.

Hindsight is a wonderful thing sometimes
We were already working with them on it.  We didn't anticipate the impact of this failure.  It's not really fair to blame it on the vendor.

Quote from: Townman

Is this yet another illustration of PlusNet's several vendor management issues?  There seem to be too many areas where PlusNet's service provision fails as a direct consequence of poor vendor performance... and there is no visibility of effective PlusNET management of third party failure.  For clarity, here is a short list of the vendor related issues I recall from the forums over recent months... 1. BTw - ordering systems 2. BTw - speed test system 3. BTw - CN20 line management via CN21 tools 4. BTOR - appointment attendance and the rest of their sins 5. Load balancers 6. Telephony system stats reporting 7. BNG - hardware faults I would guess that is just the tip of the iceberg?  What is being done by purchasing about these supplier failures? Kevin

No it isn't.  All hardware and software has issues and we work with vendors to keep stuff patched and up to date as necessary.  It just so happens this one as a nasty one we didn't anticipate.

[#4]

Kelly,
After spending 35 years running IT systems, I know all too well that hardware and software all have flaws.  The point I sought to make - which you did not answer - is that PlusNET seem to have far too many vendor / supplier issues impacting the core of service provision.  If truth be known, of all of the big issues impacting your users (other than billing) I cannot think of any other which does not have a marked vendor or supplier contributing factor.  If PlusNET could get a better service from their suppliers right across the board, the service seen by PlusNET customers would be substantially better.  Surely you do not dispute this?
The big question is what is being done about such matters - both resolution and prevention?  What has been learnt from this issue?  Is PlusNET sufficiently "a head of the curve" with its suppliers or is it destined to remain "on the back foot and fix after failure" as per the web mail hack from a few years ago, which is still giving rise to sizeable volumes of spam mail?
To be fair, it was not clear in your explanation of this issue if the known hardware fault was known about by PlusNET prior to the event.  If it was, I think it is reasonable to expect that there was a rectification plan in place and a major event management plan to mitigate the consequences prior to rectification.
If the fault was known to the vendor and not PlusNET prior to the event, then this is indicative of a challenged relationship between vendor and user, for the vendor ought to have known the consequences of the issue and the likely impact on their customers.
I know that if I had been in possession of such information and had not shared it with my customers to ensure that they were informed and could consider mitigation plans, my ass would have been justifiably fried by the said customers (major government accounts) if things had gone wrong as a consequence.
Kevin

[#5]

I think Plusnet's relationship with BT Wholesale is quite different to that of their hardware suppliers like Juniper. If you ask some of the PN staff off the record about what they think of dealing with BT Wholesale, then you will probably have many colourful words used.
With their hardware vendors, I would imagine it's quite different. Plusnet have a choice of who they can buy hardware from etc. Juniper is one of the leading suppliers of hardware to ISPs in the world and the reason they would have got to this position is by offering very good hardware/support.
No ISP is immune to issues with hardware/firmware/software,  pretty much every ISP experiences this kind of thing at some point.

[#6]

Thanks for the updates guys.
Interesting technical reading! 

[#7]

Looks like all the users dropped of pcl-bng01 at a little before 14:00 today. It did make the service status, but I didn't notice many people complaining about it here.

[#8]

Yeah, all the users reconnected almost immediately.

[#9]

Plusnet servers seem to be down again.

[#10]

Temporarily locked to keep posts about the current outage here http://community.plus.net/forum/index.php/topic,131874.0.html

[#11]

Now unlocked but please try and use this thread for updates http://community.plus.net/forum/index.php/topic,131874.0.html