topic Re: Can Plusnet's internal systems cope with the increased users? in Plusnet Feedback

Can Plusnet's internal systems cope with the increased users?

jelv — Wed, 02 Feb 2011 17:35:07 GMT

It is become apparent to me that some of Plusnet's internal systems are running right on the edge. When problems occur and there is a delay in processing data it takes some time to catch up which implies there is very little spare capacity in the systems:
Evidence:
[quote=http://usertools.plus.net/status/archive/1296667090.htm]Broadband Usage Reporting (65301) - UPDATE
Portal
Posted on: Wednesday 2 February 2011, 17:18
Further to yesterday's post, our engineers are still working to process the backlog of usage data. For certain account types, there is no usage showing after 10am on the 31st January. This usage has been collected but not yet processed.
We even saw that some broadband usage data was being ditched.
[quote=http://usertools.plus.net/status/archive/1296642987.htm]Our billing systems are running a little behind at the moment, but that's simply due to the number of accounts the system needs to process towards the beginning of each month.
While we have not seen delayed billing, this statement indicates to me that the system is at full stretch on some days of the month and billing on those days takes longer than is desirable.

Processing call data for home phone: we often see posts about this being behind and it takes a while to catch up.
Is Plusnet really ready to handle all the new users it is attracting?

Re: Can Plusnet's internal systems cope with the increased users?

TicnTac — Wed, 02 Feb 2011 20:09:41 GMT

Quick answer = No.
I don't think it'll be too long before we see a major collapse of a few internal systems and traffic management systems. However, I also believe that we'll see a wave of people leaving which will then help things settle till the next daft marketing drive.
......it is very sad to see Plustnet slowly falling apart in public.

Re: Can Plusnet's internal systems cope with the increased users?

adamwalker — Thu, 03 Feb 2011 09:20:37 GMT

Whilst I can't go over a post mortem in detail of what caused the afore mentioned issues, we can say with confidence that they were not caused by capacity problems.
As part of our business roadmap we are looking at redesigning our platform of internal systems this year to avoid both issues like this and enable us to expand in a scalable fashion so that we don't have issues with capacity on any level.

Re: Can Plusnet's internal systems cope with the increased users?

davidj66 — Fri, 04 Feb 2011 09:22:23 GMT

Quote from: _Adam_Walker_

As part of our business roadmap we are looking at redesigning our platform of internal systems this year to avoid both issues like this and enable us to expand in a scalable fashion so that we don't have issues with capacity on any level.

Think that statement is definitely up for the "plain English! award 2011!!

Re: Can Plusnet's internal systems cope with the increased users?

adamwalker — Fri, 04 Feb 2011 10:02:26 GMT

Quote
As part of our business roadmap we are looking at redesigning our platform of internal systems this year to avoid both issues like this and enable us to expand in a scalable fashion so that we don't have issues with capacity on any level.

Translation: As part of our future plans we will be redesigning our server platform to avoid us going bang as we take on more new customers.

Re: Can Plusnet's internal systems cope with the increased users?

jelv — Fri, 04 Feb 2011 10:05:34 GMT

Quote from: _Adam_Walker_
Whilst I can't go over a post mortem in detail of what caused the afore mentioned issues, we can say with confidence that they were not caused by capacity problems.

Really?
[quote=http://usertools.plus.net/status/archive/1296812434.htm]Broadband Usage Reporting (65301) - UPDATE
Portal
Posted on: Friday 4 February 2011, 09:40
Further to Wednesday afternoon's post, our engineers carried out some maintenance in the early hours of this morning which we hope will solve the recent usage reporting problems we have seen.
We have discarded most of the usage between 5pm on Monday 31st up until yesterday. We are still collecting usage since this time and it will now continue to process sporadically. We're hoping the work this morning will mean that we start to see usage processed in the usual time by the end of the day.
Perhaps you could enlighten us with the actual reason for the above.

Re: Can Plusnet's internal systems cope with the increased users?

adamwalker — Fri, 04 Feb 2011 10:23:58 GMT

We've had some hardware issues which weren't capacity related, beyond that see the the first of the two comments you've quoted.
We still have some work to do on this but so far things are more stable, things are catching up and we're seeing improvements.

Re: Can Plusnet's internal systems cope with the increased users?

avatastic — Fri, 04 Feb 2011 10:25:26 GMT

Just migrate all the systems to the cloud. That never fails

Re: Can Plusnet's internal systems cope with the increased users?

davidj66 — Sat, 05 Feb 2011 12:04:13 GMT

Quote from: avatastic
Just migrate all the systems to the cloud. That never fails

Ah ....blue sky thinking

Re: Can Plusnet's internal systems cope with the increased users?

fourfourdevon — Sat, 05 Feb 2011 12:48:05 GMT

It is clear to me that Plusnet are suffering from a number of problems related to growth.
And even a hardware failure cannot be discounted as either caused or unearthed by growth, for instance a HD with faults, but which has low utilisation may never show a fault in its life time, but you can be sure that one that has a high utilisation will.
But its more than the missing broadband data, its the recent speed drop offs many have suffered from, its the inability to respond to tickets in a timely fashion, sending out emails demanding updated Credit Card details when the person pays by DD anyway.
None of these things are necessarily capacity problems as such, it may well be that the hardware has to capacity to cope with 10 times the volume, but systematic errors are strange beasts, lets say 10 customers on average generate one 1 problem per year requiring a Customer Service person to spend 5 hours researching and fixing. You might think, so 10,000 customers will generate 5,000 hours of work for Customer Service reps and all you need is to divide that by the average working hours of one person per year (5,000 / (37.5 * 52) ~= 2.5) but it doesn't work that way, errors and problems don't arise at a steady rate, they arise at a fluctuating rate and you have to be able to cope with not the average demand, but peak demand, which could be many times larger than average demand.
The upshot of this phenomena is that business normally manage down the quality of service, and allow, happily, for say 3% of their customers to walk away having had an appalling service in return for keeping the other 97% happy enough that they manage to grow their customer base by 20% year on year.
You only have to look at mobile phone providers, Sky, Virgin Media and oh so many more business's to see this in action. Plusnet are increasingly unlikely to win any my customer service awards because they can no longer (due to growth of the business) afford to provide the quality of service that they once did.

Re: Can Plusnet's internal systems cope with the increased users?

julesandsand — Sat, 05 Feb 2011 18:00:24 GMT

Quote from: _Adam_Walker_

As part of our business roadmap we are looking at redesigning our platform of internal systems this year to avoid both issues like this and enable us to expand in a scalable fashion so that we don't have issues with capacity on any level.

Sir Humphrey Appleby springs immediately to mind.

Re: Can Plusnet's internal systems cope with the increased users?

adamwalker — Mon, 07 Feb 2011 10:54:04 GMT

Ok when you've all done taking the mick
Its important to look at the problems we do have at the moment, separate them and look at the causes. Billing issues, call processing and Usage monitoring issues have unfortunately been impacting our customers. On the whole these issues have been due to hardware issues which we are in the process of over coming at the moment.
Call processing is up to date, usage is still an issue but is being worked on and we don't currently have any open high priority issues with billing at the moment. None of these issues were directly related to each other in any way or to the fact that we are expanding our customer base.

Re: Can Plusnet's internal systems cope with the increased users?

jelv — Mon, 07 Feb 2011 11:17:49 GMT

Has the time taken to recover from each of these separate incidents been longer due to the increased user numbers?

Re: Can Plusnet's internal systems cope with the increased users?

adamwalker — Mon, 07 Feb 2011 12:57:19 GMT

No, the numbers of customer bear no relation to this and don't reflect on how long it took to resolve these problems.

Re: Can Plusnet's internal systems cope with the increased users?

avatastic — Mon, 07 Feb 2011 13:10:12 GMT

The extra delays were because nobody noticed the 'Turbo' button was unpressed on the 386 that does the data runs

Re: Can Plusnet's internal systems cope with the increased users?

gswindale — Tue, 08 Feb 2011 12:35:16 GMT

Surely that should make it quicker.
From memory; you pressed the "turbo" button in when you needed to slow the machine down to cope with older software.

Re: Can Plusnet's internal systems cope with the increased users?

davidj66 — Wed, 09 Feb 2011 10:09:44 GMT

More likely to be the 1MB of RAM - Told them they needed to upgrade to 2MB!!!!