cancel
Showing results for 
Search instead for 
Did you mean: 

Traffic Management Update - 7th December

Community Gaffer
Community Gaffer
Posts: 12,810
Thanks: 639
Fixes: 62
Registered: 04-04-2007

Traffic Management Update - 7th December

This is an update to the previously reported problems related to our broadband network management system. A copy of the last announcement can be found here:-
http://usertools.plus.net/status/archive/1165333427.htm

We are continuing to investigate and work to resolve speed and performance issues, particularly during peak hours of the day, across certain traffic types. Some customers are reported slow speeds affecting all traffic types including web browsing.

These issues are caused by the bandwidth available on our platform being shared out disproportionately across customers. Usage has increased slightly on average, but across a small group of Premier and Business customers usage has increased massively since upgrades to up-to-8Mb speeds.

Whilst the changes made over the weekend to distribute priority web traffic were met with some reports of success, there are still many complaints of speed and performance issues affecting web browsing and other priority web-based Internet traffic. A knock on effect of this is that lower priority applications even for light users are being affected.

Diagnosing the sources of these issues has been extremely complex and our investigations continue for a number of reasons. Firstly, the experience does not seem to be consistent across each product tier. Whilst there are other factors that can affect speed and performance (such as exchange contention), there are cases where customers with identical account configurations are seeing large differences in performance.

Secondly, our data in some areas was inconsistent with what some users are reporting in our discussion forums. Changes we have made over recent days have not had the effect we anticipated and our graphing in places seemed to be at odds with what our customers are reporting.

Over recent days we have been contacting individual customers to help diagnose and resolve specific issues. Yesterday afternoon we contacted a number of users reporting substandard email performance. Initial diagnosis of the customers' problem revealed no discrepancies with the way their accounts or our systems were set up. As part of the investigation we moved the customers from their current profile to a new profile and then immediately reverted back to the original configuration. This instantly resolved the traffic issues they were reporting even though the configuration was the same as when originally checked. What that means is that even within identical profiles, customers were being treated differently by our platform.

This was obviously a cause for serious concern, so we took the decision to reload all the Ellacoya switches which was completed by 6am this morning. Analysis since we did this has returned a more consistent and accurate set of results. Our graphing is substantially different and we are now reporting increased usage in the silver queues and a decrease in gold traffic on the network. This suggests that some traffic was not being prioritised in the correct queues. Further investigation proved that changes to our Ellacoya database were not being loaded correctly into the switches. This is in part due to the increase in changes made over the last two weeks in an effort to resolve the issues customers were reporting. After speaking to senior engineers at Ellacoya, a problem has been identified that both ourselves and Ellacoya believe to be the cause of the issues we reported to them. We have a ticket open with Ellacoya and are awaiting further details regarding this matter.

This issue explains why the changes we were making were not benefiting customers immediately. It also explains why changes made that initially seemed to work became progressively worse. This in part has led to a lack of visibility in the customer forums as we have not been able to fully explain the issues customers were reporting.

Before the reloading of the switches all Usenet traffic on port 80 was being incorrectly marked in the gold queue. This was having a significant impact on all gold traffic and subsequently on silver & bronze traffic. This is now in the silver queue where it belongs.

Our graphing and network side monitoring is also now reporting accurately. This and the information from Ellacoya should help us further our investigations and assist in improving performance for customers. We will shortly be adding more graphing to the portal so that customers can see traffic across different gateways.

We are continuing to take action where a relatively small number of customers have a disproportionate amount of usage which is impacting other customers. We are sending MAC keys in extreme cases but in general, updating our automated and manual network implementation to redistribute their usage fairly across other customers.

We believe that the recently identified problem has impacted PAYG usage significantly in the past few weeks. Work to protect the peak time performance of customers who pay for their bandwidth per gigabyte will commence immediately and will be resolved next week. Light and average usage Premier and Business customers experiencing speeds issues caused by our platform will see progressive improvements through the weekend and next week.

Kind Regards,

Bob Pullen
Plusnet Products Team
If I've been helpful then please give thanks ⤵

214 REPLIES
N/A

Traffic Management Update - 7th December

Hi, well Bob I hope that something happens soon because for me it is worse now then ever. Once again I can't run speedtest.net as it will not load the page fully.
It's just taken me 2 minutes to even get into here, my downloads are running to the equivilent of a dialup, 6k's. As I've said in my posts before at present this service isn't fit to used as it should and something needs to change pretty quickly...
ChrisC
N/A

Traffic Management Update - 7th December

Why cant the latest changes be rolled back until you know what the problem is? I dont see why we should suffer due to PN's mistakes.
Community Veteran
Posts: 2,322
Registered: 01-08-2007

Traffic Management Update - 7th December

Quote
These issues are caused by the bandwidth available on our platform being shared out disproportionately across customers. Usage has increased slightly on average, but across a small group of Premier and Business customers usage has increased massively since upgrades to up-to-8Mb speeds.


This is getting as near the truth as possible I suppose. PN have not altered their network design to account for an increase in usage due to an increase in line speed. So capacity is an issue, just nobody at PN has the brass tacks to say it.

Sadly this is yet another round of tweaking the management, albeit some more serious tweaking.

How come limiting a minority of users, and issuing MAC's to a further minority will free up enough space for us all to breath easy, but extra capacity wont?
Firejack
Grafter
Posts: 921
Registered: 26-06-2007

Traffic Management Update - 7th December

While I commend the lengths Plus.net are going to make sure everyone gets their fair share of bandwidth which we all pay for. And I applaud Plus.net for issuing MAC's to those taking the piss and downloading their own personal copy of the Internet. When are Plus.net going to face the bottom line and admit they are out of bandwidth and need to invest in additional capacity.

I too pay my fair share. However when I log into my website via FTP to transfer updates that are only a few megabytes in total and the estimated transfer time is 20+ minutes. Why aren't I too allowed my fair share as the person watching Skateboard stunts on youtube? You can't prioritise one type of traffic over another. To customers every type of traffic they use is important.

Buy more bandwidth :shock:
Neil_A
Grafter
Posts: 450
Registered: 04-04-2007

Traffic Management Update - 7th December

Apologies for cross-posting, this was originally put in the 1st December update thread, but seems better to be here now:

----
Quote
chesterfield wrote:

But what IS the problem?


Sorry guys, I thought we'd been clear already on the the problem and the causes. But for the avoidance of doubt here goes Smiley

The problem
-----------
The problem is our network is full from 7am until about 3am. And it is over-full from about 3pm until 11pm. What happens then is that titanium squeezes gold, which squeezes silver, which squeezes bronze. The mainfestation of that is that silver has become very slow at peak times for many customers since the beginning of September and in recent weeks we've seen gold degrade at peak times.

The causes
----------
1) We buy bandwidth based on revenue contribution from customers not usage requirement. We've been clear on that for some time
http://www.plus.net/support/broadband/network/capacity_guide.shtml
Therefore if customer numbers don't increase we won't add bandwidth unless wholesale costs come down. And customer numbers have been flat since May so we haven't added any bandwidth. Many ISPs have gone bust or been forced to sell because they have kept adding bandwidth without the revenue to do so and we're not interested in going bust. So within the bandwidth we have we have to share that our fairly across all our customers according to how much they pay. That approach isn't going to change.

2) Average usage has been increasing across our customer base since mid August. The main reasons for this are customers upgrading to 8Mb services, shorter days associated with normal seasonality and the changing usage trends including more downloading of music & video and an increase in gaming. We have shared our data with PUG and will publish it here shortly for all customers to see and draw their own conclusions.

3) The highest usage customers across our products, predominantly on Premier and Business services, have significantly increased their usage in September and October. The platform was keeping this in check during the Summer, but we failed to keep our automated rules updated and did not do enough manual checking to protect other customers so that this has hit us much harder than we expected in September and October. We're in the process of sharing this information with PUG and will publish it here shortly for all customers to see and draw their own conclusions.

The solution
------------
So what are we doing about it?

*We've put in place a huge amount of new reporting so that we stay on top of this in the future.

*In the last three weeks in particular we have made a large number of changes to the platform to try and resolve the issues. Some of these have been successful but not as successful as we had hoped. Some of them have had no positive effect, some have made things worse. We need to make some more changes throughout this weekend and next week before we can be fully confident that we have fixed this and we will not close this issue until customers are satisfied with the experience they are receiving

*We have discovered an issue in our Ellacoya implementation that has caused problems on profiles, particularly for PAYG customers. A Service Status this afternoon which Bob has published gives more on this - basically it isn't the primary cause of the issue but it has certainly hasn't helped and indeed has stopped us solving it up to now.

*We also will get back to doing what we were doing in April-July, which was properly managing the very heaviest usage customers so that they cannot impact other's services. In extreme cases that involves MAC keys, but for most of those heavy customers it involves automated shaping so that they can decide whether to stay or go

We remain a referral business and will do so even with new owners. So delivering a poor experience to our customers is most certainly not in our interests!
N/A

Traffic Management Update - 7th December

Hi, interesting quote: "Therefore if customer numbers don't increase we won't add bandwidth unless wholesale costs come down. And customer numbers have been flat since May "

So it's which comes first, the chicken or the egg. If +net continue to put out a poor service then word will get around that their service is poor. Thus they will attract no customers and the ones thay have will leave, or are leaving anyway

Mmmm, isn't it a basic rule of commerce that you have to invest to grow?

It's not rocket science...

ChrisC
N/A

Traffic Management Update - 7th December

Quote
So capacity is an issue, just nobody at PN has the brass tacks to say it.


I think that "The problem is our network is full from 7am until about 3am. And it is over-full from about 3pm until 11pm." is a close to admitting the truth as your going to get without them mentioning the 'C' word.
N/A

Traffic Management Update - 7th December

I would suggest one of the issues with heavy users might be your refusal to advise what actually IS heavy usage.

I have an ongoing issue on a business account (£66 pm to plus), as part of the work to solve it I was for a brief time put on a test profile.

During the testing we used, according to your stats, around 50GB during peak times.

We were told this was much too high - however your residential premier packages at a much lower cost say that you can use up to 60GB peak before getting any restrictions.

So on a business account we were punished for using a lower ammount of traffic than is allowed for your residential type accounts.

You need to clarify what exactly we are allowed to use rather than puenlaising people for getting confused by the morass of differing and vague messages plus are sending.

Be straight with us and then we know what the limits are.

If you can't oput a figure to it then you have no right to penalise people for goin g over a line in the sand you won't even tell them exists. Especially when you still use words like unlimited in your advertising of the relevant products.
N/A

Traffic Management Update - 7th December

Quote
Therefore if customer numbers don't increase we won't add bandwidth unless wholesale costs come down. And customer numbers have been flat since May so we haven't added any bandwidth.


So it looks like there are 2 choices then:

1) Increase your prices for the heaviest users so you can get more capacity so the 'good ones' amongst us have a better 'experience'.

2) Leave things as they are with everyone getting less and less bandwidth and you losing more and more customers.

I haven't reffered anyone to Plusnet since around May because basically you weren't good enough anymore. I've no doubt there are a lot of other people doing the same. Perhaps that's why things have been flat since May. So will you be able to increase the capacity anymore unless wholesale prices change?

Also everyones bandwidth requirements are going to just keep on increasing. Without wholesale prices dropping how are you going to manage that?

But I guess just reading the two Plusnet posts in this thread your just admitting that you don't have the capacity for your customers anymore. Is that right?

John
Zathras
Grafter
Posts: 295
Registered: 01-08-2007

Traffic Management Update - 7th December

So essentially this is all down to the roll out of 'upto8mb' if you dont increase bandwidth without revenue.

You give people upto8mb theyre gonna use it, hence the 18% bandwidth increase I saw metioned, its poor forward planning at the very least, and its cost Plusnets name for quality .
Neil_A
Grafter
Posts: 450
Registered: 04-04-2007

Traffic Management Update - 7th December

Quote

Also everyones bandwidth requirements are going to just keep on increasing. Without wholesale prices dropping how are you going to manage that?


Indeed - how can any ISP continue to operate profitably?
They can't. That's why most ISP share values have taken a battering in the last 9 months.

Recently BT have published new pricing that will come in next year. That will make it easier for larger ISPs with growing bandwidth needs. But it will cripple smaller and medium sized ISPs who can't control their bandwidth more so than CBC ever did.

So as usage increase costs will come down for those ISPs able to manage large networks. Those who can't get the economies of scale will have to buy from other larger suppliers who can.
Neil_A
Grafter
Posts: 450
Registered: 04-04-2007

Traffic Management Update - 7th December

Quote
Mmmm, isn't it a basic rule of commerce that you have to invest to grow?


Of course. And we always keep a central pipe on a rolling order so that we can deploy it at relatively short notice if we need to.

But the mistakes we made in 2004 was to implement capacity too soon and the result was more pipes than we could pay for - hence introduction of 15 different flavours of FUP, SUP, AUP, shaping etc etc in order to get back to a sustainable model. A good search of these and the ADSLguide forums will give you plenty of comentary on that if you missed it the first time.

Choosing when to deploy and when not to deploy is a balancing act. If we'd been managing the heaviest users in Aug-Nov as well as we did April-July we wouldn't be experiencing these issues and no-one would be demanding another pipe. Ironically (and in partial agreement with your point) if we'd been on top of this in September, referral rates would no doubt have increased and right now we'd have another pipe because instead of customer numbers being flat they would have been increasing in October and November which are traditionally the busiest sales months.
Neil_A
Grafter
Posts: 450
Registered: 04-04-2007

Traffic Management Update - 7th December

Quote
What happens then is that titanium squeezes gold, which squeezes silver, which squeezes bronze.


Incidentally, Bob has just pointed out to me that I've inadvertently told a porky. Gold, silver and bronze all have minimum platform share % values assigned to them. Titanium can over-ride these but gold and silver can't force silver and bronze below their minimum values.

Make sense? This explains why we got so picky about downloads disguised as VoIP and Gaming (and had to put in the temporary rate limits) because too much traffic in titanium can kill everything else.

So apologies for that porky - thanks Bob Cheesy
N/A

Traffic Management Update - 7th December

Quote
Quote

Also everyones bandwidth requirements are going to just keep on increasing. Without wholesale prices dropping how are you going to manage that?


Indeed - how can any ISP continue to operate profitably?
They can't.


There's honesty. In other words you can't make a profit so our service is going to suffer. I assume that's what your saying - that Plusnet is not profitable so you can't provide the service we require? So we're better off else where? Or do Plusnet make a profit and should they be investing that into their current capacity to keep the customers they have happy in the hope that referals will increase?

I understand that the wholesale prices need to drop, that they are currently to expensive. So in the meanwhile why not move those customers needing more bandwidth onto a higher tariff. It would either stop them using large amounts of bandwidth or you'd get more money out of them. Either way, if you spent the extra revenue raised on more capacity, your current customers would see an improvement and hence be happier and then more likely to start referring again.

Alternatively stop all the apparent secrecy over what is too much to download and what isn't, set some firm limits within your current capacity so customers know where they stand. Keep us to it (or charge extra if we're over it). Then hold your hat and take a chance that not to many will leave.