cancel
Showing results for 
Search instead for 
Did you mean: 

Traffic Management system (64987)- UPDATE

puddy
Grafter
Posts: 1,571
Registered: ‎10-06-2007

Re: Traffic Management system (64987)- UPDATE

What I don’t understand is why are Plusnet based in Sheffield but there equipment based in London would it be better if they were in the same office as their equipment?
In addition we were told the internet was brilliant it would always find a way through even if its way was blocked as others said why did one hard drive bring the whole network down
Puddy
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,893
Thanks: 4,986
Fixes: 316
Registered: ‎04-04-2007

Re: Traffic Management system (64987)- UPDATE

Quote from: wisty
I don't know Bob, but the existing support page very clearly ( and carefully?) focuses on DOWNLOADING as the activity which is impacted. The page is about downloading , and the CDN section only mentions download sites. You clearly cannot list all of them, but a mention that some of the UK's most popular sites ( BBC & E-Bay would be good examples , are Facebook & YouTube also in there?) are included in this  category might just alert people to your strategy a little more clearly. As it stands the current page is awfully close to being deliberately misleading in my view. I have read that page several times, and a s reasonably technical user I never realised that my use of the BBC website or E-bay would be impacted by DOWNLOADING traffic shaping.

How about...
[quote author="Possible support page revision"]"Most downloads from Apple/Microsoft, PSN/XBLA downloads and multimedia content from sites such as the BBC, eBay and Facebook are hosted on one of the above Content Distribution Networks. Depending on your account type, downloads from these sources may be slowed during busy times."
Quote from: wisty
Quote from: Bob
Quote from: wisty
However late this afternoon I can't get BBC to news or E-bay at all. The BBC is sitting transferring data from node2.bbcimg.co.uk. E-bay lets me log in, but once in it sits waiting for include.ebaystatic.com (neither of which are on the CDN list or would reasonably be classified as download sites by a user).

Both node2.bbcimg.co.uk and include.ebaystatic.com are Akamai hosted.
Some Download Site/Server traffic is also in the 'Gold Plated' traffic queue. This queue only has a finite amount of bandwidth assigned to it. As more and more people have been disconnecting/reconnecting, the less bandwidth there would have been to go around.

THAT is the explanation that should have come out much earlier - All the people who have been transferred to THE profile by the fault are now/were sharing a small subset of Plusnet's bandwidth and should expect very poor performance.
Why could the queue not have been allocated more bandwidth for the duration of the fault?

This is something I asked. It would have involved making queue weighting changes across every single one of our edge routers (which would then need to have been reverted once the problem was fixed). The focus was on fixing the underlying problem, rather than spending the engineers' time making configuration changes elsewhere that could have made the situation worse, and almost certainly would have detracted from the issue at hand. Having said that, this is something that needs looking at in the aftermath. This is due to those who were reporting a *complete inability* to fully render certain websites. Despite the fact I was on a default profile, I couldn't replicate any of these problems which begs the question why?
Quote from: wisty
I think you are going to have to get much clever about dealing with CDN traffic. If you are only allocating a finite amount of bandwidth to the BBC, E-bay and other very popular sites I think you should clearly say so rather than obfuscating the issue behind  "downloading" .

The 'finite' amount of bandwidth assigned to the 'Gold Plated' queue and the rate limiting on Download Sites/Servers works very well under normal conditions.  If it didn't, we'd have Value and legacy Broadband Plus customers complaining every night.
Quote from: geofftswin
Would be interested to know the reason why a single hard disk failure caused this much chaos though?

Hopefully Fletch has helped answer that for you?
Quote from: puddy
What I don’t understand is why are Plusnet based in Sheffield but there equipment based in London would it be better if they were in the same office as their equipment?

Our centrals, edge routers and DNS platform are all hosted down in London. It makes sense therefore for the traffic management platform to be hosted down there. It reduces network latency and negates the traffic overheads you'd see if everything had to travel backwards and forwards between London and Sheffield.
Anyway, looking at my usage, it looks like usage is slowly catching up. I've no usage data from today and only half of it from yesterday but that's an improvement on the situation yesterday. I'll find out what the latest is with the profiles etc. and report back/update the Service Status thread...

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Traffic Management system (64987)- UPDATE

Quote from: Bob
How about...
[quote author="Possible support page revision"]"Most downloads from Apple/Microsoft, PSN/XBLA downloads and multimedia content from sites such as the BBC, eBay and Facebook are hosted on one of the above Content Distribution Networks. Depending on your account type, downloads from these sources may be slowed during busy times."


I'd suggest "and multimedia content from many sites (for example the BBC, eBay and Facebook)"
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Traffic Management system (64987)- UPDATE

Bob, I think you also need to explain why the Centrals etc are in London.
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,893
Thanks: 4,986
Fixes: 316
Registered: ‎04-04-2007

Re: Traffic Management system (64987)- UPDATE

Quote from: jelv
I'd suggest "and multimedia content from many sites (for example the BBC, eBay and Facebook)"

Sounds fair, I'll ask our content folk to get this changed (ref: 330473).
Quote from: jelv
Bob, I think you also need to explain why the Centrals etc are in London.

Probably better to answer taking this blog post into consideration, as we'll be doing away with the centrals soon anyway. In their place we'll have four 10Gbps host links using BT Wholesale's WBMC network.
There are five WBMC shared nodes, three in London, one in Manchester, and one in Birmingham (We're using the three in London). London presents the best choice because it has better connectivity to the wider Internet. Linx, LoNAP, Level 3 and most of the main transit/peering suppliers are located in London. If our host links/centrals terminated in Sheffield then we'd have to route traffic across half of the country in order to to get it to our transit/peering partners. This creates latency, costs us money, and generally isn't a very clever way of approaching things.
Anyway, here's the latest on the traffic management problem.

  • Usage data is up to thirteen hours behind and whilst it's catching up, it's likely to take the rest of the day.
  • Almost all customers have re-authenticated with the traffic management platform and should now be assigned the correct profile.
  • There are approximately 2000 customers however who remain on an incorrect profile. This is typically customers who were activated yesterday, recently billed (or failed billing), recently changed account type or recently added a static IP address/IP block. We're looking at fixing these today, but I don't have an ETA at the moment, I'll report back once I do.

Edit: We're aiming to have the broken accounts cleaned up by 3pm this afternoon.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

Force9Original
Grafter
Posts: 352
Registered: ‎05-02-2010

Re: Traffic Management system (64987)- UPDATE

WWWombat
Grafter
Posts: 1,412
Thanks: 4
Registered: ‎29-01-2009

Re: Traffic Management system (64987)- UPDATE

In response to both this...
Quote from: Bob
This is something I asked. It would have involved making queue weighting changes across every single one of our edge routers (which would then need to have been reverted once the problem was fixed). The focus was on fixing the underlying problem, rather than spending the engineers' time making configuration changes elsewhere that could have made the situation worse, and almost certainly would have detracted from the issue at hand. Having said that, this is something that needs looking at in the aftermath. This is due to those who were reporting a *complete inability* to fully render certain websites. Despite the fact I was on a default profile, I couldn't replicate any of these problems which begs the question why?

and this...
Quote from: Bob
The 'finite' amount of bandwidth assigned to the 'Gold Plated' queue and the rate limiting on Download Sites/Servers works very well under normal conditions.  If it didn't, we'd have Value and legacy Broadband Plus customers complaining every night.

Assuming you don't have such complaints, it looks like the "finite amount" of gold-plated traffic allowed is enough to deal with the existing number of Value & BB+ customers (or at least, the statistical number of them actively "downloading" at one time), but that the same "finite amount" is insufficient when more people end up trying to use the same "finite amount".
In one answer, you reported that you didn't see any problem with viewing certain sites, but that some others obviously did. In a separate answer, you noted that you didn't re-configure the "finite amount" because that would involve reconfiguring every edge router. I can't help but feel that these two are linked. Perhaps some parts of your network were running out of their own chunk of this "finite amount" while other parts weren't.
I can't help but feel that having any type of quota associated with a prioritisation-level is the wrong thing to do. Surely "gold-plated" should always be higher priority than "silver", no matter how much has been used?
Quote from: Bob
Quote from: jelv
I'd suggest "and multimedia content from many sites (for example the BBC, eBay and Facebook)"

Sounds fair, I'll ask our content folk to get this changed (ref: 330473).

I'm a bit leery of this change, and have some extra things to suggest...
The reason I'm a bit leery is, as dirtymonkey pointed out, the CDN is gradually being used as a cache for more than just multimedia content, and is now including many common "small files" such as javascript and even text content, to aid in the speed of loading of sites to end-users. In the long run I'd rather that PN's traffic management was clever enough to allow the "small files" through at the "Gold" level to match standard "Browsing & Email" - which would make Jelv's suggestion valid. In the meantime, I'd suggest a different wording:
"Content Distribution Networks (CDNs) are used behind the scenes to deliver some of the content for many popular websites (for example, the BBC, eBay and Facebook) and many Music, Video and Game downloads (such as from Apple, Microsoft, PSN and XBLA).  For some websites, the CDN may be delivering only the larger multimedia files, while for other sites, even smaller image and text files will be included. Depending on your account type, content and downloads from these sources may be slowed during busy times."
If you're not happy with such a big change to the paragraph, my other option would have been to change "multimedia" into "cached", but I thought this would be less understood by most people:
"Most downloads from Apple/Microsoft, PSN/XBLA downloads and cached content from many sites (for example the BBC, eBay and Facebook) are hosted on one of the above Content Distribution Networks. Depending on your account type, downloads from these sources may be slowed during busy times."
The extra things I'd suggest are:

  • The Traffic Prioritisation page mentions the two download categories, but makes no mention of the CDN being impacted. It is only if you follow the link to the "download servers" page do you see the CDN impact (but why would you, if you were only browsing the BBC website?). I think it should be clearer on the main page.

  • Likewise CDNs aren't mentioned on the Download Speeds page.

  • Then, on the Download Servers page, the CDNs are mentioned. However, it isn't clear whether CDN traffic is categoried in the "sites" or "servers" categorisation. This is important where the speed differs between the two categories (which is every package) or where the prioritisation differs (which is the business packages). The distinction between "sites" and "servers" isn't particularly clear, so I can't guess the category...

  • Back on the Traffic Prioritisation page, the only reference to "gold-plated" is to say "*Gold Plated traffic (and some Gold Traffic) may be rate limited to prevent impact on interactive/Gold traffic at busy times. See Download Sites & servers to find out more about this."
    Unfortunately, there is nothing explained about the gold-plated rate limits on the linked page.

Plusnet Customer
Using FTTC since 2011. Currently on 80/20 Unlimited Fibre Extra.
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,893
Thanks: 4,986
Fixes: 316
Registered: ‎04-04-2007

Re: Traffic Management system (64987)- UPDATE

@WWWombat, i'll reply your post when it's a little more convenient (I'm typing on a smartphone at the moment)
Heads up that the fix for the 2000 broken accounts is taking a little longer than expected. We're now looking at around 7pm for the scripts to finish.
I'll update Service Status shortly...

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

David_W
Rising Star
Posts: 2,305
Thanks: 33
Registered: ‎19-07-2007

Re: Traffic Management system (64987)- UPDATE

Once 4PM came along my download speed has dropped, I'm struggling to get 10K/sec from fileserve, even with throttling I *should* be experiencing 100K/sec, youtube is also playing up.  It only occured at 4PM when my peak hours came into effect, before then I was yo-yo'ing but getting a decent enough speed.
BT Speedtester gives :
Download speedachieved during the test was - 2684 Kbps
For your connection, the acceptable range of speeds is 600-7150 Kbps.
Additional Information:
Your DSL Connection Rate :8128 Kbps(DOWN-STREAM), 448 Kbps(UP-STREAM)
IP Profile for your line is - 7000 Kbps
So I *should* be downloading a lot faster from fileserve.  My bandwidth usage reset yesterday but before that I was on level 1 management, am I somehow still on that because I'm getting the same symptoms.
dvorak
Moderator
Moderator
Posts: 29,534
Thanks: 6,636
Fixes: 1,484
Registered: ‎11-01-2008

Re: Traffic Management system (64987)- UPDATE

Yep, that was on the latest status update, should be srted by 7pm i think.
Can't link I'm on my iPad Wink
Customer / Moderator
If it helped click the thumb
If it fixed it click 'This fixed my problem'
David_W
Rising Star
Posts: 2,305
Thanks: 33
Registered: ‎19-07-2007

Re: Traffic Management system (64987)- UPDATE

I'm not entirely sure I qualify as one listed.  I'm not billed for a couple more days "We'll soon be taking a payment of £20.99, for your PlusNet service. This will leave your account on or after 23/12/2010. ".  Not failed (never failed).  No product change or anything, the only thing that happened yesterday was my monthly cycle was reset, my VMBU does show now that it has been reset, but anyhow, give it a bit till 7 and keep my fingers crossed?
/edit - also, before 4PM my connection was fine, I even watched some HD streams, it's just after 4PM that my speed has dropped quicker than a really quick thing, dropping, err, quickly?
spraxyt
Resting Legend
Posts: 10,063
Thanks: 674
Fixes: 75
Registered: ‎06-04-2007

Re: Traffic Management system (64987)- UPDATE

If it is related to being on an incorrect profile due to the weekend problems you should be able to fix that by disconnecting from Plusnet using the router's web interface, then connecting again (note this doesn't involve restarting the router/re-syncing with the exchange).
However if it is an account setting problem (Level 1 management) Plusnet will have to fix that.
David
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,893
Thanks: 4,986
Fixes: 316
Registered: ‎04-04-2007

Re: Traffic Management system (64987)- UPDATE

Quote from: David
No product change or anything, the only thing that happened yesterday was my monthly cycle was reset...

That suggests that your billing date was yesterday. Despite your VMBU being reset, there's a possibility that that you're still one of the 2000.
I've just posted another update. Unfortunately, we encountered some network issues shortly after I posted that which has further delayed the clean-up script Sad
I'll report back later on tonight to let you know whether or not things have finished...

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

David_W
Rising Star
Posts: 2,305
Thanks: 33
Registered: ‎19-07-2007

Re: Traffic Management system (64987)- UPDATE

One of 2,000? Out of several hundred thousand? LUCKY! I'm so getting a lottery ticket  Crazy
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,893
Thanks: 4,986
Fixes: 316
Registered: ‎04-04-2007

Re: Traffic Management system (64987)- UPDATE

Of all the days to get billed! Sad
Sorry to say it looks Looks like the network issue isn't going to get sorted until tomorrow now. Hopefully we'll be able to get the script running pretty early though and have everybody fixed by lunch time...

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵