cancel
Showing results for 
Search instead for 
Did you mean: 

Network Capacity hit by minority of users

Oldjim
Resting Legend
Posts: 38,460
Thanks: 787
Fixes: 63
Registered: ‎15-06-2007

Network Capacity hit by minority of users

Quote
On Tuesday, we peaked at just over 950Mbps of iPlayer traffic. Yesterday, we think that about 2,300 of our customers tuned in to watch Usain Bolt set a new record in the 200m final, hitting a peak of 934Mbps.
http://usertools.plus.net/status/archive/1219306893.htm
Quick calculation shows that the usage rate per customer is about 406kbps - not sure why this would seriously hit the overall capacity.
Perhaps someone more knowledgeable than me can explain
14 REPLIES 14
zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Network Capacity hit by minority of users

The most restrictive part of the Plusnet network (and the most expensive) are the "BT Centrals"
Each central is considered to be a 622Mbps "pipe", split into 4 x 155Mbps "segments" (plus some overhead).  As I understand it, there is a 90 day lead time to install a new "Central", but you can light individual segments in a much shorter lead time (30 days?), so not all of the 155Mbps "segments" on all of the available Centrals are lit, allowing Plusnet to increase capacity fairly quickly.
So, essentially Plusnet are connected to their customers by lots of 155Mbps pipes.
At an average of 406Kbps per stream, it would only take around (155 x 1024) / 406 = 390 users to completely saturate one 155Mbps segment (excluding all other "normal" traffic on the segment).
The problem occurs when there is a sudden "surge" in required bandwidth, concentrated on just one segment.  Statistically speaking, which segment or central you "land" on when connected is random (although there are tricks that PN can use to 'direct' you towards a particular central).  This also means that statistically speaking any surge should be evenly spread across those segments.  In practice it is often apparent that there is a concentration on one particular segment which may be saturating it.
This is what is happening when you start seeing some gateways performing worse than others.  This is also where the Packet Loss by Gateway graph is useful, as it will show which gateways are overloaded and which would be a good choice to "jump" onto by dropping your internet connection and re-initiating.
So, it's not a matter of the entire Plusnet network being overloaded, it's that individual segments are likely to be overloaded (especially as they will be trying to carry the "normal" daily traffic on them).
That's a kinda basic overview and I've simplified some things but I HTH,
B.
Oldjim
Resting Legend
Posts: 38,460
Thanks: 787
Fixes: 63
Registered: ‎15-06-2007

Re: Network Capacity hit by minority of users

Thank you Barry,
Correct me if I am wrong - and I am sure you or someone else will  Grin - but I thought that traffic balancing between segments of a central pipe was automatic so the capacity limit was by central pipe not segment which gives more latitude.
Notwithstanding that 5 centrals starting getting significant gold packet loss mid afternoon yesterday. My point is really that the apparent headroom for gold traffic must be much less than 950/5 = 190Mbps or put it another way it would appear that gold traffic on the affected centrals jumped by about 20% (making some alliowance for the centrals which didn't overload).
The part I would like a bit of guidance on is the proportion of the bandwidth allocated to gold traffic (I thought it was less than 50%) and why the dynamic controls didn't jump in to allocate a higher proportion to gold.
Laser
Grafter
Posts: 206
Registered: ‎23-09-2007

Re: Network Capacity hit by minority of users

Not sure it even has to be that complicated. If the calculation of 406kbps/person is correct, that translates to the equivalent of a continuous use of 130GB/person/month, which is way higher than the average usage, and beyond what even the max usage on the various packages is. Therefore, for the duration that this is taking place, if done by enough people simultaneously, the network isn't set up to cope with it.
Whether 2300 people (about 1.5% of the customer base) is a sufficient proportion to justify that argument, I can't say.
zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Network Capacity hit by minority of users

Actually, it gets more complicated than that Sad
Each 622Mbps pipe is split into 4 segments and each segment into 3 LTSs (L2TP Tunnel Switches) so there are 12 LTSs in total, per central. Depending on the load on each LTS you could have 2 customers on the same gateway but different LTSs seeing very different performance.  I *believe* traffic will balance across same-segment LTS, but not cross segment (and definately not cross-central)
As I understand it, the way that the traffic management works is that titanium gets all the bandwidth it needs then x% goes to gold, y% to silver, etc. and that's done on a per LTS basis rather than per gateway.
The queue weightings for Gold, Silver, Bronze/Best-Effort etc. are assigned a minimum value. This minimum value represents the lowest possible bandwidth allocation to be assigned to that queue at any particular time. Think of it as the lowest it can get squeezed to. The way it would generally work is:
- Network becomes busy
- Bronze/Best effort squeezed to lowest weighting, if more capacity required then ..
- Silver traffic squeezed to lowest weighting, if more capacity required then ..
- Gold traffic squeezed to lowest weighting
The Titanium queue always takes precedence over the others and I think it works in a slightly different way. AFAIK it's the only queue that could theoretically squeeze the others below their minimum weightings.
However, if I could wake Tommo up, I suspect he could answer with a greater degree of accuracy Smiley
B.
zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Network Capacity hit by minority of users

As a side issue, the Plusnet network is "flexing" much better than some others.
B.
Oldjim
Resting Legend
Posts: 38,460
Thanks: 787
Fixes: 63
Registered: ‎15-06-2007

Re: Network Capacity hit by minority of users

Barry,
From one of your links
Quote
Shows what a shocking waste of bandwidth transmitting TV over the web is – also as it looks like the BBC licence payers are funding free viewing for everyone else, I would much rather see them putting whatever it has cost to fund this towards actually making programs.
zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Network Capacity hit by minority of users

Hmm, I'm undecided on that one.
I personally use the iPlayer because it allows me to timeshift very conveniently.  I don't see the necessity for HD streams, and I do think that these are probably ahead of the times (at least until we get a faster network infrastructure able to cope with them).
Also, I read today (and I've been trying to find where I read it..) that the BBC are changing their content provider from Akamai to Level 3.  This has a knock-on effect of costing ISP's more.
This is because the current BBC peering arrangements allow what is essentially a "free" peering connection.  Akamai will peer directly into LINX (for example) and it won't cost ISPs very much to exchange data across that path.
Level 3 are a Tier 1 transit provider, and as such will only peer "for free" with other Tier 1 transit providers.  Any other customers have to pay more to peer data with them.
This effectively reduces the BBC's transmission costs, as the charge for transmitting the data can be recouped from the peering charges that ISPs have to pay. 
It's not likely to affect ISPs who have a direct peering agreement with the BBC (like Plusnet who do have a 1Gbps connection straight to them) but it could affect ISPs who are reliant upon the current Akamai distribution model.
Still trying to find the article.  I'll link it if I do.
B.
EDIT:  It was on TBB, linky
Oldjim
Resting Legend
Posts: 38,460
Thanks: 787
Fixes: 63
Registered: ‎15-06-2007

Re: Network Capacity hit by minority of users

That is interesting in more ways than one.
Obviously cost and reliability for the smaller ISP's
Less obvious - assuming your information is correct - the Plusnet direct link was effectively saturated and limited the total bandwidth iPlayer could consume
zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Network Capacity hit by minority of users

I would say that the Plusnet <->BBC link was probably close to saturation.  However they could also have used bandwidth coming via LINX if the link was saturated completely so it's not necessarily a matter of the 1Gbps link being the 'hard limit'.
B.
Oldjim
Resting Legend
Posts: 38,460
Thanks: 787
Fixes: 63
Registered: ‎15-06-2007

Re: Network Capacity hit by minority of users

They could have but I bet they wouldn't as it puts an inbuilt restriction on the effect.
Another factor - and I may be wrong here - is that via the Akamai link Plusnet traffic gets hit for the traffic coming into the system and out again whereas with the dedicated and independent  link it is only the traffic going out.
zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Network Capacity hit by minority of users

Again, the premise (as I understand it) is incorrect.
Plusnet will pay a fixed fee for having a connection into LINX as per https://www.linx.net/govern/servicesfees.html
Plusnet currently have 1 x 10Gbps link from Telehouse East and 1 x 2Gbps link from Telehouse North.  This will cost them around £2,600 per month, plus port fees and rack rental, regardless of how much traffic passes over the link (I'm ignoring the 80% utilisation charge to simplify any math)
Akamai peer directly into LINX for free, so there is no "cost" for transferring via LINX.  It's just not as convenient as using a dedicated link from the BBC.
B.
EDIT:  Of course, the Content providers pay Akamai to host the service, which is how Akamai stay profitable and still peer "freely" into LINX.  The similar is true with Level 3 but the content providers pay less, and the ISP *also* have to pay.
Laser
Grafter
Posts: 206
Registered: ‎23-09-2007

Re: Network Capacity hit by minority of users

Quote
Shows what a shocking waste of bandwidth transmitting TV over the web is

I'm afraid I have to side with this. The network configuration of the interweb is not well suited to mass distribution activities. Every time there's a "big thing" (sporting events, news events, patches released, aliens discovered, etc.) there's a problem. Since the material is already freely available via a TV aerial, which 99.9% of the population already have, I really cannot see the point of TV over the internet.  Undecided
[quote=Barry Zubel]I personally use the iPlayer because it allows me to timeshift very conveniently
I have a PVR. Does the same thing except is easier to use, plays on my normal living room TV rather than hunching round a pokey computer monitor, has full res images (OK - not very good res on some Freeview, I'll admit!), and doesn't eat up all my broadband quota...  Tongue
dave
Plusnet Help Team
Plusnet Help Team
Posts: 12,257
Thanks: 306
Fixes: 4
Registered: ‎04-04-2007

Re: Network Capacity hit by minority of users

Quote from: Barry

However, if I could wake Tommo up, I suspect he could answer with a greater degree of accuracy Smiley
B.

Near enough, although the ERX side is done per pipe rather than per LTS, so the queue weighting applies to the whole pipe not each individual LTS, but because there's room for each LTS to flex against each other it doesn't really make much difference, although with less segments lit there's less room for the flexing on each pipe. Of course the dropped traffic applies on a per LTS basis rather than the whole pipe...
Dave Tomlinson
Enterprise Architect - Network & OSS
Plusnet Technology
MrToast
Grafter
Posts: 550
Registered: ‎31-07-2007

Re: Network Capacity hit by minority of users

Quote from: Barry
It's not likely to affect ISPs who have a direct peering agreement with the BBC

I think it will.
tracert to the L3 iPlayer host (eg 205.128.67.11) and the route goes via Telia.
L3 are hosting iPlayer on L3's network not the BBC INet network.
I'm not sure how significant the transit charges are compared to paying for BT Centrals though.
However, I think it raises another interesting question. L3 now have exclusive BBC content carrying transit traffic to sell. Should a publicly funded body distribute its content this way or continue to make it available by the no-fee peering relationships?