cancel
Showing results for 
Search instead for 
Did you mean: 

Broadband Update - P2P being throttled around the world

Plusnet Staff
Plusnet Staff
Posts: 12,169
Thanks: 18
Fixes: 1
Registered: 04-04-2007

Broadband Update - P2P being throttled around the world

I originally posted the following in this thread on AG and this thread here. I guess that it also follows on from this post as well as this. ADSLGuide suggested stickying it on their forum for a couple of weeks, so I'm also doing the same here.
----------------------------------------------------

As an Internet Service Provider we aren't really that bothered about what (in terms of content) our customers are using their connections for. If you are downloading material without the owner's permission then that's really between you and the copyright holder (or agent acting on their behalf). Our involvement would only go so far of what is required of an ISP under UK law. i.e. we are under no obligation to pass on any customer details to a third party unless required to by current legislation.

We do not monitor what websites our customers go to, what files they upload and download or who they talk to on IRC/MSN.

We only record how much people upload and download, what types of traffic are being used and the normal logging for the servers we operate (e.g. portal webstats or ftp logs to homepages).

If pointed out to us we would, for example, remove copyright material from hosted websites or child pornography from usenet (with about 1TB of data a day on Usenet we can't actively police it, plus there's a whole other legal minefield that I won't go into).

As such, it's the amount and type of traffic that is our concern, and this would be something that every ISP would be monitoring and managing to a lesser or greater extent.

I would not consider an ISP to be competant if they didn't have some kind of network management in place, even if they didn't need to use it all of the time, as part of a severe outage a minimum of setting up QoS on the traffic is necessary. If memory serves correct Zen had to do just this and QoS traffic when they had problems with a cross country link.

At £1.7m per year for a 622Mbps pipe it doesn't make sense to have lots of centrals knocking about empty just in case they are needed. To keep the costs as low as possible the central pipes need to be utilised as best as possible. The big cost driver for any ISP using CBC as you can see is the number of centrals required. How many pipes that are required are based on many factors, but usage and in particular the peak time usage is key.

A question I see quite often is how heavy users have an effect on light users. I'll try and illustrate an example.

Imagine you have 75 people on 2Mbps lines who like to download at a maximum 8 hours a day. If they all picked the same 8 hour slot then that's 150Mbps of traffic, or the equivalent of a single central pipe segment.

Now if they chose to download when the network is busiest, that's adding an extra 150Mbps to the peak time load which either means an extra central pipe is required or we hit capacity, without any form of traffic management hitting capacity is a very bad thing. All traffic is equally affected as are all customers.

Now change the example around and encourage those customers to download when the network is quietest. Whilst this increases traffic at off peak times, it doesn't increase it to such as level as it gets to at peak time.

So, as you can see, the key to keeping the cost of the BT Centrals down is to use the pipes in as an efficent manner as possible, because at the costs of extra pipes it doesn't take much to send the costs spinning out of control. You need to have a proper capacity management programme in place to ensure that central pipes are only added in line with customer and revenue milestones and not just by usage levels.

By encouraging off peak usage, in particular for high usage apps like P2P and Usenet we can achieve the goal of an efficiently utilise network.

Additionally the traffic management we have in place allows for several things. It allows first of all to ensure that we can protect latency sensitive applications when the network is busiest. Applications that from experience see ping spikes and packet loss and thus become pretty much unusable. Also if the network is busy then it's likely that P2P would be seeing a slowdown anyway, so all we are doing is managing that slow down rather than let it just happen to any protocol.

The other thing we can do with traffic management is again when the network is busiest allow different types of accounts to get a bigger slice of the bandwidth pie. So essentially we can set it so that the higher subscription accouts will see the biggest slices. Without this type of management every customer on your network, regardless of subscription cost, would be fighting it out for an equal amount.

Regards,

Dave Tomlinson
PlusNet Customer Support
2 REPLIES
Plusnet Staff
Plusnet Staff
Posts: 12,169
Thanks: 18
Fixes: 1
Registered: 04-04-2007

Broadband Update - P2P being throttled around the world

The following two posts have been taken from this thread and as per a suggestion I’ve made them a sticky as there is a lot of information here that may answer a lot of questions customers may have.
----------------------------------------------------

Quote
I don’t mean to scaremonger from my post, but that’s
the view I take - you are of course entitled to your own.


Indeed, but I'm sure you agree that some posts from some people (no names
mentioned) just seem to attack and scaremonger just for the sake of it. I
can understand genuine concern, and where information is unclear or
missing, or where it's not written in an easy to understand way.

Some of the things we do behind the scenes are difficult to explain and
some will rely on prior knowledge. We know that lots of customers aren't
interested in how their connection to the Internet works, and so long as
they can turn on their PC and go to say eBay that's fine. There are other
customers that want to know all the finer details of everything that
happens. And there are plenty of others in between.

If I can try and clear things up, please allow me to do so and try and put
your mind at ease.


Quote
I can say that I have not noticed any degradation of
service on my connection


Me neither, and I'm pretty sure we aren't alone. My connection works fine
all the time, Usenet downloads at 230+KB/s at any time, other P2P is good
if the speed is there from the source(s), pings are good and low, no packet
loss, browsing nice and snappy, I could go on, but I won't.

Yes, we are seeing customers who are reporting problems, but so too are
customers on other ISP's and no matter what, an Internet service will never
be 100% problem free for every customer, and I'm sure you don't expect
that.

I'd also say that there are a number of reasons for the increase in faults
and problems that are being reported, the first is simple in that we've
never had as many customers as we have at the moment, so it's likely the
volume of faults will increase as the customer numbers increase.

Additionally, BT Wholesale have record numbers of faults being reported to
them by all ISP's, again in part this is because of the increase in
broadband customers in the UK, but also there has been an increase in early
life failures with 2Mbps services. This has meant that the limits for 2Mbps
have been tightened up as we mentioned a few weeks ago.

Also with the volume of faults on the up, the turn around times have been
on the up as well. The change of limits for 2Mbps is helping as are other
measures they are taking and turnaround times are now heading back down as
are the fault volumes.

The rollout of MaxDSL is going to help the fault volumes too as the RAMBO
kit at the exchange should help to ensure that lines aren't provided at
speeds they can't support.


Quote
however I am not running at 2mb, so I am not
expecting as larger piece of the pie as many others, perhaps this is why I
do not see the effects that others are apparently seeing.


I would have said that was unlikely, but possible. I've looked at some of
the speed problems that are being reported and there seem to be several
different causes, and in some cases a combination of causes.

The first case, and probably the most common, is where the slow down is
across the board on everything. Our speed test shows a slow down, as does
ADSLGuide, as well as downloads. In these cases the BT Speedtester will
determine whether the cause of the speed decrease is on our side of the
network, or on BT's (or customer equipment, but that's easy to rule out of
the equation). So far most, if not all the cases I've looked at in this
type of scenario have been caused by contention at exchange/VP.

When an exchange is flagged up as running "hot" by BT it will show as red
on the exchange checker. However, the data on the exchange checker can be a
couple of weeks out of date due to how the data is collated and presented.
Also, it's possible for an exchange to see contention but still be within
the design limits. You just need to ask Kitz about the recent speed
problems she's had because of exchange contention.

Other speed problems we're seeing reported are where the BT Speedtester
shows the speed as normal but a particular application or protocol is
showing as being slow.

In the main we are talking about P2P and Usenet speeds, although I have
seen reports of sluggish web browsing and ftp and increased pings.

In these cases we need a bit more investigation because it isn't always
obvious what the cause or causes may be.

For example, if your BitTorrent speeds are normally 230KB/s, but you have a
10% decrease on the BT speedtester from your normal speed and you are
downloading when the network is busiest, so you may see a further up to 10%
cut due to the traffic management then your speed could be down to 230 *
0.9 * 0.9 = ~ 185KB/s.

If the BT Speedtester drops by more than 10% then that figure falls
further. Even with a normal BT Speedtest, you could still see an up to 10%
decrease at peak because of the traffic management taking you to ~205KB/s.

Also, the nature of P2P traffic means that it is a very bad measurement of
speed. Again with BitTorrent, the speed you can download at on a particular
torrent is determined by several factors. The number of people uploading
and how fast they can upload, how many people are downloading and how much
of the data you can grab for yourself, the load on the tracker can be a
factor as high load may stop you connecting to all of the seeds, and the
speed of your upload can cause a decrease in download because if the upload
is maxing out there will be little room for you "ack" packets. If the ack
packets can't get through then the download speed decreases.

In a similar situation with Usenet, the number of connections you make to
the server can affect the overall download speed. I know from personal
experience that using one or two connections to usenet will mean that my
connection doesn't max out, but 8 connections always does.

It also sound obvious but any other use of the connection will cause
P2P/Usenet speeds to decrease, but because of the way the QoS queues work
it may be that an HTTP download will rank above a Usenet download and the
HTTP download will take the lion's share of the bandwidth. Similar a small
increase in uploads might cause a large decrease in download speeds. E.g.
upload a torrent a 20KB/s might cause your Usenet download to drop from
220KB/s to 110KB/s.

Going back to your original point, on our network you are allocated the
same share of the network for your account type regardless of what your
sync speed is. The determining factors of where your account type sits and
what profile it has are the account type (Plus, Premier, etc.) the account
speed (up to 2Mbps, up to 4Mbps, etc.) and the PlusNet network contention
ratio (10:1, 30:1, 50:1, etc.). The sync speed and BT side contention ratio
aren't factors here.


Quote
Yes another segment has been activated, but this only
Takes the bandwidth back to somewhere near what was available in January -
and thousands more have joined since then.


There are two factors here. First a 622Mbps pipe has greater efficiency
than 4 x 155Mbps pipes. The architecture of the pipes means that the
overheads are greatly reduced, 622's are Ethernet connections, whereas
155's are ATM. As such a single 622 pipe is near enough the equivalent of 5
x 155Mbps pipes in terms of how much "customer data" they carry. We now
have 17 active segments across the 622Mbps pipes, had we stayed on 155Mbps
that would be the equivalent of 21 x 155Mbps central pipes.

The second big factor here is that yes, we've seen considerable increase in
customer numbers this year, but the actual growth has been in Broadband
Plus customers, both with new signups and customers changing from Premier
(or legacy equivalent) to Broadband Plus.

You can see from the utilisation graph

http://www.plus.net/support/adsl/adsl_utilisation.shtml

that Broadband Plus usage peaks at around 225Mbps. In the same time period,
Premier has been fairly static, so although customer numbers have
increased, the amount of peak time bandwidth, which is what drives the
number of pipes that are required and thus the main cost factor, has only
marginally increased.


Quote
In terms of "sorting" I mean explaining to customers
exactly what shaping takes place and when - also what are the limits before
we get a warning mail? PlusNet seem the only ones able to determine what
sustainable and unsustainable usage is - everyone else just has to guess at
the limits.


I think I've explain the shaping elsewhere, but I'll see if I can sum it
all in one place.

Essentially there are two processes that take place on the traffic as it
flows through the network.

First, the traffic hits the Ellacoya e30's. Several things happen here,
initially the Ellacoyas use deep packet inspection to see what the traffic
is. They then place a rate limit if needed on overall level of P2P traffic
going through. Finally the Ellacoyas will apply the profile settings to the
traffic.

Once through the Ellacoyas the traffic is divided into three input queues -
gold, silver and bronze.

For reference

Gold = http/https, and email
Bronze = Usenet & P2P
Silver = Everything

(makes a little clearer to list them in that order).

The traffic in these three input queues is then put into a single output
queue. So long as the three inputs are less than the output no traffic is
dropped at this stage.

If the amount of traffic is greater than the input queues then traffic in
the bronze queue would start to get dropped. The amount of traffic dropped
would be just enough to ensure that all gold and silver went through and as
much of bronze as possible. What traffic is dropped is also proportionate
to the overall level of that type of traffic. As P2P traffic as by far and
aware the largest traffic on the network and is far larger than Usenet,
this is the traffic that will see the first drops. The design of this
management means that except under exception circumstances you wouldn't see
more than a 10% drop in P2P traffic. It's possible for Usenet traffic to be
dropped too, but because of the usage patterns on Usenet at the moment,
usage peaks at 1am whereas HTTP peaks earlier in the evening, and P2P is
pretty constant throughout the day, therefore you would generally only see
P2P dropped.

Also as each ERX and Ellacoya work independently it's possible that speeds
could be a little different depending on which gateway you connect to.

This is another reason for the implementation of idle timeouts, at the
moment we have plenty of spare sessions, but it's possible that a single
segment could have lot's of idle connections. This isn't an efficient use
of the network because while these are idle and not using bandwidth,
another segment could be seeing a larger than necessary decrease in speed.
Also, if there is a major service outage because of a loss of one or more
central pipes, the number of available sessions would be reduced and so by
freeing up unused sessions we can a) ensure that everyone can connect and
b) free the Ellacoyas and the ERX QoS queues we can deliver a level of
service so that people can still get their emails and browse the web, etc.

Regarding the limits and what constitutes unsustainable usage, this is
something we actively looking at. Neil recently posted what we say said was
considered unsustainable which was in effect 100GB for a £21.99 Premier up
to 2Mbps account, with higher figures for the higher priced Premier
accounts.

The exercise we did last month where we contacted customers whose usage was
above those unsustainable levels will be done again later this month where
we will ensure that exactly what is unsustainable is clear.


Quote
I agree, why would you want to make the service worse
for your customers?


Exactly, we wouldn't. Our customers are our life blood, it's our existing
customers that recommend us to friends and family. We want to make things
the best that we can.

Quote
I don't know, and it would seem neither do PN - why
introduce idle timeouts on to a broadband package? The whole ethos around
the entire planet is "always on", while I agree this makes the network more
efficient for those on it, it doesn’t help those who may need their always
on connection - so good for some = bad for others.


You've got to look forward, you've got to plan ahead and ensure the service
you are delivering is the best possible. As I've said above we've seen what
could happen and want to ensure that this doesn't happen. I've used the
water example before, so I won't use it again, but service could still be
considered "always on" even on Broadband Plus. We are removing the "always
on" references though and won't be going down that route, but for those
that do need the PPP session staying up all the time they can always change
to Premier or PAYG.

In most cases though to reconnect a disconnected connection will be a 2
second job to click reconnect, for many using routers it will automatically
reconnect.


Quote
Why introduce traffic shaping on your "premier"
products? There could be a multitude of other options to increase
efficiency without tampering with your most elite of home user
packages?


The reason is simple, to ensure the best quality of service for all
applications when the network is busiest. From experience at a previous ISP
I've seen situations where the central pipe hits capacity and it isn't nice
for anyone. Everything sees slowdowns, packet loss and ping times. With
proper network management we can ensure that the latency sensitive
applications are protected and that the customers paying the higher
subscriptions see the benefit of this through a higher slice of the
available bandwidth.

The ISP that doesn't have network management is the foolish ISP. They are the ones that don't care about their customers, that are happy to see increasing costs, that aren't planning for the future.

We are here to deliver the best Internet service we possibly can, we aren't here for the short-term, we are in this for the long haul. We are planning now for the future and planning now to give more services and options for our customers.
Plusnet Staff
Plusnet Staff
Posts: 12,169
Thanks: 18
Fixes: 1
Registered: 04-04-2007

Broadband Update - P2P being throttled around the world

Quote
Thanks for that post, you have put some effort into that. It certainly goes some way to explaining what is taking place at the moment.


No problem, that's exactly what I want to do. I want to ensure that our customers get the answers to the questions they are answering. As we all know not everyone will like all the answers, but I least want to ensure that where customers don't understand or know why we are doing something that the information is there for them.


Quote
I still feel that the method other ISP's are taking (spare capacity and higher prices over maximising efficiency) would seem to be the way many P2P folk would want to move.


This is certainly possible, however we have to look at our customer base as a whole rather than just certain people and certain types of usage.

We also need to look towards the future and what's going to happen when faster speeds become available to more people.

We all know that 8Mbps speeds are just around the corner as such we have already planned for this. The question I would ask is how much would it cost to service the needs of someone using an 8Mbps connection to the max 24/7? The answer is in excess of £20,000 per year.

If you have 20 x 8Mbps customers all maxing their connections 24/7, then between them they are using the equivalent of a single 155Mbps segment. At over £400,000 per year for a 155Mbps segment that's £20,000 per customer.

It's unlikely that any ISP would be able to sustain that amount on CBC without some kind of network management or bandwidth control.

As we've seen with the measures that other ISP's have taken, where PlusNet lead others will follow.

From an economic point of view it just doesn't make sense to have empty pipes lying around just in case you need them. To have an empty 155 as I said above just lines BT's and VAT man's pockets with £400,000 of your customers' hard earned cash for little or no return.

To utilise you central pipes to the max means you keep your costs down and you keep your prices down.

For any ISP using CBC or UBC then they will have done the same maths as us, and looked at the same equations as us and if they want to stay in business will have come to the same conclusions as us regarding sustainability (granted they may come up with different points at which usage is unsustainable), but for an ISP to offer an 8Mbps service using CBC or UBC I would very much doubt they can do that at a price people want to pay without looking long and hard at how they manage their network. For reference standard charging won't offer the faster speeds and LLU obviously has different costings.


Quote
I personally feel that PN are trying too hard to be a jack of all trades rather than a true premier service provider or indeed a true low budget provider.


The choice of services we offer is pretty much suitable to everyone (bar those with requirements to do download vast quantities), hence the different choices of account type from Plus to PAYG to Premier.


Quote
It may be the case that measures being taken will improve overall performance of latency sensitive applications for those paying a premium, but some of those on the premier package (paying a premium) may consider P2P as their most important use of the net - should they not expect to get a premium service because you consider the traffic to be less important than they do?


P2P isn't latency sensitive, if you start seeing packet loss and ping spikes then all that happens with P2P is that it slows down a bit. By managing the network in the way that we do, when the network is busiest the services that are affected the most are protected and P2P sees exactly the same sort of performance (give or take a percentage point or three of throughput) as if we weren't managing the traffic at all.

P2P usage is pretty constant throughout the day, and we are ensuring that we deliver the best performance on P2P that we can, but when the network is busiest the P2P traffic would slow down along with everything else, so it isn't as though they really lose anything because it would have dropped by 5-10% in speed anyway.


Quote
I agree that any issues experienced by people on the products that have idle timeouts would be very minor (those wanting to remote connect may suffer more)


To be honest I wouldn't have thought on Broadband Plus that would be very high because of the dynamic IP. Yes, they could set up something like DYnDNS, but if you are going to that effort then in all likelihood you would probably be looking at Premier anyway.


Quote
but I doubt it would be these issues which would cause PN the most damage - I would think that being known as one of the only (if not the only) providers to implement timeouts on broadband will do more harm to PN than any tickets raised.


I'm not so sure that it will. The session limits on BT's central pipes are the same for all ISP's. BT Wholesale were looking at developing BT Central Extra which would allow the session limit to be increased, but this was dropped. Discussions are ongoing with BT Wholesale, and not just from ourselves but other ISP's too. As I said above where PlusNet lead others will follow.



Quote
Could I suggest that some sort of scale be added to the utilisation charts as this may quell some of the storm.


I can ask, there have been several suggestions regarding changes to the graphs so I'll make sure they are all raised.


Quote
May I be so bold as to suggest your reply or the main sections of it be posted as a sticky? It could help people before they post similar concerns or questions.


Good idea. Might be worth adding to the existing sticky.