After reading and discussing some of the comments on our recent iPlayer usage blog and the follow-up I thought that for the next topic it would probably be useful to put together a piece on how UK ISPs like PlusNet are charged for the broadband services that they provide.
Different Delivery Methods
As an ISP there are a number of choices on how to deliver broadband service, we'll cover them off first with a short overview (This list isn't exhaustive, but covers most scenarios):
- BT IPStream - The most common way of providing broadband, this is a product that is provided by BT Wholesale and is available nationally at almost all BT telephone exchanges.
- BT Datastream - Another BT Wholesale product that is available at all exchanges but differs from IPStream in how the data is routed from each exchange to the ISP.
- LLU - LLU or Local Loop Unbundling is where the ISP installs its own hardware at the BT telephone exchange rather than using BT's. LLU comes in two forms, shared and fully unbundled. One important note about LLU is that at the moment it is only available at a subset of the exchanges as the LLU operators can choose which exchanges to unbundle and as yet none have done every exchange. As such for an ISP that uses LLU to offer a national service they need to utilise IPStream or Datastream as well.
- Shared LLU - In this case BT (or a.n.other telephony provider) provide the telephone voice service and the ISP provides broadband.
- Fully unbundled LLU - Here the ISP provides both the voice and broadband services.
- Cable - Virgin Media don't currently offer a wholesale product and aren't currently increasing the areas they cover.
- Others - Wholesale Broadband Connect (a subject for a blog or three on its own), Wifi, etc.
For this blog we will concentrate mainly on IPStream but also discuss the differences between IPStream and LLU. We should probably have a quick look, as part of this discussion, at how broadband works and specifically how a customer's traffic gets from their PC to the Internet. That will then help illustrate where the different cost elements come from.
The Local Loop
The first part is the connection from the PC to the local exchange via the phoneline. This is the actual ADSL part of the connection. In the IPStream world we pay a rental to BT of between £5.77 and £11.63 depending on a number of factors - whether it's a fixed rate or max service, whether it gets a higher priority on BT's network and which exchange it is (BT offer a discount for the top 1016 exchanges, known as Band 1 exchanges). The costs work out as the following per month (note all costs are ex-VAT).
IPStream Max - £7.01 IPStream Max Premium - £11.01 IPStream Home 256kbps/512kbps/1Mbps/2Mbps - £7.63 IPStream Office 256kbps/512kbps/1Mbps/2Mbps - £11.63
With the following rebates on the Band 1 exchanges
Rebate Max/Max Premium - £1.36 Rebate Home/Office - £1.24
Based on customer numbers of around 200,000 the monthly rental after the rebate works out at approximately £1,700,000 On LLU, because the connection doesn't use BT's broadband equipment, the line rental works out at £15.60 per year on shared LLU and £80 per year on full LLU. The LLU operator will also need to install their own broadband equipment in the exchange and also cover ongoing maintenance and upgrades. Furthermore, there is a a £34.86 activation fee and £4.90 cease fee for each of the LLU options, IPStream and Datastream. Official pricing for IPStream and Datastream is available on BT Wholesale's site and on LLU on Openreach's site.
The Backhaul
Going back to IPStream, from the exchange the customer's traffic is routed across BT's network to the Point of Presence (PoP) where the initial authentication is done (to see which ISP you are trying to connect to). The traffic is then routed to the nearest PoP for that ISP. Each ISP using IPStream has one or more central pipes. These connect the ISP to BT's network and are normally provisioned in chunks of 155Mbps; the number of central pipes determines how much data in total the ISP's customers can upload and download. A central pipe can be ordered with a maximum capacity of 622Mbps (we could buy pipes with less, e.g. 155Mbps but we only buy 622's now). There are four charges associated with the central pipes. The first is a one off installation fee of £175,000. In general there is a lead time of around 3 months for a new installation, but this can take longer if new fibre needs blowing, especially if that requires extra digging. Secondly, there's a per customer rental charge of £1.25 per month. With around 200,000 customers on IPStream that equates to £250,000 per month. Then there's a base rental for the 622Mbps central pipe of £160,000 per year and from there a £166,800 yearly rental for each 155Mbps segment that is active. 
So you have yearly charges as follows:
0Mbps of capacity - £160,000 155Mbps of capacity - £326,800 311Mbps of capacity - £493,600 466Mbps of capacity - £660,400 622Mbps of capacity - £827,200
As you can see, because of the £160,000 base rental it actually costs less per Mbps the more segments are active. It will cost less per year to have one fully active 622Mbps than have two centrals with two segments lit each. But you then have to balance that against the lead time for installing a new central pipe. As such, 1Mbps costs per month costs the following (the calculations are based on 139Mbps per segment which is the usable amount excluding overheads):
With 155Mbps of capacity - £195.92 With 311Mbps of capacity - £147.96 With 466Mbps of capacity - £131.97 With 622Mbps of capacity - £123.98
At the time of writing we have 25 active segments across 7 pipes giving us a total annual cost of:
(7 x 160,000) + (25 * 166,800) = £5,290,000
Which equates to a per Mbps cost of £126.86 per month before we consider any transit or routing costs on our own network. When you compare these prices firstly with the cost of other types of bandwidth then there is a huge price premium. But you can't really compare the IPStream costs with that of say a hosting provider or a leased line. In the case of a hosting provider you are paying to move the data from point A to point B (your server to the host's gateway to the Internet) whereas the IPStream central pipe actually connects 5000+ point As to point B (each local telephone exchange to the ISP). The price premium is actually paying for a national network that covers every ADSL enabled exchange. That includes laying undersea cabling or Microwave links in places like off-shore wales, Northern Ireland and the Scottish Highlands and Islands. It provides links to some of the smallest exchanges in the country that perhaps only serve a couple of hundred customers, some of which are probably very uneconomical. Only a handful of very small exchanges remain unavailable for Broadband and some of them are being done via other schemes like the Exchange Activate scheme. 
This little rural exchange will be unlikely to get LLU but I believe it is ADSL enabled with IPStream. LLU works differently. The ISP is free to decide how they route the traffic from each exchange to their network. This may work out as a mix of different methods. In a city with several exchanges they may rent capacity from a supplier that already operates in that area, they might put their fibre in to each exchange or link several exchanges together and aggregate the data before transferring it to their network. There's therefore no per customer month usage rental like on a BT central or the price premium for the central pipes but then with LLU you are being selective over the exchanges selected and have to provide your own means of transferring the data from each to your network. So for example, for an ISP based in Sheffield that wanted to LLU several cities in the north they might build their own network that links all the Sheffield exchanges together, pay an ISP in Manchester to link all the Manchester exchanges together and have then provide a cross country 1Gbps fibre link to Sheffield (likely as part of that ISPs existing infrastructure). Same with Leeds and Newcastle, maybe a microwave link to connect Chesterfield to Sheffield and a dedicated 100Mbps fibre link to York. There's quite a lot of planning involved and when you start adding up the figures that actually a fair cost going in, maybe not as much per Mbps as IPStream but still a fair chunk of cash and the coverage isn't as wide as IPStream. It's very difficult to estimate the cost of linking up all the exchanges because you don't necessarily need to link each one directly to your own network so it's not a case of doing number of exchanges times the cost of a 1Gbps link.
Transit and Peering
Lastly, and one which is the same on both LLU and IPStream, we have transit and peering costs. This is the cost to us of transferring data between our network and the rest of the Internet. The price of this varies depending on where the data is coming from. Let me explain further. We peer with certain ISPs over data exchanges such as LINX and LoNAP. These allow large numbers of ISPs to easily transfer data between each other and generally the cost is a small fixed fee plus the cost of the necessary hardware. We also peer directly with a number of ISPs, one of note is the BBC. There are also a number of inter-site links that connect our various data centres together. 
Of course we can't peer directly with every ISP around the world so we have a number of transit providers that charge us based on the amount of traffic transferred. We estimate that our transit and peering costs (based on the capacity of our central pipes) equates to around £20 per Mbps per month. That gives a total bandwidth cost of around £147 per month per Mbps. The actual cost here can fluctuate as it depends on what customers are doing. For example, customers accessing our web portal won't be using either our peering or transit links but will use one of the London to Sheffield inter-site links, someone going to the BBC will have their traffic go straight there via the BBC peering link, viewing Think Broadband should go via LoNAP, whilst microsoft.com may get routed over the Telia transit link. If we peered with more ISPs directly or they joined LINX or LoNAP or if customers usage shifted so they were using more data that went via the peering then that cost would decrease, similarly if more usage went via the transit links the cost would increase slightly.
Can You Reduce The Bandwidth Costs?
A number of people have suggested that ISPs set up local caches, for example with content providers like the BBC, in order to reduce costs. If the data is already being transferred across a peering link then there's going to be little if any cost saving (and maybe actually a cost increase in running the servers that make up the cache). Even if the data is being transferred over a transit link the savings would still be small in comparison to the BT Wholesale charges. The transit and peering cost make up less than 14% of the total bandwidth costs. So even moving 20% of traffic from the transit links to a local cache only saves a couple of quid per Mbps at best out of £147 and at worst saves nothing but increases data centre running costs. 
The same thing applies with multicast. Multicast can be used to more efficiently distribute content. A number of customers wanted to watch the same video stream then using multicast rather than there being one stream per customer between the content provider and the end user, where the network paths are the same there only needs to be one stream. At the moment on IPStream that would mean that if you were watching a multicast stream there would only need to be one stream between the content provider and the ISP's network and it splits into multiple feeds at central pipes. Which of course is the major problem with using multicast at the moment, the only cost saving to the ISP is the transit and/or peering costs, which as we've seen above may not represent any real savings or at best only a tiny reduction because you still have one stream per customer going across the central pipes. For multicast to be successful in the UK the stream must be split further down the line, preferably at the BT PoP but at the very least at the local exchanges. This is something we might see as part of BT's 21st Century Network upgrades which are taking place over the next few years.
The Unlimited Myth
When you look at the costs it's fairly easy to understand why the unlimited broadband deal just doesn't work. A constant speed of 1Mbps equates to around 300GB in a month and without the costs of building, staff, hardware, billing, etc. costs an IPStream ISP over £150 per month. 
Then there's the debate that not every GB/Mbps is equal. Usage isn't evenly spread across the day and across the week. Overnight, when usage is free on BBYW, most of the traffic on our network is either P2P and binary usenet downloads and very little interactive traffic (the graph to the left shows the percentage of different types of traffic between 4am and 5am across the last few months, graphs for the eveningcan be seen in the last iPlayer blog). While in the evenings it's mainly browsing, gaming and streaming and there's less non-interactive traffic. As such if ten thousand Broadband Your Way customers all started using an extra 2GB per month overnight between midnight and 8am there'd be little if any impact because there's plenty of spare capacity overnight. But the same doing that amount in the evening between 4pm and midnight when the network is busy would mean that more capacity would be required to support them (which fortunately would be paid for from the extra usage charges on BBYW). On a similar vein 1GB of binary usenet downloads isn't the same as 1GB of web browsing (not watching video). The usenet download would generally be one or more periods of sustained downloading at the maximum speed possible whilst the web browsing would be lots of very short bursts of traffic, not necessarily at the maximum possible speed. Consider a fixed capacity of say 6Mbps and you have one customer downloading from usenet, if his maximum speed is 6Mbps too then he could use all the available capacity. With browsing instead you can probably fit dozens of customers into that same capacity because that type of usage isn't as bandwidth intense. So comparing two customers that both use 100GB per month on two types of broadband product, BBYW Option 3 with 20GB at £19.99 and "unlimited broadband" from Made-up name ISP at £19.99. The customer on BBYW Option 3 might use their usage so they do 10GB during the chargeable hours and 90GB overnight, plugging those figures in 10GB equates to about £5 in bandwidth costs, another £7.50 in BT rental costs and £3 to VAT man leaving a few quid over for all the costs and a little profit. The customer on the "unlimited broadband" product has no real incentive to download overnight from the free overnight usage allowance so his usage is just spread out evenly across the day. Plugging these figures in you're up to £60 in costs for the bandwidth, BT rental and VAT without considering anything else.
Conclusion
Providing broadband in the UK doesn't come cheap and more people use the more capacity is required and thus the more it costs. If, as an ISP, your business model means that as customers use more they pay more, then so long as your charges aren't less than the extra cost for the necessary capacity your biggest headache is getting your forecasting right. 
We analyse data on a daily basis, looking at usage per hour, per product and per protocol. This let's us look at trends over days, weeks and months. The BBC iPlayer launched at Christmas and since then there's been a massive increase in streaming traffic (the graph shows streaming traffic over the past 12 months). It's probably safe to say that it wasn't unexpected to see an increase, forecasting how much and over what period is where it counts. We try and keep track of events that may cause changes in usage (whether that's an increase or a decrease) using a calendar and our discussion forums. The forum also includes some graphs showing what the actual traffic was like such as this England football game in Russia shown on Sky Sports where we saw an increase in traffic and this game against Croatia which was on BBC1 where there was a decrease in traffic. This post is a little wordy and one of the longer blogs I've written but I hope that some people have found it interesting (or even just made it this far) but it should hopefully serve as a useful reference point. My next blog, hopefully later this week, intends to go into more detail on our plans for traffic management and how that fits into our product design. There'll also be some more on our forecasting and how that ties into our capacity planning. Dave Tomlinson PlusNet Product Team