BT Major Broadband failure

[#1]

I was suffering failed connection several times during the day:

http://www.telegraph.co.uk/technology/2016/07/20/bt-down-broadband-service-suffers-major-outage-acro...

[#2]

And I think it (or something similar) just happened again about 7:49 am this morning and continuing...

[#3]

~ 40% Packet loss at BT on routing to destinations via Cogent:

[root@netflow ~]# mtr 130.117.50.74

                                        My traceroute  [v0.85]               
netflow (0.0.0.0)                                             Thu Jul 21 08:53:20 2016
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                               Packets               Pings
 Host                                                        Loss% Drop   Snt   Avg  Best  Wrst StDev
 1. 192.168.1.254                                             0.0%    0   124   3.1   0.7 268.0  24.0
 2. lo0-central10.ptn-ag03.plus.net                           0.0%    0   124  21.2  14.5 191.1  20.4
 3. link-a-central10.ptn-gw01.plus.net                        0.0%    0   124  16.2  14.4 114.2   9.0
 4. xe-5-3-0.ptw-cr01.plus.net                                0.0%    0   124  15.3  14.2  37.2   2.6
 5. ae1.pcl-cr01.plus.net                                     0.0%    0   124  17.4  14.2  63.3   8.6
 6. 195.99.126.96                                             0.0%    0   124  15.8  14.6  23.5   1.0
 7. core3-te0-19-0-26.faraday.ukcore.bt.net                   0.0%    0   124  15.7  14.9  17.8   0.4
 8. 213.137.183.38                                            0.0%    0   124  15.6  14.5  35.9   2.6
 9. t2c3-xe-1-0-2-0.uk-lon1.eu.bt.net                        36.6%   45   123  96.7  73.3 653.2  66.5
10. t2c4-xe-1-3-3-1.uk-lon1.eu.bt.net                        30.1%   37   123 101.0  73.5 576.1  57.3
11. hu0-1-0-0.ccr22.lon01.atlas.cogentco.com                 33.6%   41   123  80.5  73.6 499.1  47.2
12. be2870.ccr41.lon13.atlas.cogentco.com                    29.5%   36   123  78.9  73.7 422.1  37.5
13. be2111.ccr21.man01.atlas.cogentco.com                    34.4%   42   123  80.5  79.6  83.1   0.4

[#4]

Yep...having the same issues here (some websites working fine, some slow, some non-functioning). Strange that the same issue starts at the same time, roughly, 2 days in a row! Can't imagine it is a power failure at the same place again surely?

[#5]

From http://www.voipfonestatus.co.uk/incidents/qh874zmmzx3y

BT have confirmed there is a major outage in Telehouse North, London affecting our interconnect. Staff are on-site to resolve the issue as quickly as possible. We will keep you updated as soon as we have more information.

So not good

[#6]

As per post #4 -some sites unreachable,some working perfectly normally -come on BT !!

[#7]

I have the same issue here again today - same time of the morning as well.

This is beginning to annoy me - working at home now and being the company's server administrator makes it impossible to do my job.

BT get criticised for poor performance and ironically two days on the trot have major problems for a lot of people!

Good thing the prices are going up in September though - because of "improvements" to our service - I've certainly yet to see them and am disgusted at paying for another increase!

Going to try changing DNS to see if it helps, but this shouldn't have to be done!

[#8]

@psings

From http://www.voipfonestatus.co.uk/incidents/qh874zmmzx3y

BT have confirmed there is a major outage in Telehouse North, London affecting our interconnect. Staff are on-site to resolve the issue as quickly as possible. We will keep you updated as soon as we have more information.

Posted 21 minutes ago. Jul 21, 2016 - 09:00 BST

[#9]

Already tried changing to Google DNS -but as per yesterday ,no improvement!

[#10]

Unlikely changing DNS will offer any help whatsoever, unless your DNS server(s) happen to be on the route(s) affected.

Essentially this issue looks to me like overload on inadequately sized redundant routing equipment due to router failure (power loss or otherwise), for a portion of network traffic routed through that equipment which will be to various destinations, other destinations not using that equipment will not be affected.

That's all IMHO of course, whether we hear the real truth it's yet to be seen... Yesterday I saw an issue where traffic to a single IP was fine but other IPs on the same /21 network block were not! that didn't make sense as a router shouldn't have a BGP table full of single IPs, only networks (it would explain an overload if it were actually trying to store individual /32 addresses in its routing table!) but weird all the same.

[#11]

One of our internet connection partners in the docklands has suffered a substantial power failure. This is affecting BT and other providers.

We are redirecting traffic to reduce the impact on customers. Engineers are on site and are fixing the problem.

Another power failure?  What are the chances two days in a row ad the same time of day?  Where's the redundant and backup supplies?

Also shocked to see how long it's taken Plusnet to start re-routing traffic!

[#12]

Two outages within days from the same provider - what is going on? Where is the redundancy and it sounds like some of the routers are not programmed sensibly or the outage really is serious. I note that a BT 'spokesperson' claimed the first outage was repaired in 20 minutes and 80% of traffic continued. I think the BT spokesperson needs to go and look at a few logs. Perhaps BT even believe what they say.

[#13]

One of my reserve browsers is TORR and that performed flawlessly on URL's that weren't rendering on IE11;

I use the Google DNS settings on my PC's as of course the HubOne router doesn't allow to make change at router level

[#14]

Apparently ongoing problems on Telehouse North;

http://www.telegraph.co.uk/technology/2016/07/21/bt-down-broadband-internet-service-suffers-second-d...

[#15]

So the BT spoekesperson quotes:

"The issue, which affected BT and other providers, was due to Telehouse North, one of our internet connection partners in the docklands, suffering power issues. The issue affected less than five percent of our customers’ internet usage, and BT took immediate action to minimise these issues by redirecting traffic."

Well, I couldn't access a whole range of sites this morning for at least an hour or so. Tthis BT statement is just plain wrong and the re-directing wasn't helping me much. It wasn't just my home that was affected yesterday but my local gym that uses the Cloud for entry gates, gym equipment and catering ordering (there's a moral there ref use of the Cloud). Apparently BT say the fault was a circuit breaker at their partner - Hmm I wonder? If so then this partner has a big problem with redundancy and reliability. I hope Plusnet are able to lean on BT a bit to improve system resilience.