Issues to some IPs affecting everyone on the BT/Plusnet network

[#1]

Hi, I'm hoping someone can forward this to whoever has the power to fix this issue since it's network related.

I run a server hosted in Germany and recently I've been having real trouble reliably accessing it and downloading from it at a good speed. It's only happening to people running on the BT infrastructure, so Plusnet included (and I'm on a Plusnet line so also affected) as I've confirmed with a few friends who are also on BT or Plusnet as well as confirmed it doesn't happen to those who aren't.

The data center say there is no issue and that it must be ISP related, and I'd tend to agree since it doesn't happen to anyone else.

I used to get a solid 7MB/s from the server until recently (a few weeks ago) when something changed and now I might get that speed or I might get <300KB/s. Either way the connection is unstable as it hangs a lot. The same happens to others on the BT/Plusnet network. On other ISPs they can comfortably download at whatever speed their ISP can provide to them.

So how do I go about getting this obvious routing issue fixed? From what I can tell it's flapping between two different routes all the time and something somewhere is getting confused and is causing dropped packets.

Thanks

Andy

[#2]

Try running a traceroute to the server from both plusnet and non-BT ISPs, or use a tool like MTR/WinMTR to get slightly better stats.

https://github.com/White-Tiger/WinMTR

You can also try doing traceroutes from other websites to see what their path to your server is

EG https://network-tools.com/

It is also possible to see routing information for BT based on BGP here - http://lg.bt.net/

It should show where they are learning the route to your server from.

[#3]

Here are the trace routes from my connection to the server and back again.

As you can see going OUT of Plusnet/BT the connection goes through Telia, but coming back again it routes directly from Hetzner onto the Plusnet/BT network. Surely they should be routing directly both ways since they're direct peering partners?

I'm willing to bet that's the source of the issue.

HOME PLUSNET to BA

|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| pfSense.localdomain - 0 | 272 | 272 | 0 | 0 | 2 | 0 |
| 192.168.0.1 - 0 | 272 | 272 | 1 | 1 | 3 | 1 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| 129.hiper04.sheff.dial.plus.net.uk - 99 | 56 | 1 | 11 | 11 | 11 | 11 |
| be3-3100.pcn-ir01.plus.net - 0 | 272 | 272 | 11 | 11 | 23 | 13 |
| core1-BE1.colindale.ukcore.bt.net - 0 | 272 | 272 | 11 | 11 | 23 | 12 |
| core3-hu0-14-0-7.faraday.ukcore.bt.net - 0 | 272 | 272 | 12 | 12 | 27 | 13 |
| 213.137.183.32 - 0 | 272 | 272 | 12 | 12 | 40 | 12 |
| ldn-b3-link.telia.net - 0 | 272 | 272 | 11 | 13 | 114 | 13 |
| ldn-bb4-link.telia.net - 1 | 264 | 262 | 11 | 15 | 180 | 12 |
| prs-bb4-link.telia.net - 0 | 272 | 272 | 19 | 28 | 79 | 21 |
| ffm-bb4-link.telia.net - 0 | 272 | 272 | 29 | 29 | 46 | 29 |
| ffm-b4-link.telia.net - 0 | 272 | 272 | 28 | 31 | 92 | 29 |
| hetzner-ic-326013-ffm-b4.c.telia.net - 0 | 272 | 272 | 27 | 32 | 120 | 28 |
| core22.fsn1.hetzner.com - 0 | 272 | 272 | 33 | 34 | 43 | 34 |
| ex9k2.dc1.fsn1.hetzner.com - 0 | 272 | 272 | 33 | 34 | 52 | 34 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 55 | 0 | 0 | 0 | 0 | 0 |
|________________________________________________|______|______|______|______|______|______|

BA to HOME PLUSNET
|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|static.129.67.130.94.clients.your-server.de - 0 | 224 | 224 | 0 | 0 | 7 | 0 |
| core21.fsn1.hetzner.com - 0 | 223 | 223 | 0 | 0 | 22 | 0 |
| core11.nbg1.hetzner.com - 0 | 224 | 224 | 2 | 2 | 7 | 2 |
| juniper5.dc2.nbg1.hetzner.com - 0 | 223 | 223 | 2 | 2 | 11 | 10 |
| ae5-710.nbg40.core-backbone.com - 0 | 224 | 224 | 2 | 3 | 8 | 3 |
| ae1-2003.fra20.core-backbone.com - 0 | 223 | 223 | 5 | 5 | 42 | 5 |
| t2c3-xe-11-1-1-0.de-fra.eu.bt.net - 0 | 224 | 224 | 6 | 6 | 45 | 6 |
| 166-49-195-40.eu.bt.net - 0 | 223 | 223 | 21 | 21 | 40 | 21 |
| 166-49-211-231.eu.bt.net - 0 | 223 | 223 | 21 | 21 | 25 | 22 |
|core2-hu0-10-0-1.colindale.ukcore.bt.net - 0 | 223 | 223 | 20 | 20 | 27 | 20 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
| Request timed out. - 100 | 46 | 0 | 0 | 0 | 0 | 0 |
|________________________________________________|______|______|______|______|______|______|

[#4]

Hi @andrewwhyman, I've asked a colleague on our networks team if they can look in to this further to give some insight.

The initial response is that it doesn't look like packet loss, but they want to look in to the routing more after discussing this with someone else.

I'm sorry this isn't much of an answer right now, hopefully I'll have more news on this next week for you.

[#5]

Thanks HarryB, that's good news at least since it didn't look right to be using Telia as a route one way but direct the other.

Looking forward to your response next week.

Andy

[#6]

@HarryB any news on this yet? Cheers.

[#7]

Hi @andrewwhyman

We've raised this with the Internet team within BT group. They've taken a look at your traceroutes and looked at the routing/links within the BT Networks.

From Plusnet/BT to the site routing is via Telia, on the return it is via Core Backbone, not direct.

BT don't peer with Hetzner hence these routes, asymmetry is quite normal and does not indicate an issue, just BT & Hetzner prefer different peers.

The traceroutes are not showing high-latency or any packet loss and BT are not seeing any issues with their network to/from the peering locations to explain the issues you are seeing.

Based on the above info the issue could be outside of BT/Plusnet and hence not something we can easily resolve.

Have you raised this with your hosting provider? They should be able to check the links to Telia or Core Backbone from their side too.

It may also be worth keeping an eye on the traceroutes when you see an issue as a significant jump in latency or packet loss on a hop might help us pinpoint the issue.

[#8]

Core Backbone from what I understand is actually Hetzner, it's their parent company so essentially their network.

My concern is that coming from the server to Plusnet it is directly via Core Backbone, but the other way it hops onto Telia and then Hetzner rather than going through Core Backbone and then Hetzner. It makes no sense to go via Telia if Plusnet/BT already link with Core Backbone if the traffic the other way uses it too.

I'm sure it used to be like this until recently and since then I now suffer all kinds of issues because of the odd routing such as my maximum speed per connection is now about 1.5MB/s not the 7MB/s I used to get and I get consistent remote desktop dropouts. It only happens to servers hosted with Hetzner.

[#9]

Core Backbone from what I understand is actually Hetzner, it's their parent company so essentially their network.

My concern is that coming from the server to Plusnet it is directly via Core Backbone, but the other way it hops onto Telia and then Hetzner rather than going through Core Backbone and then Hetzner

As per my previous response:

BT don't peer with Hetzner hence these routes, asymmetry is quite normal and does not indicate an issue, just BT & Hetzner prefer different peers.

Have you raised this with your hosting provider? They should be able to check the links to Telia or Core Backbone from their side too.

[#10]

I have asked the datacenter a few weeks ago and they said they could not see any issue from their side and suggested it was a server configuration issue, since confirmed otherwise by trying another server within their network to the same end.

Does Plusnet/BT peer with Core Backbone?

Thanks for your replies so far