Possible routing issue? Can't connect through to specific site.

ErTnEc — Thu, 28 Jul 2022 12:58:48 GMT

I'm trying to get to the bottom of a possible routing issue where I can't seem to connect through to our local hospitals NHS website. I've noticed this has been an issue a few times before, and I thought nothing of it until recently when I was trying to get some specific information.

The website in question is https://www.boltonft.nhs.uk/

Any attempt to connect to it (curl, chrome, any other browser) simply results in a timeout. Here it the output from curl and traceroute when ran within my network:

root@raspi01 ~ # traceroute -I www.boltonft.nhs.uk traceroute to www.boltonft.nhs.uk (213.104.98.149), 30 hops max, 60 byte packets 1 254.core.plus.net (195.166.130.254) 13.194 ms 13.203 ms 13.177 ms 2 84.93.253.115 (84.93.253.115) 13.468 ms 13.700 ms 13.707 ms 3 core1-BE1.southbank.ukcore.bt.net (195.99.125.130) 13.151 ms 13.191 ms 13.664 ms 4 peer3-et-0-0-2.redbus.ukcore.bt.net (62.172.103.240) 13.754 ms 13.777 ms 13.781 ms 5 109.159.253.63 (109.159.253.63) 20.649 ms 20.696 ms 20.733 ms 6 * * * 7 bagu-core-2a-ae10-0.network.virginmedia.net (62.254.84.2) 25.504 ms 23.142 ms 23.385 ms 8 bagu-metnet-3b-lag-56.network.virginmedia.net (82.8.124.178) 23.168 ms 23.247 ms 23.272 ms 9 bnft-bl4-ia1.network.virginmedia.net (213.104.213.126) 24.374 ms 24.743 ms 24.755 ms 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * root@raspi01 ~ # curl -v -m30 https://www.boltonft.nhs.uk/ -o out * Expire in 0 ms for 6 (transfer 0x671950) * Expire in 30000 ms for 8 (transfer 0x671950) * Expire in 1 ms for 1 (transfer 0x671950) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 1 ms for 1 (transfer 0x671950) * Expire in 2 ms for 1 (transfer 0x671950) * Expire in 1 ms for 1 (transfer 0x671950) * Expire in 1 ms for 1 (transfer 0x671950) * Expire in 1 ms for 1 (transfer 0x671950) * Trying 213.104.98.149... * TCP_NODELAY set * Expire in 200 ms for 4 (transfer 0x671950) 0 0 0 0 0 0 0 0 --:--:-- 0:00:29 --:--:-- 0* Connection timed out after 30001 milliseconds 0 0 0 0 0 0 0 0 --:--:-- 0:00:30 --:--:-- 0 * Closing connection 0 curl: (28) Connection timed out after 30001 milliseconds

If I access this via a different connection (in this case route out via VPN, then the site returns just fine with no issues:

root@raspi01 ~ # traceroute -T www.boltonft.nhs.uk traceroute to www.boltonft.nhs.uk (213.104.98.149), 30 hops max, 60 byte packets 1 10.35.0.1 (10.35.0.1) 17.397 ms 17.383 ms 17.502 ms 2 te-3-3-4006.pe3.man4.uk.m247.com (217.64.114.161) 18.457 ms 19.779 ms 19.908 ms 3 xe-1-2-1-0.core1.man4.uk.m247.com (83.97.21.144) 18.941 ms vlan2902.bb1.fra2.de.m247.com (82.102.29.128) 20.146 ms xe-2-1-0-0.core1.man4.uk.m247.com (77.243.185.12) 20.035 ms 4 te-12-3-0.core-dc2.man4.uk.m247.com (83.97.21.70) 47.011 ms te-13-4-0.core-dc2.man4.uk.m247.com (77.243.176.47) 47.090 ms te-12-3-0.core-dc2.man4.uk.m247.com (83.97.21.70) 46.992 ms 5 te-5-8-0.bb1.man2.uk.m247.com (77.243.185.137) 20.013 ms 20.061 ms te-6-5-0.bb1.man2.uk.m247.com (77.243.185.1) 20.102 ms 6 tcma-ic-2-xe-210-0-0.network.virginmedia.net (212.250.14.189) 20.118 ms 18.890 ms 18.910 ms 7 * * * 8 bagu-core-2a-ae10-0.network.virginmedia.net (62.254.84.2) 20.681 ms 20.534 ms 20.497 ms 9 bagu-metnet-3b-lag-56.network.virginmedia.net (82.8.124.178) 20.556 ms 20.569 ms 20.592 ms 10 bnft-bl4-ia1.network.virginmedia.net (213.104.213.126) 20.860 ms 20.891 ms 20.795 ms 11 91-187-250-212.static.virginm.net (212.250.187.91) 20.805 ms 25.048 ms 25.142 ms 12 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 24.953 ms 25.062 ms 25.025 ms root@raspi01 ~ # curl -v -m30 https://www.boltonft.nhs.uk/ -o out * Expire in 0 ms for 6 (transfer 0xa74950) * Expire in 30000 ms for 8 (transfer 0xa74950) * Expire in 1 ms for 1 (transfer 0xa74950) % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Expire in 1 ms for 1 (transfer 0xa74950) * Expire in 2 ms for 1 (transfer 0xa74950) * Expire in 1 ms for 1 (transfer 0xa74950) * Expire in 1 ms for 1 (transfer 0xa74950) * Expire in 1 ms for 1 (transfer 0xa74950) * Trying 213.104.98.149... * TCP_NODELAY set * Expire in 200 ms for 4 (transfer 0xa74950) * Connected to www.boltonft.nhs.uk (213.104.98.149) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.3 (IN), TLS handshake, Server hello (2): { [122 bytes data] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): { [21 bytes data] * TLSv1.3 (IN), TLS handshake, Certificate (11): { [3030 bytes data] * TLSv1.3 (IN), TLS handshake, CERT verify (15): { [264 bytes data] * TLSv1.3 (IN), TLS handshake, Finished (20): { [52 bytes data] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): } [1 bytes data] * TLSv1.3 (OUT), TLS handshake, Finished (20): } [52 bytes data] * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use http/1.1 * Server certificate: * subject: C=GB; L=Bolton; O=Bolton NHS Foundation Trust; CN=*.boltonft.nhs.uk * start date: Jan 7 00:00:00 2022 GMT * expire date: Jan 5 23:59:59 2023 GMT * subjectAltName: host "www.boltonft.nhs.uk" matched cert's "*.boltonft.nhs.uk" * issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1 * SSL certificate verify ok. } [5 bytes data] > GET / HTTP/1.1 > Host: www.boltonft.nhs.uk > User-Agent: curl/7.64.0 > Accept: */* > { [5 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [265 bytes data] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): { [265 bytes data] * old SSL session ID is stale, removing { [5 bytes data] < HTTP/1.1 200 OK < Date: Thu, 28 Jul 2022 12:39:49 GMT < Server: Apache/2.4.41 (Ubuntu) < Link: <https://www.boltonft.nhs.uk/>; rel=shortlink < Vary: Accept-Encoding < Transfer-Encoding: chunked < Content-Type: text/html; charset=UTF-8 < { [6 bytes data] 100 278k 0 278k 0 0 721k 0 --:--:-- --:--:-- --:--:-- 723k * Connection #0 to host www.boltonft.nhs.uk left intact

Weirdly, I have to perform ICMP traceroute when running through Plusnet otherwise the TCP one used earlier returns the same hop multiple times:

root@raspi01 ~ # traceroute -T www.boltonft.nhs.uk traceroute to www.boltonft.nhs.uk (213.104.98.149), 30 hops max, 60 byte packets 1 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 10.311 ms 10.278 ms 10.556 ms 2 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 11.169 ms 11.113 ms 11.372 ms 3 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 11.366 ms 11.282 ms 10.925 ms 4 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 11.473 ms 11.168 ms 11.137 ms 5 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 20.637 ms 18.463 ms 18.378 ms 6 * * * 7 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 28.309 ms * 28.130 ms 8 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 28.190 ms 28.067 ms 28.014 ms 9 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 28.021 ms 27.969 ms 27.915 ms 10 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 27.915 ms 27.904 ms 27.809 ms 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * *

So yes, a bit of a loss but also puzzled at the odd behaviour of running traceroute over tcp when going out via Plusnet. This behaviour appears to happen to every endpoint via tcp.

I've also tethered to my mobile phones data connection and ran the same tests, and they worked just fine including showing the correct hops via traceroute when going via tcp.

For reference, I run pfSense which has multiple gateways configured:

WAN -> Plusnet
OVPN -> Via external VPN Provider
HEV -> Hurricane Electric IPv6 Gateway (not used for this test)

To test the connection, I change the outbound firewall rule to send traffic via the OpenVPN gateway instead of the Plusnet gateway for a particular destination (in this case the IP in which resolves against the Bolton NFT Trust website).

Re: Possible routing issue? Can't connect through to specific site.

Champnet — Thu, 28 Jul 2022 17:41:55 GMT

I can get to the website but using tracert boltonnft.nhs.uk is not relying to pings.

Re: Possible routing issue? Can't connect through to specific site.

bobpullen — Thu, 28 Jul 2022 20:59:45 GMT

What public IP address are you assigned during the times you're having problems?

I've just tried from an address in the 80.229.0.0/16 range and everything is fine: -

~$ sudo traceroute -T -p80 www.boltonft.nhs.uk traceroute to www.boltonft.nhs.uk (213.104.98.149), 30 hops max, 60 byte packets 1 100.115.92.193 (100.115.92.193) 0.037 ms 0.010 ms 0.007 ms 2 100.115.92.25 (100.115.92.25) 0.175 ms 0.127 ms 0.118 ms 3 home.gateway (192.168.1.254) 4.074 ms 3.973 ms 3.871 ms 4 195.166.130.255 (195.166.130.255) 9.616 ms 9.528 ms 9.421 ms 5 84.93.253.123 (84.93.253.123) 9.328 ms 84.93.253.127 (84.93.253.127) 9.242 ms 9.102 ms 6 core1-BE1.southbank.ukcore.bt.net (195.99.125.130) 10.986 ms 195.99.125.142 (195.99.125.142) 11.118 ms 195.99.125.134 (195.99.125.134) 10.972 ms 7 peer7-et-3-0-5.telehouse.ukcore.bt.net (109.159.252.188) 10.874 ms 10.799 ms peer3-et7-0-6.redbus.ukcore.bt.net (194.72.16.100) 14.565 ms 8 109.159.253.101 (109.159.253.101) 16.546 ms 16.547 ms 16.423 ms 9 * * * 10 * bagu-core-2a-ae10-0.network.virginmedia.net (62.254.84.2) 19.036 ms 18.953 ms 11 bagu-metnet-3b-lag-56.network.virginmedia.net (82.8.124.178) 18.845 ms bagu-core-2a-ae10-0.network.virginmedia.net (62.254.84.2) 18.674 ms bagu-metnet-3b-lag-56.network.virginmedia.net (82.8.124.178) 18.757 ms 12 bagu-metnet-3b-lag-56.network.virginmedia.net (82.8.124.178) 17.685 ms bnft-bl4-ia1.network.virginmedia.net (213.104.213.126) 19.344 ms bagu-metnet-3b-lag-56.network.virginmedia.net (82.8.124.178) 18.961 ms 13 bnft-bl4-ia1.network.virginmedia.net (213.104.213.126) 18.871 ms 18.756 ms 91-187-250-212.static.virginm.net (212.250.187.91) 19.810 ms 14 149.98-104-213.static.virginmediabusiness.co.uk (213.104.98.149) 19.388 ms 91-187-250-212.static.virginm.net (212.250.187.91) 18.776 ms 18.613 ms

Re: Possible routing issue? Can't connect through to specific site.

ErTnEc — Fri, 05 Aug 2022 08:56:44 GMT

My public IP is 81.174.148.33, and it's a statically assigned one.

Re: Possible routing issue? Can't connect through to specific site.

stephenw10 — Wed, 10 Aug 2022 13:26:42 GMT

@ErTnEc wrote:

Weirdly, I have to perform ICMP traceroute when running through Plusnet otherwise the TCP one used earlier returns the same hop multiple times:

Do you have outbound fq-codel shaping enabled?

Do you mean TCP or UDP there?

Sounds like you may be hitting something similar to this: https://redmine.pfsense.org/issues/9263
Though that would not prevent you accessing the site in general.

Traceroute fails for the last hop for me too just as you see it but I can still access the site.

topic Re: Possible routing issue? Can't connect through to specific site. in Broadband

Possible routing issue? Can't connect through to specific site.

Re: Possible routing issue? Can't connect through to specific site.

Re: Possible routing issue? Can't connect through to specific site.

Re: Possible routing issue? Can't connect through to specific site.

Re: Possible routing issue? Can't connect through to specific site.