Extremely slow response from US server site via PN BB - no problem over 4G : Network bottleneck?

[#1]

Slightly odd issue.

I regularly (several times a day) fetch data from an American university server. 

The data is got over https using API access into their server, via a small Python program. We literally fetch only a few hundred bytes of data each time.

The University claim that their response time, from receiving a request to transmitting the response, is about 3 seconds. I have no reason to doubt this. 

Over my PN connection (standard ADSL, no speed problems, get about 18MB/s), the response time for this activity, start to end, is never less than 13 seconds, and can be up to a 70 seconds or more.

If however I set my 'phone up as a WiFi hotspot and connect via 4G (Tesco/O2), and thus bypass the PN network altogether, I consistently get response times around 5-7 seconds.

Doesn't sound a big problem, but this single data fetch is on a critical path as part of a larger function, and it slows the entire process down by a factor of at least 7. 

So, I conclude there's a bottleneck somewhere. 

Can one of the PN network team pick this up please? It's driving me a bit nutty.

I can provide some MTR (traceroute) or ping or tcpdump data,  or whatever's needed - I'm on Linux btw.

[#2]

Hi PaulH, yes we'd just need to see some results of a traceroute to be able so start looking into this for you.

[#3]

thanks very much Adam.

Here's an MTR trace from last night.

I've PM'd you the target address separately.

[#4]

Thanks for that, hops 7, 8 and 9 definitely look problematic. Could you run a tracert to the same destination over the 4g connection just for comparison please?

[#5]

Sorry Adam only just seen this.

Unfortunately, I can't get a traceroute when I'm on 4G  - or rather I can, but it only returns the first hop.

After a bit of googling, it does seem this is a often a problem with 4G  telecoms providers - variously attributed to CGNAT, ICMP blocking or possibly Brexit. 

There doesn't seem much I can do about this, sorry

[#6]

Hi there.

Could you run a traceroute to pingbox1.thinkbroadband.com to compare?

[#7]

[#8]

No worries, the response times there look a lot better. It does look like there maybe some kind of routing issue. I want to take a look at this with one of our net ops team so bear with me and I'll get back to you ASAP when we've been able to review it. 

[#9]

Thanks Adam.

Didn't expect there to be an instant answer as it does seem to be off at the edge of the known universe