topic Re: DNS timing out from all the Plus servers in Broadband

DNS timing out from all the Plus servers

simon_mcqueen — Fri, 09 Dec 2022 15:18:14 GMT

I'm getting timeouts from the Plus DNS servers and hence dreadfully sluggish responsiveness with pretty much everything. I've tried 212.159.6.9 and 10 and 212.159.13.49 and 50 and they are all look to be the same, and look like:

C:\Users\SimonMcQ>nslookup www.msn.com 212.159.6.10 Server: cdns02.plus.net Address: 212.159.6.10 DNS request timed out. timeout was 2 seconds. DNS request timed out. timeout was 2 seconds. Non-authoritative answer: Name: a-0003.a-msedge.net Address: 204.79.197.203 Aliases: www.msn.com www-msn-com.a-0003.a-msedge.net

I note the Plusnet Hub One (which is what Wor Lass seems to have here) doesn't let you set an alternate DNS. I've tried turning it off and on and again.

Is there any route to get this investigated? The connection settings just now are:

Broadband username: graciegirlpixie@plusdsl.net TCP/IP settings Broadband network IP address: 51.9.179.41 Default gateway: 172.16.19.109 Primary DNS: 212.159.6.9 Secondary DNS: 212.159.6.10

Is there possibly a 'Hub Two' or anything like that that my other half might be able to request that would let me set the DNS to something else that works better?

TIA for any help anyone can offer!

Re: DNS timing out from all the Plus servers

simon_mcqueen — Fri, 09 Dec 2022 15:25:30 GMT

... sorry - should of course have made clear other DNSs do not appear to be timing out similarly from the same vantage point. See for e.g:

C:\Users\SimonMcQ>nslookup www.google.com 8.8.8.8 Server: dns.google Address: 8.8.8.8 Non-authoritative answer: Name: www.google.com Addresses: 2a00:1450:4009:81e::2004 172.217.169.4

Re: DNS timing out from all the Plus servers

seebee — Fri, 09 Dec 2022 17:40:29 GMT

I've not looked for a while, but I often used to see DNS losses with the 4 PlusNet DNS servers, but not with the 2 PN SafeGuard DNS servers (which I typically use) nor public ones like 8.8.8.8 / 1.1.1.1 / 9.9.9.9 etc

For example with this list of DNS servers:

pi@rPi3b:~ $ cat testdns.txt # The two PN SafeGuard 213.120.234.42 213.120.234.38 # a Google 8.8.8.8 # The four PlusNet 212.159.6.10 212.159.13.49 212.159.6.9 212.159.13.50

At random times I might see some or no loss with the 4 PlusNet servers, but never with SafeGuard or Public servers.
Eg using a Raspberry Pi on an ethernet cable and "dnseval" from https://dnsdiag.org/ I have just seen:

pi@rPi3b:~ $ dnseval -f testdns.txt bbc.co.uk. server avg(ms) min(ms) max(ms) stddev(ms) lost(%) ttl flags ----------------------------------------------------------------------------------------------------------- 213.120.234.42 14.786 13.638 16.265 0.816 %0 197 QR -- -- RD RA -- -- 213.120.234.38 14.064 13.451 14.792 0.457 %0 135 QR -- -- RD RA -- -- 8.8.8.8 14.668 14.159 15.491 0.495 %0 115 QR -- -- RD RA -- -- 212.159.6.10 38.829 13.715 176.905 54.580 %10 149 QR -- -- RD RA -- -- 212.159.13.49 22.663 12.845 82.081 24.044 %20 145 QR -- -- RD RA -- -- 212.159.6.9 14.257 12.833 19.940 2.112 %0 145 QR -- -- RD RA -- -- 212.159.13.50 16.823 13.568 26.457 4.426 %10 145 QR -- -- RD RA -- --

and

pi@rPi3b:~ $ dnseval -f testdns.txt bbc.co.uk. server avg(ms) min(ms) max(ms) stddev(ms) lost(%) ttl flags ----------------------------------------------------------------------------------------------------------- 213.120.234.42 17.798 13.546 38.783 8.445 %0 86 QR -- -- RD RA -- -- 213.120.234.38 13.540 13.223 13.979 0.257 %0 72 QR -- -- RD RA -- -- 8.8.8.8 14.433 13.744 15.312 0.545 %0 292 QR -- -- RD RA -- -- 212.159.6.10 14.904 13.639 18.238 1.630 %30 146 QR -- -- RD RA -- -- 212.159.13.49 14.536 13.477 16.467 1.034 %30 138 QR -- -- RD RA -- -- 212.159.6.9 43.527 12.875 195.738 58.600 %10 135 QR -- -- RD RA -- -- 212.159.13.50 14.745 13.695 17.180 1.163 %10 133 QR -- -- RD RA -- --

and

pi@rPi3b:~ $ dnseval -f testdns.txt bbc.co.uk. server avg(ms) min(ms) max(ms) stddev(ms) lost(%) ttl flags ----------------------------------------------------------------------------------------------------------- 213.120.234.42 14.223 13.199 15.545 0.800 %0 300 QR -- -- RD RA -- -- 213.120.234.38 13.974 13.466 14.392 0.277 %0 263 QR -- -- RD RA -- -- 8.8.8.8 14.013 13.700 14.754 0.318 %0 193 QR -- -- RD RA -- -- 212.159.6.10 62.386 13.052 218.889 72.698 %10 122 QR -- -- RD RA -- -- 212.159.13.49 21.716 13.067 83.026 21.591 %0 60 QR -- -- RD RA -- -- 212.159.6.9 24.804 13.292 65.956 16.796 %0 60 QR -- -- RD RA -- -- 212.159.13.50 17.453 13.345 38.413 7.703 %0 121 QR -- -- RD RA -- --

So some loss at some times with the 4 PlusNet servers -the percentages are high as I'm only sending 10 packets, but if I try 100 packets for example I might see 1% to 3% loss with the 4 PlusNet servers, but zero with SafeGuard and others.

@bobpullen Sorry to bother you, but I think you are the network expert, so can you see similar or have any idea why it only seems to happen with the 4 normal DNS servers?

(Sorry for the messy text, but I've not worked out how to format wide pastes like this!)

Re: DNS timing out from all the Plus servers

bobpullen — Fri, 09 Dec 2022 18:11:36 GMT

@seebee - I do see similar on my own monitoring, however it seems to have been a relatively new development (last few months). There is a larger piece of work going on behind the scenes that I expect will make the issue 'go away', however I can see if somebody can take a look in the interim.

Remind me next week.

Edit: e.g. this is 12 DNS requests every 300s on my own network: -

Re: DNS timing out from all the Plus servers

seebee — Sat, 10 Dec 2022 12:12:26 GMT

Regarding my messy pastes above - there is a fix to stop long lines from wrapping here from @Townman

Re: DNS timing out from all the Plus servers

seebee — Tue, 13 Dec 2022 09:59:04 GMT

@bobpullen Hope this isn't too early for a reminder

I use smokeping to monitor pings, but I'd never tried it with DNS lookups, so I had a go and I saw overnight between 1.0% and 2.8% loss with the 4 normal DNS, and zero with the 2 SafeGuard ones.

(Raspberry Pi3b, Ethernet, ADSL, IP in 51.9.x.x)

Re: DNS timing out from all the Plus servers

bobpullen — Tue, 13 Dec 2022 13:09:52 GMT

@seebee - thanks, now raised internally under ref: IMT-8230.

Edit: to clarify, I also have a direct comparison in my own monitoring and my observations match @seebee's - started around September time.

Re: DNS timing out from all the Plus servers

HPsauce — Tue, 13 Dec 2022 12:49:03 GMT

I'm intrigued by this as I've seen some odd delays recently, especially receiving emails, specifically very slow finding and talking to email servers (not PlusNets) of which I use a number at various services.

It's particularly noticeable, probably because of how send/receive works, on my iPhone.

Sometimes at home I just give up, turn off my phone's WiFi and use 4G. This happens to have a similar base speed in my area to my PN connection. But it works so much better!

Re: DNS timing out from all the Plus servers

Townman — Tue, 13 Dec 2022 12:51:17 GMT

@seebee

Thank you for the work here - this provides objective metrics to matters of concern seen before, which I have not been able (had time to) measure. This is a subject which has been under discussion with Bob over recent weeks.

Re: DNS timing out from all the Plus servers

simon_mcqueen — Tue, 13 Dec 2022 18:38:10 GMT

Thanks to all, particularly for the replies which I've read with interest.

My original problem case was that I rocked up to my partners flat (which has the Plus connection) with my work issued Windows 10 laptop and connected it to her network as I've done on many previous occasions (for e.g 16-18 November when all seemed to work fine). Prior to this time it had been connected elsewhere (my home) and working fine. I'd speculate this network switch could well cause a bit of a cache clearing event, surely starting with the Windows DNS one but maybe also some or all the browser content ones. In any event, first time page loads in Chrome of typical pages (which means the massively heavyweight cross-site Javascript laden monstrosities that make up almost all the modern www) were very noticeably sluggish right across the board. Lots of status bar "resolving..." and frequently Chrome identifying that pages were unresponsive and asking if I wanted to wait or kill them.

Having checked that connectivity and speed wasn't an issue (all fine) I suspected name resolution might be. The nslookups seemed to confirm minimally that things in that area weren't happening reliably/quickly so I figured I'd ask about it.

I'm away from where I can test to see if the issue persists unfortunately but (*iff* this wasn't a localised problem affecting me) the impact *might* be visible if one were to flush the machine DNS and then the browser caches (or maybe use an anonymous browser tab/session) and then follow up with some generic browsing. If it's still happening but you haven't had a lot of complaints I could only imagine that caching must be cushioning most users from it because it was startlingly bad.

I'll be back in Glasgow on the 21st so I'll test all this myself then and report back.

Thanks again to all who took the time to read!

Re: DNS timing out from all the Plus servers

bobpullen — Tue, 13 Dec 2022 18:40:23 GMT

@simon_mcqueen - suspect there was something else contributing to the problem if it was so blatant. What's being observed in mine and others monitoring though certainly won't be helping.

Re: DNS timing out from all the Plus servers

bobpullen — Thu, 05 Jan 2023 13:42:15 GMT

@simon_mcqueen & @seebee - how are things looking now?

There was some work carried out over the festive period and my personal monitoring looks much better now, displaying zero loss and decent round trip times for the Plusnet DNS resolvers.

Re: DNS timing out from all the Plus servers

seebee — Thu, 05 Jan 2023 16:22:02 GMT

@bobpullen yep, looking good for me too, zero DNS failures on all the 6 DNS servers, for the last 10 days. Thanks.

Re: DNS timing out from all the Plus servers

seebee — Thu, 05 Jan 2023 16:52:28 GMT

@bobpullen

On a tangent - the SD card in my Raspberry Pi failed so I rebuilt it over Xmas. "dig" (which smokeping calls) is giving me results SOMETIMES of 9ms DNS resolution time, which I'd think impossible as I've always seen my ADSL line have an inherent delay of about 13ms (eg pinging gstatic.com or 1.1.1.1 etc).

I'd be tempted to say its something in the recent version of the rPi build. Don't suppose you see "too short" DNS times in your monitoring?

$ dig @212.159.13.49 gstatic.com.

; <<>> DiG 9.16.33-Raspbian <<>> @212.159.13.49 gstatic.com.
...
gstatic.com. 68 IN A 216.58.212.195
;; Query time: 9 msec
;; SERVER: 212.159.13.49#53(212.159.13.49)
...

then

$ dig @212.159.13.49 gstatic.com.

; <<>> DiG 9.16.33-Raspbian <<>> @212.159.13.49 gstatic.com.
...

gstatic.com. 180 IN A 142.250.187.195
;; Query time: 19 msec
;; SERVER: 212.159.13.49#53(212.159.13.49)
...

yet

$ ping -c 100 -q 212.159.13.49
PING 212.159.13.49 (212.159.13.49) 56(84) bytes of data.

--- 212.159.13.49 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 99155ms
rtt min/avg/max/mdev = 12.278/13.911/50.144/3.969 ms

So the smokeping charts are a bit "castellated"!

Zero loss though, which is the important bit.

Re: DNS timing out from all the Plus servers

seebee — Fri, 06 Jan 2023 10:46:12 GMT

After looking a bit harder, I'm pretty sure this is a problem with my "dig" command.

It is not related to PN DNS servers, I see the same with 1.1.1.1 for example.

$ for n in {1..30}}; do dig @1.1.1.1 gstatic.com ; done | grep -F time ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 9 msec ;; Query time: 9 msec ;; Query time: 19 msec ;; Query time: 9 msec ;; Query time: 9 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 9 msec ;; Query time: 9 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 9 msec ;; Query time: 9 msec ;; Query time: 19 msec ;; Query time: 19 msec ;; Query time: 9 msec ;; Query time: 19 msec

So dig is only telling me 9ms (which I think is impossible) or 19ms.

If I use the "-u" option to dig, to make it give answers in microseconds instead of milliseconds, it seems to give much more believable results:

$ for n in {1..30}}; do dig -u @1.1.1.1 gstatic.com ; done | grep -F time ;; Query time: 19334 usec ;; Query time: 18390 usec ;; Query time: 18704 usec ;; Query time: 19279 usec ;; Query time: 18448 usec ;; Query time: 18586 usec ;; Query time: 19222 usec ;; Query time: 18407 usec ;; Query time: 18830 usec ;; Query time: 19326 usec ;; Query time: 13941 usec ;; Query time: 15118 usec ;; Query time: 14309 usec ;; Query time: 14134 usec ;; Query time: 14326 usec ;; Query time: 19068 usec ;; Query time: 18895 usec ;; Query time: 19229 usec ;; Query time: 18694 usec ;; Query time: 19321 usec ;; Query time: 18690 usec ;; Query time: 19534 usec ;; Query time: 19450 usec ;; Query time: 18931 usec ;; Query time: 18081 usec ;; Query time: 18632 usec ;; Query time: 18633 usec ;; Query time: 17768 usec ;; Query time: 18800 usec ;; Query time: 18844 usec

So fastest of 13.941 ms and several 14ms, 18ms, 19ms - all seem much more likely.

I only noticed this since rebuilding my Pi to the latest version, so it must be something on my end

Re: DNS timing out from all the Plus servers

MJN — Fri, 06 Jan 2023 12:30:26 GMT

@seebee wrote:

After looking a bit harder, I'm pretty sure this is a problem with my "dig" command.

...

I only noticed this since rebuilding my Pi to the latest version, so it must be something on my end

I wouldn't necessarily assume a 'problem', it might be more a case of unrealistic expectations regarding timing accuracy. 😉

The Pi's system clock is performed entirely in software so that will have an inherently impact accuracy and coupled with the likelihood that different parts of dig's codebase will be involved in determining query time (i.e. there must be two timestamps involved - one marking when a query is sent and another when an answer is received - and these may come from different running routines) may mean that the result should be interpreted with some caution, particularly at the micro/millisecond level of definition.

Re: DNS timing out from all the Plus servers

bobpullen — Fri, 06 Jan 2023 13:24:38 GMT

@seebee wrote:

@bobpullen

Don't suppose you see "too short" DNS times in your monitoring?

Nope. Also running on Raspian (3b board).

In my environment, ~8ms is achievable...

~ $ for n in {1..30}}; do dig @212.159.13.49 gstatic.com ; done | grep -F time ;; Query time: 10 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 13 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 13 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 11 msec ;; Query time: 16 msec ;; Query time: 15 msec ;; Query time: 16 msec ;; Query time: 17 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 8 msec ;; Query time: 17 msec ;; Query time: 15 msec ;; Query time: 16 msec ;; Query time: 13 msec ;; Query time: 17 msec ;; Query time: 7 msec ;; Query time: 17 msec ;; Query time: 16 msec ;; Query time: 14 msec

~ $ for n in {1..30}}; do dig -u @212.159.13.49 gstatic.com ; done | grep -F tim e ;; Query time: 8431 usec ;; Query time: 8746 usec ;; Query time: 16887 usec ;; Query time: 15764 usec ;; Query time: 16611 usec ;; Query time: 16282 usec ;; Query time: 16606 usec ;; Query time: 16719 usec ;; Query time: 16305 usec ;; Query time: 14455 usec ;; Query time: 17828 usec ;; Query time: 19351 usec ;; Query time: 8599 usec ;; Query time: 8342 usec ;; Query time: 8501 usec ;; Query time: 8923 usec ;; Query time: 16266 usec ;; Query time: 16390 usec ;; Query time: 15716 usec ;; Query time: 8351 usec ;; Query time: 8847 usec ;; Query time: 14803 usec ;; Query time: 17276 usec ;; Query time: 13872 usec ;; Query time: 8532 usec ;; Query time: 8317 usec ;; Query time: 8731 usec ;; Query time: 11273 usec ;; Query time: 14801 usec ;; Query time: 16834 usec

Re: DNS timing out from all the Plus servers

seebee — Sat, 14 Jan 2023 16:03:22 GMT

Over the years I've used ping/dig/traceroute/iperf/wireshark/whatever on my connection and everything I have seen, as fastest response, makes me believe my PPP tunnel is at best about 12ms "long". I can't get to the Internet quicker than that. For an ADSL service I think that's perfectly reasonable.

I have used dig from the command line countless times over the years, and never seen weird answers like 9ms - that only started happening when my SD card failed and I had to rebuild my Pi. So that made me wonder if it is a bug in a current version of dig (particularly as "-u" gives believable answers, as it is using a different code path in the binary).

To rule out the Pi itself, I then installed dig on my OpenWrt router (so completely different processor, OS and packages) and it was showing the same weird answers - 9ms sometimes, but the "-u" option looks fine. Same behaviour as the Pi.

So instead I installed the "drill" binary, an alternative implementation of dig (package named ldnsutils), which gives similar output to dig, on both my Pi and OpenWrt router. It behaves like dig used to be, sensible 14ms-ish replies. As the output from "drill" looks like the output from "dig", it is compatible for smokeping to parse, so I just used "drill" in the smokeping config instead (the "binary" entry in "/etc/smokeping/config.d/Probes") - now my charts for DNS response times look better.

Raspbian GNU/Linux 11 (bullseye) has - DiG 9.16.33-Raspbian - drill version 1.7.1 (ldns version 1.7.1).
OpenWrt 22.03.3 has - DiG 9.18.10 - drill version 1.8.1 (ldns version 1.8.1).

See the graphs of DNS response times. I made the change from dig to drill about halfway through.

Left hand side is dig with typically 9ms or 19ms responses almost a binary choice, compared with the right hand side with drill, with 13ms to 15ms responses, variation over the general range.

It's not proof there is some oddity in recent "dig" versions (as it works for Bob and presumably the rest of the world), but all I can say is - drill works for me!
And dig used to work for me too - see sensible graphs here from before my Pi failed.

This post was originally about DNS (minor) packet loss, and it was fixed and I agree there is no packet loss at all now. I was just curious about why my own monitoring was looking unusual.

Re: DNS timing out from all the Plus servers

Townman — Sat, 14 Jan 2023 17:28:12 GMT

Scary when one cannot trust the "tape measure" (instrumentation)!!

Your findings suggest that the fabric of gig's tape measure is elasticated!

Re: DNS timing out from all the Plus servers

MJN — Sat, 14 Jan 2023 17:44:36 GMT

@seebee wrote:

So that made me wonder if it is a bug in a current version of dig (particularly as "-u" gives believable answers, as it is using a different code path in the binary).

Looking at the source of the master it's not really a different code path - it merely divides the microsecond measurement by a thousand to give it in milliseconds if the -u flag hasn't been set:

if (query->lookup->stats) { const char *proto; diff = isc_time_microdiff(&query->time_recv, &query->time_sent); if (query->lookup->use_usec) { printf(";; Query time: %ld usec\n", (long)diff); } else { printf(";; Query time: %ld msec\n", (long)diff / 1000); } [...]

You'd have to check your particular package to see if there's any difference, but I'd be surprised if there was (would seem like an odd thing to deviate).