Dns seems high

[#1]

What DNS IP address are you using and what gateway were you connected to?

[#2]

pcl-ag06
212.159.6.9,212.159.6.10 which are set static per device, open dns is fine, all the tests above are using dns_bench via port 53 using google.com.for testing. Apart from the pings of course

[#3]

Hang on a minute...
Today the loss i'm seeing which is only over UDP and is now showing up over multiple dns servers, i've tested on mutiple devices direct from the router. ie the problem seems to be getting worse.
Any thoughts ?

Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	17.51/17.91/18.41/0.32/1
212.159.6.10        	17.17/17.86/18.87/0.60/0
208.67.222.222      	17.22/17.53/17.82/0.22/0
208.67.222.220      	17.44/17.79/18.81/0.51/0
208.67.222.123      	17.27/17.50/17.68/0.15/0
208.67.220.123      	17.09/17.38/17.62/0.18/0
8.8.8.8             	22.94/24.08/26.34/0.69/1
8.8.4.4             	22.94/23.60/24.89/0.68/0
4.2.2.1             	17.17/17.53/17.75/0.22/1
4.2.2.6             	17.05/17.31/17.43/0.16/0
4.2.2.2             	17.11/17.35/17.52/0.14/0
4.2.2.5             	17.23/17.46/17.70/0.15/0
4.2.2.5             	17.00/17.25/17.66/0.22/0
4.2.2.3             	16.97/17.35/17.64/0.30/0

This is the first time i've seen an issue with any other dns, some kind of spike happening lasting seconds. Let me know if you want me to hop gateway, but i would rather leave it as is for the moment.

[#4]

Any ideas Kelly ?
See the waiting times on the followiing.

[#5]

Is it only the Plus.net showing those slowdowns?  That's not DNS.

[#6]

No, everything is the same. I have no packet loss either

(0.0.0.0)                                      Tue Jan 28 17:55:41 2014
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.0.254                     0.0%    35   94.7  42.5   1.9  99.5  25.5
 2. lo0-central10.pcl-ag06.plus.net   0.0%    35   16.8  35.1  16.6 282.0  56.0
 3. link-a-central10.pcl-gw01.plus.n  0.0%    35   16.6  17.0  16.4  19.6   0.4
 4. xe-10-3-0.pcl-cr01.plus.net       0.0%    35   18.8  19.0  15.8  50.9   7.7
 5. po2.pcl-gw01.plus.net             0.0%    35   16.5  16.7  16.1  17.5   0.0
 6. gi5-8.ptp-cr01.plus.net           0.0%    35   24.4  24.2  23.7  25.0   0.0
 7. vlan2658.ptp-elb02.plus.net       0.0%    34   25.3  25.5  24.5  29.9   0.8
 8. portal.plus.net                   0.0%    34   25.0  25.2  24.2  27.1   0.4

[#7]

Weird.  Might be worth a gateway hop and see if it disappears.  I'm not away of a significant problem that could be causing that though 😕

[#8]

Ok i'll try moving gateway a bit later, I agree though it's very strange that i'm not seeing any loss but i seem to be getting upto 2000ms pauses when loading web pages. this reminds me of Super Zooms issue. 

[#9]

DNS Benchmark is showing a significant number of failed lookup for the plusnet dns resolvers.
ie the red through the ip address represents failed lookups.

[#10]

It's not making much sense to me at the moment 

Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	18.16/20.22/22.68/1.72/1
212.159.6.10        	19.88/21.19/22.92/1.14/0
212.159.13.49       	18.64/19.10/19.58/0.34/0
212.159.13.50       	19.65/20.03/20.61/0.24/1

Nigel, I'm seeing loss of upto 2000ms represented by 1 retries on the above test which I tested at the time of this post, i've also posted a little .jpg test below as well.
However! as you can see below i was getting failures across other dns-servers with the same test as above. But i can't replicate failures that i was getting from the tests below tonight.

Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	17.51/17.91/18.41/0.32/1
212.159.6.10        	17.17/17.86/18.87/0.60/0
208.67.222.222      	17.22/17.53/17.82/0.22/0
208.67.222.220      	17.44/17.79/18.81/0.51/0
208.67.222.123      	17.27/17.50/17.68/0.15/0
208.67.220.123      	17.09/17.38/17.62/0.18/0
8.8.8.8             	22.94/24.08/26.34/0.69/1
8.8.4.4             	22.94/23.60/24.89/0.68/0
4.2.2.1             	17.17/17.53/17.75/0.22/1
4.2.2.6             	17.05/17.31/17.43/0.16/0
4.2.2.2             	17.11/17.35/17.52/0.14/0
4.2.2.5             	17.23/17.46/17.70/0.15/0
4.2.2.5             	17.00/17.25/17.66/0.22/0
4.2.2.3             	16.97/17.35/17.64/0.30/0

[#11]

what those tests tell me is that level3, google and plusnet failed at the same time, which is a very-very long shot. All i do know is something isn't as it should be. And not many people are coming forward with any other info. I can see that the patch/fix has been added to all those plusnet dns servers now by the looks of it.

[#12]

I'm on pcl-ag02.
I was about to blame the load unbalancer, but 212.159.6.129 managed to catch up with the timeouts towards the end of the testing, to finished tied with 212.159.6.9 in a race to be the worst DNS server.
Out of 406 queries to each:
212.159.6.9: 19 timeouts
212.159.6.129: 19 timeouts
212.159.6.130: 0 timeouts
212.159.6.131: 11 timeouts
212.159.6.132: 0 timeouts

[#13]

Moved from pcl-ag06 tp ptn-ag03 and it looks like the loss has vanished for now. 2 sets of tests below, top set pcl-ag06, bottom set ptn-ag03

lo0-central10.pcl-ag06.plus.net
Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	17.21/17.73/18.04/0.31/0
212.159.6.10        	17.67/17.77/17.86/0.07/0
212.159.13.49       	17.23/17.68/18.30/0.24/1
212.159.13.50       	17.58/17.79/18.07/0.19/1

Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
208.67.222.222      	17.37/17.67/17.88/0.20/0
208.67.222.220      	17.62/17.72/17.85/0.08/0
208.67.222.123      	17.47/17.94/18.58/0.38/0
208.67.220.123      	17.40/17.68/17.91/0.22/0
8.8.8.8             	23.09/23.60/24.07/0.40/0
8.8.4.4             	22.87/23.43/23.97/0.39/0
4.2.2.1             	17.08/17.51/17.89/0.28/0
4.2.2.6             	17.40/17.69/18.02/0.22/0
4.2.2.2             	17.13/17.54/17.77/0.22/0
4.2.2.5             	16.99/17.49/17.86/0.28/0
4.2.2.5             	17.36/17.57/17.79/0.16/0
4.2.2.3             	17.22/17.51/17.90/0.27/0
192.168.0.254       	18.51/19.03/19.53/0.41/0
192.168.0.15        	0.92/0.95/0.97/0.02/0

--- bbc.co.uk ping statistics ---
116 packets transmitted, 116 received, 0% packet loss, time 123252ms
rtt min/avg/max/mdev = 16.225/16.810/17.348/0.248 ms

lo0-central10.ptn-ag03.plus.net
Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	17.46/18.13/18.78/0.45/0
212.159.6.10        	17.73/18.14/18.53/0.26/0
212.159.13.49       	17.94/18.23/18.55/0.24/0
212.159.13.50       	17.77/18.10/18.83/0.37/0
Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
208.67.222.222      	17.57/17.94/18.37/0.26/0
208.67.222.220      	17.50/17.79/18.16/0.23/0
208.67.222.123      	17.44/18.02/18.58/0.41/0
208.67.220.123      	17.52/18.08/18.38/0.29/0
8.8.8.8             	23.18/23.52/23.95/0.29/0
8.8.4.4             	22.98/23.52/24.65/0.59/0
4.2.2.1             	17.76/18.04/18.38/0.21/0
4.2.2.6             	17.25/17.48/17.67/0.18/0
4.2.2.2             	17.24/17.43/17.63/0.17/0
4.2.2.5             	17.12/17.54/17.83/0.25/0
4.2.2.5             	17.01/17.57/17.89/0.32/0
4.2.2.3             	17.60/18.01/18.24/0.25/0
192.168.0.254       	18.88/19.24/19.55/0.21/0
192.168.0.15        	0.95/0.98/1.01/0.03/0

--- bbc.co.uk ping statistics ---
116 packets transmitted, 116 received, 0% packet loss, time 115167ms
rtt min/avg/max/mdev = 16.429/16.823/17.404/0.263 ms
Which gateway was you on npr ?
I'll see how this pans out again later Kelly, but for now things have improved on this gateway.

[#14]

Our Ops guys have been in the process of rolling out updated config to the DNS platform.  As of yesterday they completed the PTW/PTN side of the DNS platform.  We're starting PCL today.  That might explain the perceived improvement when you reconnected to the PTW gateway last night?
>11110_110  Are you still seeing the website delay

[#15]

Was on pcl-ag07.
Just tested again from pcl and then ptw.
From PCL

From PTW

So PTW is indeed improved.
Hope there's still some more dns upgrades to come, it needs them.