Dns seems high

[#1]

Unfortunately the downside of using a load balancer to hand out requests vs. anycast and a massively shared cache on the backend (i'm looking at you google)...
...To make more wide-ranging changes will unfortunately have to wait until major scaling at which point a lot of things will be addressed, so please bear with us.
In the meantime, if this exact same config is deployed across the rest of the cachedns estate then performance will be much improved.
[Moderator's note by Dick (Strat):  Full quote of preceding post removed, as per Forum Rule]
And again, as an aside - i'm pretty bored of having posts moderated for merely answering questions now when other forum users don't get the same treatment. Just an observation mods.

[#2]

Quote from: Chris

Unfortunately the downside of using a load balancer to hand out requests vs. anycast and a massively shared cache on the backend

Would a single cache be that much bigger?
I would have thought the vast majority of entries would be the same in all 10 caches.
Do you have any data to clarify this?
Sorry Chris but I'm not seeing anything at this end to say the new resolver is a step forward.

[#3]

Quote from: Chris

In the meantime, if this exact same config is deployed across the rest of the cachedns estate then performance will be much improved.

Well, there is only 1 way to find out if it will be a success i guess. I'm also adding a view from the south below, taken @22:45 ish.

[#4]

Also from down south at 23:45, and the same four DNSes at the bottom of the chart 

[#5]

Quote from: ejs

Linux, Fedora 20, the clue is in: # ; <<>> DiG 9.9.4-RedHat-9.9.4-8.fc20 <<>> @212.159.6.9 www.gnu.org +tries=1 (RedHat / Fedora, fc20 = Fedora 20) Here are some more: http://ejs1920.users.sourceforge.net/more-dig+adflag.pcap They aren't all 0x80, that capture was mainly me trying to spot any patterns with what causes the responses to be marked 0x20 - it seems to depend on which server (9 and 10 vs. 49 and 50) and what hostname you look up.

This should be fixed now and in titanium.

[#6]

I seem to be getting problems on DNS as well now.

Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	18.82/20.72/22.02/1.13/0
212.159.6.10        	18.56/19.00/19.58/0.42/0
212.159.13.49       	18.51/19.00/19.52/0.24/1
212.159.13.50       	18.29/18.86/19.78/0.55/0
Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	18.28/18.83/19.57/0.51/0
212.159.6.10        	18.11/18.51/18.92/0.32/0
212.159.13.49       	18.24/18.75/19.28/0.37/0
212.159.13.50       	18.96/19.19/19.51/0.21/1
Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	18.70/19.21/19.73/0.38/0
212.159.6.10        	18.15/18.70/19.54/0.47/0
212.159.13.49       	18.16/18.83/19.57/0.52/1
212.159.13.50       	18.57/18.97/19.56/0.45/0
Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	18.22/19.64/22.64/1.69/1
212.159.6.10        	18.35/18.85/19.31/0.39/0
212.159.13.49       	18.02/18.70/19.55/0.52/0
212.159.13.50       	18.20/18.89/19.54/0.46/0

[#7]

I assume the problem in those numbers is the retries.
One current problematic hostname for the PN DNS servers seems to be radar.cedexis.com (some advertising / tracking rubbish on some website) which gets a SERVFAIL response much of the time (but not always).
There should be very little point in testing each of 212.159.6.9, 212.159.6.10, 212.159.13.49 and 212.159.13.50 - they are all effectively the same.

[#8]

Yes the timeout/retries ejs
I assume dns_bench is testing against google.com which is what i'm using. I think that timeout equates to 2000ms
results from dns top give me
Queries: 0 new, 133 total                              Sun Jan 26 17:07:24 2014
Source       Query Name                    Count      %   cum%
------------ ------------------------- --------- ------ ------
192.168.0.15 www.google.com                  132   99.2   99.2
192.168.0.15 30.0.168.192.in-addr.arpa         1    0.8  100.0
I've been having trouble streaming as well, I've just not managed to put my finger on it as yet, so i just threw the net wide just incase anybody else is having trouble at all.

Edit: As you say it makes no odds testing 2 of the same, however,  nothing else i'm testing is having any issue dns wise.

Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
212.159.6.9         	19.30/19.59/19.76/0.19/1
212.159.6.10        	18.40/18.89/19.46/0.37/0
208.67.222.222      	17.90/18.79/19.55/0.64/0
208.67.222.220      	18.20/18.45/18.67/0.15/0
208.67.222.123      	17.97/18.39/19.02/0.36/0
208.67.220.123      	18.20/18.70/19.29/0.40/0
8.8.8.8             	23.39/25.24/29.50/2.18/0
8.8.4.4             	24.27/24.66/24.99/0.31/0
4.2.2.1             	17.94/18.32/19.05/0.41/0
4.2.2.6             	18.13/18.39/18.88/0.27/0
192.168.0.254       	19.54/19.70/19.93/0.14/0
192.168.0.15        	0.61/0.62/0.67/0.02/0
Nameserver          	Response Time (ms)
                    	min/avg/max/stdev/retries
208.67.222.222      	17.90/18.20/18.67/0.28/0
208.67.222.220      	17.94/18.23/18.38/0.16/0
208.67.222.123      	18.15/18.40/18.73/0.22/0
208.67.220.123      	18.02/18.24/18.49/0.18/0
8.8.8.8             	23.57/24.37/26.36/1.03/0
8.8.4.4             	23.78/24.35/25.15/0.54/0
4.2.2.1             	17.88/18.16/18.66/0.29/0
4.2.2.6             	18.02/18.32/18.50/0.16/0
192.168.0.254       	19.26/19.73/20.21/0.35/0
192.168.0.15        	0.60/0.61/0.61/0.00/0
212.159.6.9         	18.55/19.13/19.88/0.39/1
212.159.6.10        	18.35/19.08/19.80/0.58/0

[#9]

My "Sam knows" stats say this:-
Things looking bad (again)...

[#10]

That makes more sense, My ps3 is running on plusnet dns which is what I use to stream iplayer on.
I'd nearly forgot all about my whitebox 

[#11]

Well i'm still seeing issues with "plusnet" dns after things have settled down.

 Nameserver              Response Time (ms)                        min/avg/max/stdev/retries
 
212.159.6.9             18.00/18.42/18.73/0.27/1
212.159.6.10            17.67/18.42/18.87/0.42/0
208.67.222.222          18.00/18.24/18.60/0.20/0
208.67.222.220          18.03/18.19/18.40/0.17/0
208.67.222.123          17.91/18.12/18.34/0.15/0
208.67.220.123          17.91/18.10/18.36/0.19/0
8.8.8.8                 23.73/24.04/24.27/0.18/0
8.8.4.4                 23.69/24.09/24.42/0.26/0
4.2.2.1                 17.74/18.07/18.38/0.23/0
4.2.2.6                 17.73/18.01/18.38/0.25/0
192.168.0.254           19.26/19.57/19.80/0.18/0
192.168.0.15            0.60/0.63/0.70/0.04/0

[#12]

Our graphing looks more wobbly since Thursday afternoon.  I've flagged into Ops.

[#13]

One of the servers in PTN/PTW was misbehaving. It's been given a swift talking to and is now back in service.

[#14]

Many thanks for taking the time to reply Kelly, Much appreciated.

[#15]

Here we go again.... no packet loss yet I'm getting dns failing.
--- bbc.co.uk ping statistics ---101 packets transmitted, 101 received, 0% packet loss, time 100152ms
rtt min/avg/max/mdev = 17.424/18.476/23.675/0.980 ms

     Nameserver              Response Time (ms)                       
                        min/avg/max/stdev/retries
 
212.159.6.9             17.90/18.64/19.06/0.44/1
212.159.6.10            18.21/18.88/19.39/0.43/0
208.67.222.222          18.25/18.49/18.61/0.14/0
208.67.222.220          17.89/18.15/18.35/0.18/0
208.67.222.123          17.68/18.36/19.30/0.53/0
208.67.220.123          18.48/18.70/18.83/0.13/0
8.8.8.8                 23.50/24.12/24.81/0.50/0
8.8.4.4                 23.62/24.29/25.30/0.57/0
4.2.2.1                 18.14/19.16/20.44/0.98/0
4.2.2.6                 17.74/18.20/18.62/0.30/0
   

Edit:  A bit more info added

--- bbc.co.uk ping statistics ---
590 packets transmitted, 590 received, 0% packet loss, time 589238ms
rtt min/avg/max/mdev = 16.983/18.052/23.176/0.676 ms
(0.0.0.0)                                      Mon Jan 27 20:54:55 2014
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.0.254                     0.0%   230   24.8  40.1   1.6 100.5  25.2
 2. lo0-central10.pcl-ag06.plus.net   0.0%   230   16.7  26.9  16.4 291.4  31.8
 3. link-a-central10.pcl-gw01.plus.n  0.0%   229   17.2  17.6  16.5  35.2   1.3
 4. xe-10-3-0.pcl-cr01.plus.net       0.0%   229   36.6  18.8  16.1  55.7   5.0
 5. po2.pcl-gw01.plus.net             0.0%   229   17.1  17.7  16.3  83.9   4.6
 6. vl63.pcl-lb01.plus.net            0.0%   229   18.2  18.3  16.8  31.0   1.2
 7. cdns01.plus.net                   0.0%   229   17.1  18.3  16.7  25.2   1.1
(0.0.0.0)                                      Mon Jan 27 20:55:23 2014
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 192.168.0.254                     0.0%   394   18.7  51.4   0.7 100.3  32.3
 2. lo0-central10.pcl-ag06.plus.net   0.0%   394   21.3  27.3  16.4 504.9  38.5
 3. link-a-central10.pcl-gw01.plus.n  0.0%   394   17.3  17.6  16.1  34.8   1.7
 4. xe-10-3-0.pcl-cr01.plus.net       0.0%   394   18.4  18.0  16.1  50.1   3.7
 5. ae1.ptw-cr01.plus.net             0.0%   394   17.8  18.2  16.2  47.3   3.5
 6. te9-4.ptn-gw01.plus.net           0.0%   394   20.4  17.6  16.5  32.2   1.1
 7. gi5-1.peh-cr02.plus.net           0.0%   394   24.7  26.5  23.6  92.6   5.4
 8. po5.peh-cr01.plus.net             0.0%   394   30.5  26.4  23.9  58.0   2.7
 9. vlan2658.peh-elb02.plus.net       0.0%   393   25.9  27.6  24.2  86.7   4.3
10. portal.plus.net                   0.0%   393   26.1  28.1  24.5  79.2   6.1

No sign of packet loss in that lot.