Dns seems high

[#1]

  but it looks like it was dreadful earlier today !  -

[#2]

as a real world test it got to the point that pages on PlusNet forums were taking up to 5 mins to load or in fact failing to load last night, and again this morning but it all cleared up around 0830, i'm starting to wonder if the problems aren't something to do with the dns cascade updates, this is rather too similar to the problem a few years ago just before one of BT's routers failed at a datacentre which just happened to coincide with the ipv6 trials, PlusNet aren't doing similar connectivity trials are you ?

[#3]

I've raised an internal problem to have this looked into folks (ref: 76913). Yesterday was particularly bad:

[#4]

Quote from: nanotm

as a real world test it got to the point that pages on PlusNet forums were taking up to 5 mins to load or in fact failing to load last night

Probably caused by something else. There was a separate problem affecting access to the Community Site yesterday (ref: 76907).

[#5]

Looking a lot better yesterday, fingers crossed

[#6]

Looking into the problem we're seeing significantly higher load on some of the caches. Transpires that some of the boxes are running a slightly older operating system revision. Those with the newer OS aren't seeing nearly as much load.
We've just upgraded one of the PCL servers and the situation looks much improved:

We'll be doing the rest of the PCL servers before moving onto the ones at Telehouse.
It'll take some time so shout up if things go awry again.

[#7]

Cached DNS lookups are around 40% slower from here at the moment than when things are running smoothly.
With peaks to around 300% slower.
So what's caused the higher load? Just a big traffic increase since people came back to work/school/university and stopped spending evenings outdoors?
Because the operating system revision can't have aged - the servers were like that before...  And they were really fast not so long ago.

[#8]

@SuperZoom
Just dropping you a reference point i'm on gateway ptw-ag04.

      Nameserver              Response Time (ms)            
                        min/avg/max/stdev/retries
Local                   0.61/0.62/0.62/0.00/0
212.159.6.9             17.44/17.78/18.36/0.31/0
212.159.6.10            17.02/17.59/18.30/0.46/0
212.159.13.49           17.04/18.18/19.41/0.92/0
212.159.13.50           17.01/17.53/17.89/0.30/0
208.67.222.222          17.08/17.64/18.81/0.63/0
208.67.222.220          17.09/17.47/17.66/0.20/0
208.67.222.123          17.11/17.54/18.40/0.47/0
208.67.220.123          17.11/17.31/17.51/0.14/0
8.8.8.8                 22.51/22.90/23.14/0.24/0
8.8.4.4                 22.51/22.81/23.05/0.21/0
4.2.2.1                 17.02/17.25/17.47/0.19/0
4.2.2.6                 16.81/17.03/17.27/0.19/0
 

What gateway are you on atm, i'm just qurious, I see your point with the software though

[#9]

Good, isn't it? Repeatable, easy to interpret... I know graphs are pretty and everything... The output could be graphed easily enough though...
Anyway, I appreciate the reference point. What's your ping to www.bbc.co.uk? That's pretty much the lowest you're going to get to a public website, since PlusNet have a direct link. It establishes a scale for everything else.
My ping to there flickers between 4 & 5 ms. (It's 3ms when things are good!)
By contrast, ping to 212.159.6.10 is currently between 4 and 7 ms, up and down all over the place over only 20 pings, and 212.159.6.9 varies between 4 and 19 ms during the same 20 pings.
This is what a selection of nameserver cached responses looks like at one moment in time (10 queries, as usual, and slower now than the 40% degradation I quoted earlier - if BBC ping is 3ms, the figures are 5.xxms):
[tt]Nameserver              Response Time (ms)
                        min/avg/max/stdev/retries
212.159.6.9            7.06/9.81/12.03/1.65/0
212.159.6.10            8.06/10.21/13.20/1.72/0
212.159.13.49          8.08/9.89/12.36/1.54/0
212.159.13.50          8.23/9.07/9.90/0.73/0
208.67.222.222          7.27/9.15/11.29/1.45/0
208.67.220.220          7.14/7.87/8.73/0.57/0
198.153.192.50          8.47/9.10/9.92/0.58/0
198.153.194.50          15.30/16.27/17.74/0.90/0
8.8.8.8                13.24/15.09/16.85/1.32/0
8.8.4.4                13.91/14.37/15.75/0.71/0
4.2.2.1                7.48/8.45/10.86/1.22/0
4.2.2.6                7.77/8.59/10.03/0.77/0[/tt]

I'm on ptn-ag03

[#10]

Just a quick 20.....
PING bbc.co.uk (212.58.251.195) 56(84) bytes of data.
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=1 ttl=119 time=17.7 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=2 ttl=119 time=17.6 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=3 ttl=119 time=17.1 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=4 ttl=119 time=17.4 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=5 ttl=119 time=17.6 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=6 ttl=119 time=17.7 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=7 ttl=119 time=17.3 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=8 ttl=119 time=17.1 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=9 ttl=119 time=17.2 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=10 ttl=119 time=17.2 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=11 ttl=119 time=17.4 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=12 ttl=119 time=17.3 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=13 ttl=119 time=17.6 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=14 ttl=119 time=17.2 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=15 ttl=119 time=17.5 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=16 ttl=119 time=17.5 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=17 ttl=119 time=17.0 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=18 ttl=119 time=17.2 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=19 ttl=119 time=17.5 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=20 ttl=119 time=17.2 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=21 ttl=119 time=17.2 m
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=22 ttl=119 time=17.8 ms
64 bytes from www-vip.telhc.bbc.co.uk (212.58.251.195): icmp_req=23 ttl=119 time=17.7 ms
^C
--- bbc.co.uk ping statistics ---
23 packets transmitted, 23 received, 0% packet loss, time 22034ms
rtt min/avg/max/mdev = 17.035/17.444/17.872/0.276 ms

[#11]

Yeah. I know time's moved on so it's hard to compare directly, but there's something not right at my exchange, I think. You have a lot less variation. And your name resolution times are very close to ping.
Well, I'm off to bed for now. But thanks. And hopefully we've given Bob food for thought too.

[#12]

And just now for comparison.
BBC ping up to a steady 5ms, even though traffic must be lower now...

[tt]Nameserver              Response Time (ms)
                        min/avg/max/stdev/retries
212.159.6.9            6.91/7.01/7.15/0.09/0
212.159.6.10            6.65/7.09/7.56/0.36/0
212.159.13.49          6.90/7.07/7.24/0.12/0
212.159.13.50          6.45/6.66/6.83/0.13/0
208.67.222.222          6.69/7.08/7.72/0.36/0
208.67.220.220          6.67/6.95/7.17/0.19/0
198.153.192.50          7.14/7.25/7.37/0.09/0
198.153.194.50          14.19/14.84/15.33/0.46/0
8.8.8.8                12.61/12.86/13.35/0.26/0
8.8.4.4                12.33/12.87/14.11/0.63/0
4.2.2.1                6.51/7.30/8.19/0.57/0
4.2.2.6                6.62/6.96/7.72/0.40/0[/tt]

[#13]

Quote from: SuperZoom

So what's caused the higher load? Just a big traffic increase since people came back to work/school/university and stopped spending evenings outdoors?

I'm not 100% sure if I'm honest. It could simply be that we've reached a tipping point. Or a couple of the caches were upgraded fairly recently and that upset the balance between them. I'll try and find out.

Quote from: SuperZoom

My ping to there flickers between 4 & 5 ms. (It's 3ms when things are good!) By contrast, ping to 212.159.6.10 is currently between 4 and 7 ms, up and down all over the place over only 20 pings, and 212.159.6.9 varies between 4 and 19 ms during the same 20 pings.

Whilst I appreciate the observation, we're talking milliseconds here. I'd rarely consider anything below 19ms to be a problem.
FWIW, my pings to the beeb/caches are almost identical to 30FTTC06's.
Anyway, just popped by to say that all of the PCL caches have been done now with similar results to the graphing I included in my last post. We'll be doing the Telehouse ones next.

[#14]

@'Bob Pullen' - can you put a hyperlink to the LIVE ThinkBroadband latency graph for "Plusnet DNS" into this thread ?
When the DNS is going awry, it would be useful to have a look at the current graph, rather than us posting complaints not knowing whether or not there is something wrong, followed a day later you posting the plot for the previous day.
Thanks

[#15]