cancel
Showing results for 
Search instead for 
Did you mean: 

Peak time latency spikes

Strat
Community Veteran
Posts: 31,320
Thanks: 1,589
Fixes: 565
Registered: ‎14-04-2007

Re: Peak time latency spikes

The more thinly you spread your problem the harder it is for PN to help.
Try to keep things in one thread. Wink
Windows 10 Firefox 109.0 (64-bit)
To argue with someone who has renounced the use of reason is like administering medicine to the dead - Thomas Paine
Bright
Grafter
Posts: 363
Registered: ‎02-02-2013

Re: Peak time latency spikes

Quote from: AndyH
Yeah I think that sounds right. At busy peak periods, the gateway won't respond to pings in a traceroute.

Hmmm... On the other hand, 550ms does seem a long response time during a lower traffic period! My gateway seem better behaved:
Quote
--- 195.166.128.193 ping statistics ---
100 packets transmitted, 100 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.198/10.355/36.597/6.278 ms

Maybe Kelly's boiling up a cup of coffee on your gateway  Cheesy
Anotherone
Champion
Posts: 19,107
Thanks: 457
Fixes: 21
Registered: ‎31-08-2007

Re: Peak time latency spikes

Got this just now from a tracert, but it was a one off -
  2   254 ms   135 ms   169 ms  lo0-central10.pcl-ag05.plus.net [195.166.128.186]
More typical just now
  2    34 ms    34 ms    34 ms  lo0-central10.pcl-ag05.plus.net [195.166.128.186]
And this from a ping
Pinging 195.166.128.186 with 32 bytes of data:
Reply from 195.166.128.186: bytes=32 time=35ms TTL=126
Reply from 195.166.128.186: bytes=32 time=35ms TTL=126
Reply from 195.166.128.186: bytes=32 time=34ms TTL=126
Reply from 195.166.128.186: bytes=32 time=35ms TTL=126
Ping statistics for 195.166.128.186:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 34ms, Maximum = 35ms, Average = 34ms
But would I be correct in assuming that the following is the next best thing to TBB monitor, but staying within the Plusnet network?
You are using a debugging mode - and checking IP 87.112.xxx.xxx
Array
(
    [0] =>  1  84.92.0.73 (84.92.0.73)  0.693 ms
    [1] =>  2  po3.3659.peh-cr02.plus.net (84.93.232.38)  0.428 ms
    [2] =>  3  84-93-232-60 (84.93.232.60)  7.933 ms
    [3] =>  4  ae3.ptw-cr01.plus.net (195.166.129.32)  8.095 ms
    [4] =>  5  ae1.pcl-cr01.plus.net (195.166.129.1)  9.588 ms
    [5] =>  6  te-4-2.pcl-gw01.plus.net (212.159.1.9)  8.344 ms
    [6] =>  7  link5-central10.pcl-ag05.plus.net (84.93.249.137)  9.217 ms
    [7] =>  8  *
    [8] =>  9  *
    [9] => 10  *
    [10] => 11  *
)
rookey
Grafter
Posts: 572
Registered: ‎23-01-2009

Re: Peak time latency spikes

No Graph as set up a different router to see if any difference look at it sooo far no good
Bright
Grafter
Posts: 363
Registered: ‎02-02-2013

Re: Peak time latency spikes

Quote from: Anotherone
But would I be correct in assuming that the following is the next best thing to TBB monitor, but staying within the Plusnet network?
You are using a debugging mode - and checking IP 87.112.xxx.xxx

Not sure. What is it?Huh
AndyH
Grafter
Posts: 6,824
Thanks: 1
Registered: ‎27-10-2012

Re: Peak time latency spikes

Quote from: Bright

Hmmm... On the other hand, 550ms does seem a long response time during a lower traffic period! My gateway seem better behaved:

Just now:
Quote
Ping statistics for 195.166.128.196:
    Packets: Sent = 100, Received = 100, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 5ms, Maximum = 577ms, Average = 49ms

Need someone at PN to explain if this is normal. I would have thought if the gateway was fully loaded, then all pings would be consistently high. But this is what happens when I do 100 pings:
Quote
Reply from 195.166.128.196: bytes=32 time=6ms TTL=126
Reply from 195.166.128.196: bytes=32 time=8ms TTL=126
Reply from 195.166.128.196: bytes=32 time=5ms TTL=126
Reply from 195.166.128.196: bytes=32 time=577ms TTL=126
Reply from 195.166.128.196: bytes=32 time=234ms TTL=126
Reply from 195.166.128.196: bytes=32 time=15ms TTL=126
Reply from 195.166.128.196: bytes=32 time=6ms TTL=126
Reply from 195.166.128.196: bytes=32 time=5ms TTL=126

This may however be perfectly normal as it's not impacting any of my web browsing atm.
Kelly
Hero
Posts: 5,498
Thanks: 373
Fixes: 9
Registered: ‎04-04-2007

Re: Peak time latency spikes

You need to be careful with isolated ping tests.  Ping responses are dealt with by the devices on a lower priority than actually dealing with real traffic.
Kelly Dorset
Ex-Plusnet Jobsbody
rja
Grafter
Posts: 55
Registered: ‎28-01-2013

Re: Peak time latency spikes

Quote from: Bright
Responding to pings is just a very low priority task on the gateway router...?

traceroute doesn't use pings. It's sending a series of UDP datagrams to the target host, with a progressively higher TTL count. So, the first UDP datagram exceeds its TTL on the first host which should then send back a ICMP time exceeded message and so on up the route. Maybe the gateway's stack is just slow at generating the ICMP responses?
Bright
Grafter
Posts: 363
Registered: ‎02-02-2013

Re: Peak time latency spikes

Quote from: Kelly
You need to be careful with isolated ping tests.   Ping responses are dealt with by the devices on a lower priority than actually dealing with real traffic.

Indeed, although a 550ms ping response during a low traffic load period does suggest the router's pretty busy doing something else!
For comparison:
Quote
My gateway - lo0-central10.ptn-ag04.plus.net (195.166.128.193)
--- 195.166.128.193 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.092/14.215/38.889/8.864 ms

Quote
Next node in PN network - link10-central10.ptn-gw02.plus.net (84.93.248.242)
--- 84.93.248.242 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 7.031/7.178/7.325/0.090 ms

That's a much more consistent ping response than the gateway... Not sure if it's anything suspicious, but it just looks a bit odd at this time of day.
And using traceroute:
Quote
2  lo0-central10.ptn-ag04.plus.net (195.166.128.193)  12.804 ms  8.905 ms  9.581 ms
3  link10-central10.ptn-gw02.plus.net (84.93.248.242)  6.993 ms  7.208 ms  6.939 ms
Bright
Grafter
Posts: 363
Registered: ‎02-02-2013

Re: Peak time latency spikes

Quote from: rja
traceroute doesn't use pings. It's sending a series of UDP datagrams...

Indeed, and traceroute shows significant variability in response time, similar to the pings (that I was referring to from earlier messages in the thread). Both seem to have very low priority (as would be expected), but rather more variability than you might expect when not under high load.
This is probably just a red herring, but curious why this should be...
ejs
Aspiring Hero
Posts: 5,442
Thanks: 631
Fixes: 25
Registered: ‎10-06-2010

Re: Peak time latency spikes

Windows tracert does use ICMP echo requests.
rja
Grafter
Posts: 55
Registered: ‎28-01-2013

Re: Peak time latency spikes

Quote from: Bright
Quote from: rja
traceroute doesn't use pings. It's sending a series of UDP datagrams...

Indeed, and traceroute shows significant variability in response time, similar to the pings (that I was referring to from earlier messages in the thread). Both seem to have very low priority (as would be expected), but rather more variability than you might expect when not under high load.
This is probably just a red herring, but curious why this should be...

It's my understanding that when you have to send a message to a routing device - rather than through it - that this is handled higher up the interface stack and so involves more work. So, getting a reply from hop 3 might take longer than hop 4 which logically makes no sense! So, arguably you might just be seeing the effect of different hardware - perhaps the first hop is able to context switch less efficiently than the second. Just a theory.
I'm not sure where the priority comes in though. If something is prioritising traffic then that suggests there is a store-and-forward going on somewhere. That would be an "interesting" thing to build given the amounts of traffic that PlusNet have to shift! My understandung of the "priority" of ICMP data was that a link could choose to drop it if heavily loaded. I could easily be wrong though.  Wink
Kelly
Hero
Posts: 5,498
Thanks: 373
Fixes: 9
Registered: ‎04-04-2007

Re: Peak time latency spikes

Yeah, that's it.  The device at that layer is handling things like updating routes that are broadcasted to it it as well as other housekeeping functions.  The actual traffic routing is done by the hardware.
Kelly Dorset
Ex-Plusnet Jobsbody
Bright
Grafter
Posts: 363
Registered: ‎02-02-2013

Re: Peak time latency spikes

Quote from: rja
It's my understanding that when you have to send a message to a routing device - rather than through it - that this is handled higher up the interface stack and so involves more work. So, getting a reply from hop 3 might take longer than hop 4 which logically makes no sense! So, arguably you might just be seeing the effect of different hardware - perhaps the first hop is able to context switch less efficiently than the second. Just a theory.
I'm not sure where the priority comes in though...

Sorry, my fault for not being clear (or correct!) in my terminology. I meant the priority of the task/process handling responses to ICMP packets in the router, compared to the processes (or just hardware/firmware) handling normal routing and other packet handling functions. I think we're talking about the same thing, and I agree with your theory  Smiley  I guess the capacity/loading of each node would also lead to such "illogical" results.
I think I was just rather surprised to see response times ranging up to 550ms on AndyH's gateway. And with traceroute using UDP packets I get much longer response times from the gateway than subsequent nodes currently.
Quote from: rja
If something is prioritising traffic then that suggests there is a store-and-forward going on somewhere. That would be an "interesting" thing to build given the amounts of traffic that PlusNet have to shift!

I think just DiffServ QoS tagging of traffic using the Arbor E100 platform. That does the DPI for traffic management and the different traffic class queues. May be wrong there, but I got the impression that's how it all works. Hopefully someone else can put me right if I've got that wrong!
Kelly
Hero
Posts: 5,498
Thanks: 373
Fixes: 9
Registered: ‎04-04-2007

Re: Peak time latency spikes

E100 identifies the traffic, sets the Diffserv field.  The e320 looks at that field and sends, buffers or drops it, depending on the queue its in and how busy that queue is.
Kelly Dorset
Ex-Plusnet Jobsbody