Peak time latency spikes

[#1]

Quote from: Kelly

I have a vague inkling that these are related.   This suggests a DPI problem we aren't able to see though.:/

Perhaps. You can see what's going on in the network, I can only see what's reported on the forums

[#2]

We saw similar to the Usenet slowdown on one of the test lines, it turned out that the Usenet provider couldn't cope with us hitting them so hard during peaktimes.

[#3]

@ Anotherone
I'm not sure who your question was aimed at, but I'll have a go...
The graph shows an increase in the green level (minimum latency) when all traffic is delayed. An increase in yellow (maximum latency) occurs when some traffic is delayed.
In the case of this particular fault thread, the traffic offered to a particular piece of kit during peak period was in excess of the capability of the box to carry it, so its CPU was running at 100%. In that condition every packet passing through the box has to wait until the CPU gets around to processing it. Hence all traffic through it is delayed. And when a CPU runs flat out, it ends up putting most of its processing power into context switching, not actually processing instruction threads, so the effective capacity of the box actually decreases.
If the mean traffic level is below the hard capacity limit of the box, the CPU level will be lower. As the traffic level rises, so the CPU utilisation will rise. Remember that traffic throughput is a statistical process - if you plot a graph of the throughput for a given link or node over time, it looks a bit like those yellow spikey graphs, with sharp peaks and troughs. When an instantaneous traffic peak exceeds the capacity of the box it's passing through, the traffic is buffered and delayed. So if you plot the output of the box, the peaks have been "chopped off" above the level of the capacity of the box, and the troughs have been filled in by the traffic that was in the peaks. The area under the curve is the same (as long as no packets are dropped), but the traffic has been "smoothed out" (the peak-to-mean ration has been reduced). And each time the box "chops the top off" a traffic peak that arrived at its input, you see a yellow latency spike on your ping graph.
One problem from a network management point of view can be that the CPU utilisation reported by the box looks "acceptable" because it is averaged over a much longer period than the traffic spikes. So it looks like the box is handling the load, when in fact it's causing increased latency. Hopeful other stats from the box (eg buffer utilisation) highlight the issue (although the same problem can exist there).
That's my experience anyway. Whether this is what's going on in Plusnet's network I don't know. Maybe someone from Plusnet would like to have a go at explaining it!

[#4]

Quote

Maybe someone from Plusnet would like to have a go at explaining it!

I could have a go, but would probably be wrong! I'll ask someone that can explain a bit better to post tomorrow (or give me the detail to post).

[#5]

Thanks for your explanation Bright, most of which is fairly logical. The question was aimed a Chris as he was the immediately preceding poster, but perhaps it'll need to be Kelly who can explain.
The purpose of the question, perhaps not well phrased, was to discover whether this was possibly a Network issue, or whether it was connection related.

[#6]

@Bright "Oddities, such as the reduced Usenet throughput reported by shalom2010, whilst everything else is running OK. These are few and far between and probably down to another cause."
Perhaps you could explain further? I always have line speed 24/7 with my Usenet provider so am a little amiss at your comments. Last Friday was the first time any slowdown was noticed on Usenet, it did not just
drop a few Mbps. It dropped from 57.5Mbps to less than 1Mbps!! (Speedtests & pings were all correct fast and line speed)

[#7]

An upgrade to the Traffic Management software may have a bug that's incorrectly classifying your Usenet and slowing it down. It could be something more obscure of course.
Edit: http://usertools.plus.net/status/archive/1362065619.htm

[#8]

Am I still right in thinking that it was only that one night of Usenet slowdown for you Shalom?  Can you let us know if it happens again?

[#9]

Thanks Chris. Yes there was only Friday night. I have noticed a slight slowdown in the evenings dropping to around 45Mbps - hey that's fine compared to under 1Mbps. If it does that again I'll send you a PM!

[#10]

Thanks Anotherone (I wish people used real names!). It's doubtful that Traffic Management software has kicked in because right now I have full line speed. Only major hiccup was Friday evening.

[#11]

Thanks for the feedback Chris, I'm happy to keep posting data if it's useful. At the end of the day we all just want this issue resolved.  
I noticed there was a tweet earlier about increased capacity for 21CN / Fibre, is that anything to do with trying to solve the peak time slowdown, or just a result of more customers boarding?
Fine tonight so far by the way, I'm on pcl-ag01.

[#12]

Mixture of the 2 really, we've still got capacity unused at the moment so we're not stretched for that but it's better to add in advance rather than chasing the game.

[#13]

Quote from: shalom2010

I always have line speed 24/7 with my Usenet provider so am a little amiss at your comments. Last Friday was the first time any slowdown was noticed on Usenet, it did not just drop a few Mbps. It dropped from 57.5Mbps to less than 1Mbps!! (Speedtests & pings were all correct fast and line speed)

You suffered a significant slow down of one type of traffic during peak hours. So that fits the general category of reduced throughput during peak hours, but the symptoms were different from virtually everyone else suffering peak time slow-downs. So that's why I included it in the overall category of faults, but classed it as an oddity - everyone else suffered slow down of http traffic to their speed test web site.

[#14]

Still does not compute Bright I'm afriad. Over the weekend there were many posts saying they had slowdowns whilst Usenet for me over this weekend evenings was full line speed. Maybe I was lucky being on
the best gateway ptw-ag03 ?

[#15]

Sorry shalom2010, I don't understand what doesn't compute or why you are amiss at my previous post...