Peak time latency spikes

[#1]

Quote from: Anotherone

Quote from: Kelly I'm watching. I think it could be the other problem we've been seeing, based on the drops being a lot less dramatic than before.    Which you still haven't explained, or said how/what it it's affecting

Remember this?  http://community.plus.net/forum/index.php/topic,111170.msg959759.html#msg959759

Quote

What is happening is that at our peak time, when these devices reach a certain amount of traffic they start to fail, and ask the core routers between them and the internet to slow down the rate at which they are sending traffic.

The messages that the device uses to say slow down is called a "pause frame".  What we've spotted is that the devices are sending small amounts of them all the time, increasing at peak, but only down one of their interfaces.  Each device has two paired interfaces which mean that traffic is balanced across them.  Your traffic will go down one of these interfaces based on a bunch of hashing algorithms, but that is per flow of traffic.  What this means is that some of your traffic will go down this path where the pause frames are being seen and will be slightly slower than it should be.
It's nowhere near as bad as we were seeing on the original problem (which we've informally labelled as the wall of death) but we've confirmed it dropping a couple of meg off of speed tests done by one of our engineers on a 7Mbps ADSL connection and causing small latency spikes.  Dropping the 2nd interface restored the speed.
Because it's only one of the interfaces showing the problem it's inconsistently impacting.  I.e. only some of your traffic flows will see impact.    It's also difficult to separate from normal faults.
We've got a case open with the vendor who are investigating.

[#2]

Quote from: Kelly

What we've spotted is that the devices are sending small amounts of them all the time, increasing at peak, but only down one of their interfaces.

Do you mean the device is flow controlling the core router even when it should be well able to handle the flow?

[#3]

yes.  The devices can handle 10Gbps.  They are well below that.

[#4]

What is the status now with this issue?
No service status update in nearly a week. Is it resolved, gettign better, or staying the same? Is it understood yet?

I'm waiting to place a "migrate from BE" order.
Thanks.

[#5]

It's vastly improved from last week - for most people on here, service seems back to normal (or near normal). I did 29 speed tests last night - 24 were at my regular max speed, 5 were 10-15Mb lower than the max speed but still vastly better than the single digit speeds I had at worst last week.
There are some users still experiencing slow downs, however that looks to be exchange congestion and outside of PN's control.

[#6]

I'll leave PN to give you the official answer. As a customer my perception is "getting better" and "partially understood". A few customers seem to be affected as badly as ever - it looks like their problems might have other causes. Some customers seem to get intermittent minor slow-downs during peak hours, but nowhere near as severe or often as before. And many of us seem to have no problems other than a slight rise in peak latency during busy times (which to be honest I would expect on a network which doesn't throttle p2p during busy hours) - it certainly has no perceptible effect on my broadband usage. Here's a snapshot of my TBB plot for yesterday which shows the rise in the evening when there was no internet usage happening on my connection:

EDIT: AndyH beat me to it. So just agreeing with him 

[#7]

We believe we've solved the problem which was causing the 70Mbps fibre lines turning into 3Mbps lines at peak.

[#8]

@Kelly
Thanks for the detailed explanation in reply #1335. That helps give a much clearer understanding of the situation.
It was this reply http://community.plus.net/forum/index.php/topic,111170.msg972299.html#msg972299 that led me to believe something else was involved.

[#9]

Back to the Norm at this time of the day...I'm going to have to get a night shift job so I can use my Fibre at full speed in the day....

[#10]

Just to keep you all in the loop, I won't be posting any results in this thread for probably at least a week or more.
Instead I will be investigating, with the help of Chris Pettitt, an alleged issue with my phone line that apparently is being highlighted by a BT DSL status check.  Intriguingly this problem doesn't show any symptoms in my router stats, or by line performance checks such as the DSLReports tweak test, and is not having any apparent impact on my internet usage - so is going to be interesting to fault find !.
So rather than night after night reporting in this thread that my line has slowed down between 20:00 and 22:00,  I am ducking out until I can resolve the issue with Chris, or until such time that we agree to disagree about where the problem resides.
Finally I would like to say thanks to Chris Pettitt, for firstly identifying something that we can work on, and secondly for providing some very helpful information and advice, so hopefully I will be able to move forward and get my line performance back at all times of day and night.

[#11]

Wowwwww its good tonight LMFAO

[#12]

I must say all weekend and this week so far my connection been fantastic no problem at all now well done PN

[#13]

looking good again tonight speed normal, browsing and video streaming OK. Whitebox results looking good.
on gateway pcl-ag08 which is diffrent from last few days as router dropped about 4am this morning

8-10pm period pasted by tonight with no problems, seems after 3 days it's looking good with what ever fix PN guys have put in place

[#14]

Just found a Samknows app for Android.  Hopefully managed to attach a screenshot from my phone that shows which days I've lost speed (and how it seems.to be back to normal for the last few days

[#15]

Quote from: Bjn

as router dropped about 4am this morning

BT maintenance