cancel
Showing results for 
Search instead for 
Did you mean: 

Latency spikes when streaming - especially Netflix

Bobsta
Grafter
Posts: 61
Thanks: 6
Registered: ‎23-12-2011

Latency spikes when streaming - especially Netflix

Hi folks,

 

 

I'm on an 80/20 FTTC package with a nice short line length, so achieve a good sync speed with very low error rates - zero Error Seconds downstream and 674 upstream in last 20 days - full details at http://home.bobsta.com/errors.htm and http://home.bobsta.com/fullstats.htm

 

 

On the whole my connection is great. I can achieve ~6ms pings to bbc.co.uk and easily get 70+mbps down and 17+mbps up when carrying out throughput tests.

 

However, as soon as anyone in the house starts using streaming services - particularly Netflix, things go wrong. The Netflix client appears to pull down content in distinct chunks - bursting to use around 10-12Mbps every 4 seconds or so. Each time that happens, I see my latency increase hugely which makes VoIP calls painful. Example below (and yes, this is all Ethernet, no WiFi to blame here):

 

bobsta-a01:~ bobsta$ ping bbc.co.uk
PING bbc.co.uk (151.101.128.81): 56 data bytes
64 bytes from 151.101.128.81: icmp_seq=0 ttl=57 time=15.891 ms
64 bytes from 151.101.128.81: icmp_seq=1 ttl=57 time=7.409 ms
64 bytes from 151.101.128.81: icmp_seq=2 ttl=57 time=7.064 ms
64 bytes from 151.101.128.81: icmp_seq=3 ttl=57 time=233.914 ms
64 bytes from 151.101.128.81: icmp_seq=4 ttl=57 time=6.704 ms
64 bytes from 151.101.128.81: icmp_seq=5 ttl=57 time=7.046 ms
64 bytes from 151.101.128.81: icmp_seq=6 ttl=57 time=7.000 ms
64 bytes from 151.101.128.81: icmp_seq=7 ttl=57 time=6.966 ms
64 bytes from 151.101.128.81: icmp_seq=8 ttl=57 time=6.928 ms
64 bytes from 151.101.128.81: icmp_seq=9 ttl=57 time=231.775 ms
64 bytes from 151.101.128.81: icmp_seq=10 ttl=57 time=7.241 ms
64 bytes from 151.101.128.81: icmp_seq=11 ttl=57 time=6.764 ms
64 bytes from 151.101.128.81: icmp_seq=12 ttl=57 time=7.045 ms
64 bytes from 151.101.128.81: icmp_seq=13 ttl=57 time=198.423 ms
64 bytes from 151.101.128.81: icmp_seq=14 ttl=57 time=7.261 ms
64 bytes from 151.101.128.81: icmp_seq=15 ttl=57 time=7.114 ms
64 bytes from 151.101.128.81: icmp_seq=16 ttl=57 time=7.053 ms
64 bytes from 151.101.128.81: icmp_seq=17 ttl=57 time=220.457 ms
64 bytes from 151.101.128.81: icmp_seq=18 ttl=57 time=6.766 ms
64 bytes from 151.101.128.81: icmp_seq=19 ttl=57 time=6.577 ms
64 bytes from 151.101.128.81: icmp_seq=20 ttl=57 time=6.849 ms
64 bytes from 151.101.128.81: icmp_seq=21 ttl=57 time=7.070 ms
64 bytes from 151.101.128.81: icmp_seq=22 ttl=57 time=6.968 ms
64 bytes from 151.101.128.81: icmp_seq=23 ttl=57 time=257.019 ms
64 bytes from 151.101.128.81: icmp_seq=24 ttl=57 time=102.027 ms
64 bytes from 151.101.128.81: icmp_seq=25 ttl=57 time=6.785 ms
64 bytes from 151.101.128.81: icmp_seq=26 ttl=57 time=8.220 ms
64 bytes from 151.101.128.81: icmp_seq=27 ttl=57 time=7.041 ms
64 bytes from 151.101.128.81: icmp_seq=28 ttl=57 time=7.111 ms
64 bytes from 151.101.128.81: icmp_seq=29 ttl=57 time=6.788 ms
64 bytes from 151.101.128.81: icmp_seq=30 ttl=57 time=6.794 ms
64 bytes from 151.101.128.81: icmp_seq=31 ttl=57 time=154.652 ms
64 bytes from 151.101.128.81: icmp_seq=32 ttl=57 time=96.274 ms
64 bytes from 151.101.128.81: icmp_seq=33 ttl=57 time=6.860 ms
64 bytes from 151.101.128.81: icmp_seq=34 ttl=57 time=6.850 ms
64 bytes from 151.101.128.81: icmp_seq=35 ttl=57 time=7.037 ms
64 bytes from 151.101.128.81: icmp_seq=36 ttl=57 time=6.992 ms
64 bytes from 151.101.128.81: icmp_seq=37 ttl=57 time=191.407 ms
64 bytes from 151.101.128.81: icmp_seq=38 ttl=57 time=6.798 ms
64 bytes from 151.101.128.81: icmp_seq=39 ttl=57 time=7.676 ms
64 bytes from 151.101.128.81: icmp_seq=40 ttl=57 time=6.766 ms

... and as today my daughter's been off sick from school with Netflix on most of the day, it results in my TBB graph looking like this:

 

f354dcce3092563691a429bebd72837c7b521867-06-09-2019

 

I've spent hours... no, days messing around with QoS settings on my router, even swapping out the router for alternative models, and it makes no difference.

The behaviour isn't present if I stress the line with other workloads - it's specific to Amazon Video, YouTube and particularly Netflix.

It also seems I'm not alone in this. It's been brought up on the Plusnet forums quite a few times and never seems to be resolved - instead people focusing on it being a copper line/VDSL issue. To me it seems to quite firmly be a backhaul or network QoS issue.

Example here: https://community.plus.net/t5/Fibre-Broadband/Intermittent-High-Ping-Particularly-if-Netflix-is-Stre... ... which seems to be precisely what I'm experiencing. The thread also links to this huge thread on BT forums confirms this is a thing on the BT network: https://community.bt.com/t5/Home-setup-Wi-Fi-network/Recent-online-gaming-ping-spikes-when-streaming...

 

Can someone from Plusnet advise what causes this please, and how to work around it? I find it hugely annoying that I have to put up with this, whilst a friend using a Sky Broadband FTTC connection with similar/worse line properties to mine, hammers it constantly and gets this:

e6c317eaa761794e67b4e2872f11aa0166e6e340-06-09-2019

Thanks,

Bobsta

20 REPLIES 20
Bobsta
Grafter
Posts: 61
Thanks: 6
Registered: ‎23-12-2011

Re: Latency spikes when streaming - especially Netflix

Here’s a video of the behaviour in action: https://youtu.be/Xfg6ORqrdCc
0:00 - Start ping to bbc.co.uk - RTT latency is consistent at around 7ms.
0:20 - Begin DSLReports speed test - latency moves a bit but is generally consistent during the test. Throughput is shown on router graphs on right - easily at 70Mbps and above.
1:10 - DSLReports speed test completes with A A A rating.
1:25 - Kick off HTTP download of very large (1GB) file. Throughput is again shown at over 73Mbps whilst latency barely moves - typically 9ms - 15ms.
2:25 - Stop large file download and prepare for Netflix test.
3:00 - Netflix HD TV show begins streaming on an Ethernet-connected TV in the house.
3:30 - Once the stream settles, distinct latency pulses can be seen of well over 150ms every few seconds, even though throughput during these pulses is only around 20Mbps.

Plusnet staff - can you explain what’s going on please? I’m 100% convinced this issue is in your network and not with my equipment, as I’ve done extensive testing over the past few months trying to eliminate this issue.
Gandalf
Community Gaffer
Community Gaffer
Posts: 26,563
Thanks: 10,265
Fixes: 1,599
Registered: ‎21-04-2017

Re: Latency spikes when streaming - especially Netflix

Hi @Bobsta thanks for getting in touch. I'm sorry to see you're experiencing lag spikes and packet loss while streaming.

I've tested your line and the tests aren't showing any issues as you've indicated you've got a near if not then perfect line and there's no congestion at the exchange flagging up.

I've gone through your previous forum posts and my first instinct was to ask if you experienced these problems before you migrated onto the closer cabinet but I see that the issue started on the 1st July this year.

There are two things I'd like to try, adding a static IP onto your account and arranging an engineer to investigate further as it's not uncommon for errors on a port at the cabinet to cause packet loss issues where we're not seeing anything wrong.

We'll start with the easy one first, so I've added the static IP on temporarily I hope because either way this shouldn't be a permanent solution. I've also had to drop your PPP connection momentarily to apply it.

Could you let me know how things go over the weekend and post up a fresh Quality Monitor this evening/tomorrow?

From 31st October 2022, I no longer have a regular presence here as I’ve moved on to a new role.
Anoush Mortazavi
Plusnet
Bobsta
Grafter
Posts: 61
Thanks: 6
Registered: ‎23-12-2011

Re: Latency spikes when streaming - especially Netflix

Thanks Gandalf. Appreciate you trying the fixed IP.
At the moment it doesn’t appear to have made any difference. Pretty much exactly the same behaviour as the video I recorded last night.

Updated quality monitor is at: http://home.bobsta.com/line.htm

There’s plenty of yellow already but I’ll post a static snapshot of the quality monitor tomorrow.
RichardB
Seasoned Champion
Posts: 1,038
Thanks: 385
Fixes: 39
Registered: ‎19-11-2008

Re: Latency spikes when streaming - especially Netflix

Hi Bobsta,

You are not alone!

My PN BB has for a long time exhibited the frequent high ping times when streaming video.

The maximum ping time being high (in excess of 100ms) but the average ping time being approx 20ms.

In my case it is the kids streaming video from Youtube or Sky Go etc

There are other similar graphs over at:

https://community.plus.net/t5/custom/page/page-id/_PingGraphs

and yet some that do not exhibit the high ping times...

I know it is video streaming which causing my maximum ping time to rise but I have to assume that it is video streaming that causes others ping times to rise.

A few years ago gateway hopping could often ameliorate the issue but no longer.

Regards

Richard

Bobsta
Grafter
Posts: 61
Thanks: 6
Registered: ‎23-12-2011

Re: Latency spikes when streaming - especially Netflix

Thanks for the additional info RichardB - much as I suspected. It seems to me that the ISP-level QoS must be extremely messed up to cause this. It’s almost like it’s protecting other users on the network at the expense of bundling *all* traffic to/from my IP through the same queuing mechanism.

Here’s my quality monitor snapshot from last 24hrs. You can clearly see when we paused streaming for dinner, and when the streaming stopped last night when the kids went to bed!

https://www.thinkbroadband.com/broadband/monitoring/quality/share/989bb8abd584deb28e13a8f026b489a4d9...
Gandalf
Community Gaffer
Community Gaffer
Posts: 26,563
Thanks: 10,265
Fixes: 1,599
Registered: ‎21-04-2017

Re: Latency spikes when streaming - especially Netflix

Thanks for getting back to us @Bobsta 

Unless I'm mistaken, big yellow spikes aren't necessarily bad as that just indicates the connection is being used. What's bad is the red packet loss at the top. Unfortunately we're unable to view the link you've posted as this appears to be blocked internally. Are you able to embed a live Quality Monitor stream into a forum post or send us a direct link to it? 

We also don't have any traffic prioritisation or management policies on our network, I believe we removed that in 2017 and as you've tried a static IP then the issue would highly unlikely be related to our network.

Having said that, have you tried to do anything time sensitive like VoIP or play online games since the static IP was added? If you're still experiencing problems, I think we're best off progressing this as a fault with our suppliers because there can be numerous causes for packet loss issues and latency spikes while streaming.

Before we go down that path and arrange an engineer though, would you be up for trying a Hub One? Happy to send you one as part of this troubleshooting. This should rule out any issues with the models of routers you've been using.

From 31st October 2022, I no longer have a regular presence here as I’ve moved on to a new role.
Anoush Mortazavi
Plusnet
Bobsta
Grafter
Posts: 61
Thanks: 6
Registered: ‎23-12-2011

Re: Latency spikes when streaming - especially Netflix

Hi @Gandalf - thanks for continuing to look into this.

My quality graph snapshot for the past 24hrs is here:

989bb8abd584deb28e13a8f026b489a4d9fa3aec-08-09-2019.png

 

I would agree that a bit of yellow isn't in itself a problem.... The problem is that it's so pronounced when streaming to even a single device in the house. If, instead, I kick off a regular download using all of my bandwidth, the latency doesn't rise by anything like as much. So it's either that traffic to/from iPlayer, Amazon, YouTube, Netflix, et al is treated differently, or the bursty nature of these transfers causes some sort of anomoly.

 

As you can see from the youtube video I posted in the second post, I'm able to download huge files at a sustained rate of nearly 100% of my available download speed whilst maintaining a RTT ping of ~15ms. As soon as I start a stream from Netlix using 10Mbps, my pings bounce up to 150+ms.

 

Here's a ping -t currently whilst the kids watch House of Anubis:

Pinging bbc.co.uk [151.101.64.81] with 32 bytes of data:
Reply from 151.101.64.81: bytes=32 time=7ms TTL=58
Reply from 151.101.64.81: bytes=32 time=7ms TTL=58
Reply from 151.101.64.81: bytes=32 time=16ms TTL=58
Reply from 151.101.64.81: bytes=32 time=67ms TTL=58
Reply from 151.101.64.81: bytes=32 time=10ms TTL=58
Reply from 151.101.64.81: bytes=32 time=11ms TTL=58
Reply from 151.101.64.81: bytes=32 time=9ms TTL=58
Reply from 151.101.64.81: bytes=32 time=16ms TTL=58
Reply from 151.101.64.81: bytes=32 time=152ms TTL=58
Reply from 151.101.64.81: bytes=32 time=11ms TTL=58
Reply from 151.101.64.81: bytes=32 time=9ms TTL=58
Reply from 151.101.64.81: bytes=32 time=8ms TTL=58
Reply from 151.101.64.81: bytes=32 time=10ms TTL=58
Reply from 151.101.64.81: bytes=32 time=8ms TTL=58
Reply from 151.101.64.81: bytes=32 time=198ms TTL=58
Reply from 151.101.64.81: bytes=32 time=8ms TTL=58
Reply from 151.101.64.81: bytes=32 time=8ms TTL=58
Reply from 151.101.64.81: bytes=32 time=9ms TTL=58
Reply from 151.101.64.81: bytes=32 time=173ms TTL=58
Reply from 151.101.64.81: bytes=32 time=15ms TTL=58
Reply from 151.101.64.81: bytes=32 time=9ms TTL=58
Reply from 151.101.64.81: bytes=32 time=7ms TTL=58
Reply from 151.101.64.81: bytes=32 time=233ms TTL=58
Reply from 151.101.64.81: bytes=32 time=9ms TTL=58
Reply from 151.101.64.81: bytes=32 time=26ms TTL=58
Reply from 151.101.64.81: bytes=32 time=8ms TTL=58
Reply from 151.101.64.81: bytes=32 time=9ms TTL=58
Reply from 151.101.64.81: bytes=32 time=8ms TTL=58
Reply from 151.101.64.81: bytes=32 time=127ms TTL=58
Reply from 151.101.64.81: bytes=32 time=7ms TTL=58
Reply from 151.101.64.81: bytes=32 time=7ms TTL=58
Reply from 151.101.64.81: bytes=32 time=8ms TTL=58
Reply from 151.101.64.81: bytes=32 time=195ms TTL=58
Reply from 151.101.64.81: bytes=32 time=9ms TTL=58
Reply from 151.101.64.81: bytes=32 time=9ms TTL=58
Reply from 151.101.64.81: bytes=32 time=10ms TTL=58
Reply from 151.101.64.81: bytes=32 time=166ms TTL=58
Reply from 151.101.64.81: bytes=32 time=7ms TTL=58
Reply from 151.101.64.81: bytes=32 time=8ms TTL=58

... it's those responses over 150ms that cause the VoIP and gaming nightmare.

 

So... yes please, I'd like to work with you to get this investigated further. Happy to try out a Hub One as part of the troubleshooting.

 

Thanks,

 

Bobsta

 

BTW... whilst I have your attention, could you take a look at my billing please? Earlier in the year I was mis-billed (missed a couple of months, then a big single bill, then a few months of over double what I should be paying)... my monthly payments are now correct but I believe I've over-paid by around £40 and I can't work out a sensible way to get this refunded/corrected. Could you nudge the appropriate folks please? Thx

Gandalf
Community Gaffer
Community Gaffer
Posts: 26,563
Thanks: 10,265
Fixes: 1,599
Registered: ‎21-04-2017

Re: Latency spikes when streaming - especially Netflix

Hi @Bobsta thanks for the extra detail. Actually before I order a new router I'd like to discuss this further with a colleague. I'm not sure what impact this would have however we use something called caching for popular streaming services such as YouTube and Netflix. I'd hope I'd be able to come back to you tomorrow but I may need a couple of days.

BTW... whilst I have your attention, could you take a look at my billing please? Earlier in the year I was mis-billed (missed a couple of months, then a big single bill, then a few months of over double what I should be paying)... my monthly payments are now correct but I believe I've over-paid by around £40 and I can't work out a sensible way to get this refunded/corrected. Could you nudge the appropriate folks please? Thx


I'm sorry to see your bills have been wrong, I'll get that sorted for you tomorrow when I'm back in the office.

From 31st October 2022, I no longer have a regular presence here as I’ve moved on to a new role.
Anoush Mortazavi
Plusnet
Anonymous
Not applicable

Re: Latency spikes when streaming - especially Netflix


@Gandalf wrote:

... ...  I'd like to discuss this further with a colleague. I'm not sure what impact this would have however we use something called caching for popular streaming services such as YouTube and Netflix.


 

@Gandalf , when you have that conversation, could you also find out whether the use of Plusnet caching CDN is determined by the use of Plusnet or third-part DNS ?,  i.e. presumably using Plusnet DNS would point to your caching CDN servers, whereas I'd assume that using say Goggle DNS pointing at the streaming service provider's regional servers (such as Netflix) would bypass Plusnet CDN - or is that stream somehow intercepted by your network ?, as this may be another variable with @Bobsta 's problem (if it is caching related).

 

Gandalf
Community Gaffer
Community Gaffer
Posts: 26,563
Thanks: 10,265
Fixes: 1,599
Registered: ‎21-04-2017

Re: Latency spikes when streaming - especially Netflix

@Nibiru Happy to ask the question.

@Bobsta Not been able to catch the right person yet to talk about this so I've posted a question on their portal now and I'll let you know when I know more although I'm out of the office tomorrow, but I'll follow up on Wednesday. I've also and hopefully sorted the issues with the billing of your account.

From 31st October 2022, I no longer have a regular presence here as I’ve moved on to a new role.
Anoush Mortazavi
Plusnet
Anonymous
Not applicable

Re: Latency spikes when streaming - especially Netflix

@Bobsta , I'm really interested in seeing what the answer is to your problem, as the Plusnet download QoS has been rubbish since they disabled the old system around June/July 2017 - which used to be fantastic !.

 

Fortunately I run a pfSense router so have been able to configure my own QoS to overcome most of the latency problems.  Having spent years mucking around with this stuff, you look like you're doing everything that I would have tried with your ERX.

 

A few things to try -

 

1) I think there is an updated firmware for the ERX, I doubt it will make any difference but it's worth a try.

 

2) I found pinging "bbc.co.uk" (with IPv4) to be a bit unstable, try using "ntp.plus.net" as it's fewer hops.

 

3) Does the ERX have a CPU monitor ?, or could you run "top" on a linux command line ?

    I'm wondering what the CPU load, and load averages look like when running dslreports and netflix ?

    My feeling is that for some reason the ERX is overloaded when streaming.

    I can't otherwise think of a reason why your ping graphs look so bad for specific traffic.

 

4) What happens if you significantly reduce the up and down available bandwidths on your ERX QoS,

    for example limit the downstream to say 55Mbps and the upstream to 12Mbps.

 

5) What DNS do you have configured on the ERX ?, I'm just pre-empting this detail for @Gandalf 's reply on CDN caching.

 

6) Is your ERX acting as a VPN endpoint, or IPv6 tunnel endpoint, for any traffic on your network ?

 

7) It sounds like you've setup QoS on your ERX WAN port.  Is it possible to configure the ERX to further bandwidth limit specific WAN traffic (i.e. Netflix etc) ?, or restrict the bandwidth to specific LAN devices - such as a smart-TV or whatever streaming device is being used.

 

I'll keep thinking about what else could be tried Roll_eyes

Bobsta
Grafter
Posts: 61
Thanks: 6
Registered: ‎23-12-2011

Re: Latency spikes when streaming - especially Netflix

@Gandalf - Thanks for sorting out the billing issue - appreciated.

@Anonymous - Thanks for the extra input, glad to have you along for the ride. Smiley

 

The more I investigate this the more I'm beginning to think this isn't a specific Netflix / Amazon / YouTube thing but some other weird network congestion issue. I'm leaning towards it being something misconfigured with my local FTTC cabinet (that was newly installed earlier this year) but clearly troubleshooting that is going to be outside of my direct control.

 

Things I've observed:

  • A single constant heavyweight download or two doesn't cause latency spikes.
  • Maxing out my upload doesn't cause latency spikes (recently I was away from home and was streaming content from my Plex Media Server at 15Mbps+ and my quality graph was spotless).
  • Rapid websurfing can trigger latency (especially sites with loads of graphics / media). In particular, the first load of website (maybe the initial TCP handshake/connection to a new address causing the issue?). As an example, I can sit with a constant ping running in a terminal window (9ms to a VM I have running in Digital Ocean, London)... type "sky.com" into a browser and as I hit return - boom a single 150ms ping, then straight back to 9ms. But when I refresh the page latency doesn't budge. (and this isn't machine specific - I can do this across multiple devices - MacOS and Windows) 
  • Streaming (YouTube, Amazon, Netflix, iPlayer ... even embedded videos in corporate websites served from AWS or Azure) causes big latency spikes
  • Any QoS I apply on my router seems to have zero effect. I've tried two routers (Asus RT-N56U and Ubiquiti ER-X) both don't seem to be suffering with CPU topping out.
  • Setting QoS throughput levels artificially low (currently15Mbps down) helps improve the latency spikes (50ms rather than 200ms) - maybe this indicates that reducing load on the network helps?
  • I normally run OpenDNS Family Shield DNS servers but have tested today using Plusnet's regular ones and it makes no difference.

So with that said, I think the CDN / caching point may be a bit of a red herring.

With a bit of work I could potentially put a bandwidth cap on every device in the house but I don't think that's a workable solution.  

 

Where do we go next?

 

Gandalf
Community Gaffer
Community Gaffer
Posts: 26,563
Thanks: 10,265
Fixes: 1,599
Registered: ‎21-04-2017

Re: Latency spikes when streaming - especially Netflix

@Bobsta No problem and thanks for the extra (extra) detail, based on what you’ve said it would appear that a potential caching issue may be a red herring if that’s even possible to be the cause.

It’s probably worth getting an engineer arranged to investigate further. I’m not in the office tomorrow, happy to pick this back up on Wednesday but if you want to arrange an engineer hopefully sooner I’d raise a fault to us at http://faults.plus.net dropping us a reply back here and one of my colleagues will pick this up as soon as possible. 

From 31st October 2022, I no longer have a regular presence here as I’ve moved on to a new role.
Anoush Mortazavi
Plusnet
Bobsta
Grafter
Posts: 61
Thanks: 6
Registered: ‎23-12-2011

Re: Latency spikes when streaming - especially Netflix

Hi @Gandalf ... sorry, I dropped the ball on this as have had a hectic week. The issue is still very much present (see below) so I've just tried gone through the fault reporter and logged a fault case, mentioning this thread in the comments.

 

14 Sep 201914 Sep 2019

Did you still want to send me a Hub One to test? I don't expect it to make any difference but happy to follow the process if it helps to avoid any finger-pointing towards my kit.

 

Many thanks,


Bobsta