Packet Loss at Peak Times

[#1]

I am seeing packet loss on my service, usually at peak times.

My firewall has a built in connection monitor, which is reporting loss, usually at peak times when attempting to reach 8.8.8.8. It also reports loss when pinging the PlusNet DNS servers - this loss is much higher and is 24/7.

I also have a thinkbroadband connection monitor configured - this is also showing loss when attempting to contact my public facing IP address. Again, this is mainly at peak times.

For the most part, the loss is pretty insignificant, but at its worst, it is throwing me off online game sessions, disconnecting my VPN sessions to work & disconnected my wife's RDP sessions when working from home.

My home is fitted with data cable throughout & most devices, such as my gaming PC are hardwired, but are being disconnected the same time the loss is being reported by the firewall. On the few devices which aren't wired, phones & WFH laptop, the experience is the same.

Steps I have taken to troubleshoot the issue:

• Replaced ADSL filter
• Connected router directly to master socket
• Tried alternative router/firewall
• Hardwired problematic devices where possible

I've attached a few images below, 3 of which are reports from my firewall the previous 24 hours to Google DNS, PN DNS1 & DNS2. The other shows a previous 2 hour report from now (17-May @ 21:25), which you can see two instances in the last two hours where my service has suffered approx 25% packet loss for about 2-3 minutes - this is enough to disconnect game sessions, VPN & RDP. There are also two PingPlotter reports, which both show packet loss at the exact same time to PN DNS & Google DNS - these have been ran from a PC wired in to the network. Finally, there is the graph from ThinkBroadband monitor, which pretty much correlates with all of the others.

You can also see from the reports that through the early hours of the night/morning, it is much more stable.

Given the number of other users who are reporting the same issues, are PlusNet reaching some kind of capacity on their network? Given more people are at home, working from home, etc?

Are there any further meaningful troubleshooting steps I can take, which can assist you further? 

Oh and one last thing which I have also included as an attachment, is the BT wholesale speed test & line test. The line is syncing at the correct rate & performs perfectly fine at the correct rate when this random packet loss issue doesn't occur.

[#2]

Suffered the exact same thing again this morning at 9:30am - Over 30% loss this time, kicked me off mid video conference. Very annoying.

As per yesterday, no loss and stable overnight, loss starts to occur around normal working hours.

See attached images for clarification.

[#3]

I don't know what's going on but my connection has been getting steadily worse over the past few weeks. Sync rate is still 80/20 so it's not that. I just looked at my Packet Loss graph on Samknows and it's gone from 0 to nearly 50% in the last month. What?

[#4]

It's weird - I just don't get it. So far today, for mine, it's happened at half past the hour, every hour since 8:30.

[#5]

So it's now 5pm and this issue has happened every hour since 7:30 this morning.

• 7:30
• 8:31
• 9:32
• 10:33
• 11:34
• 12:35
• 13:36
• 14:37
• 15:38
• 16:39

You can see here: Monitor 

[#6]

Ignoring your other monitors for a minute, Google rate limit ICMP traffic to their DNS resolvers. They are not intended for uptime monitoring which you seem to be using them for. I wouldn't be surprised if those periodic spikes of ~25% 'packet loss' are actually the Google DNS servers refusing your pings.

Perhaps worth setting a monitor up to use something more sensible like pingbox1.thinkbroadband.com.

Those inbound spikes on the TBB graphing are interesting, especially given they only occur during waking hours. Could it be that there's a device/machine on your network that's only active during this time and is generating bursts of upload/download traffic on the hour? Do some of your other monitors only run during this time? It's not unheard of for certain routers to periodically defer ping requests if you're generating bucketloads of ICMP traffic. I've seen it with a couple of devices myself in the past.

The loss to the Plusnet DNS addresses is very low at ~1% but it does seem almost consistent. It would be interesting to see what running WinMTR or something similar shows over a period of time as it would at least show where the loss is.

@KevinG I assume you've tried rebooting everything? No heavy uploading/downloading going on?

[#7]

Bob,

Thanks for the reply - whilst I agree Google DNS probably isn't the best thing to test, it's what my firewall had set as default and I'd just never changed it. I will add the TBB pingbox though. Having said that, all internet bound traffic on my network fails at the same time as the reports on the graphs show loss to Google, so I'm not willing to write off loss to google as ICMP rate limiting. As I say, my VPNs to work drop & reconnect and the wifes WFH solution involves RDP, which freezes, drops & reconnects, so there's definitely a complete loss, albeit brief, of connectivity.

I've attached a report which shows the devices connected to the network & usage for the previous 24 hours - which is largely the same as any other day. There's a live data stream which I've monitored and each time the packet loss is reported, there's no substantial network activity going on.

I've had ping plotter running all day to Google DNS & one of PlusNet's DNS servers, which I am happy to upload the entire data set to, if you'd like to review?

Thanks

[#8]

@bobpullen I had rebooted the router at least once but I have now powered off and on the modem and the router and removed the Samknows box from the circuit and everything is much better at present. I will keep an eye on it. There has been no particularly unusual activity although usage will have been higher during lockdown.

[#9]

So based on the latest graphs, it looks as though as soon as the usage of the service drops down, so does the packet loss.

As an experiment, I queued up about 80gigs worth of downloads (8x 10G files) before I went to bed, most of them had failed by the morning, but you can see from the TBB report that the loss begun to start again as soon as more than idle traffic was seen, at about 30 mins past every hour for about 5 hours. The downloads then failed. My own monitor to TBB pingbox also shows the same.

The only remaining piece of hardware in my house which I haven't changed, is the BT OpenReach white modem - I've ordered one and will replace it as soon as it arrives. If this doesn't fix the issue, then it must be external to me as every solitary piece of equipment & cable has been replaced in my home.

[#10 · Accepted solution]

Okay so I finally got to the bottom of this... as a network engineer by day, usually come 5pm I have absolutely no desire to start playing around with my own stuff to levels I usually only deal with at work!

So I managed to narrow the issue down to it only happening when only one device connected to the network & that just so happened to be the device I was using for my testing (My trusty PC of 5 years!). It seemed that if that device came online at say 8:30 when I started work, I would get the loss & latency every hour, at 30 minutes past the hour. If I brought this device online at 45 minutes past the hour, it was then every hour from that point. I proved this by not connecting this device for over a day & the issue disappeared. It's worth noting, when this device was connected, not only was its connection to the Internet disrupted, but every other device on the network too. Ping times to any internet destination increased & loss began to set in.

Now, to work out what was causing it... Checked all of the obvious network stuff, NIC drivers, cables, CRC errors, etc etc. All looked fine. I then wondered, could it be STP on my switch playing up, when the network port for my device came online, possibly causing a re-converge and disrupting everything? Nope... wasn't that. So I figured I'd run a wireshark capture, then connect the network cable... low and behold within a few minutes, TCP retransmission's went through the roof, latency started to increase and packets were dropped. At this point, I couldn't see anything blindingly obvious as to what was happening prior to everything going crazy, so I created a rule on the firewall on that device to block all outbound connections - the issue disappeared!! After much faffing around, I have managed to nail this down to the "iCloud Photo Library' service - when this service is running, as soon as it sees a network connection, it starts transmitting & breaks everything for a few minutes. From the Wireshark captures, you can see various mDNS queries etc all relating to Apple, so perhaps it is related to that. At this point I no longer care The software is now disabled and my connection is stable again. It's Friday night, I shouldn't be looking at Wireshark traces

In all my years of doing my job, I've never seen a piece of Apple software on one client machine break an entire LAN?!

Anyway, I figured I'd post my findings and hope that one day it might help someone else from tearing their hair out if this post ever finds its way on to Google.