Inbound traffic suddenly stops

DavidJ · ‎30-07-2007

Hi

I seem to have a problem where inbound traffic (or possibly just dns responses) from the internet suddenly ceases causing my router to disconnect and reconnect the pppoe session. Everything then works fine for a while before the problem recurs and the cycle continues.

Here's a bit more detail.

This started in December when, after a router firmware update (my router is a Ubiquiti Dream Machine), my wan connection started disconnecting/reconnecting every 20 minutes or so. Assuming this was a problem with the new firmware, I spoke to Ubiquiti tech support who looked at router log files and decided the problem was caused by wan dns response timeouts. I wasn’t convinced this was the cause as reverting to the older firmware seemed to fix the problem. However, the problem has reappeared on occasions even with the older firmware.

I've now had time to look at some wan packet captures taken when the problem was happening and I can see that dns responses do suddenly cease. What I can see is this: the pppoe session is established and everything works fine for, typically 20 minutes or so. Dns requests are made to the specified internet dns servers and responses are received. Then dns responses suddenly stop. But I think it might be more than just dns responses. When the problem occurs there suddenly seems to be no inbound traffic at all. I can see the router sending repeated dns requests using the specified dns servers and also to 1.1.1.1 and 8.8.8.8 but no responses are received. It also tries to ping those servers but no ping responses are received. In fact, there appears to be no inbound traffic at all. I don't know whether this lack of inbound traffic is caused by the inability to perform dns resolutions but I would have expected clients on my network to continue conversations using their dns caches. Anyway, the end result is the same - when the router decides it can't reach any dns servers it disconnects and reconnects the pppoe session. And the cycle continues - everything works for a while then inbound traffic stops causing another disconnect/reconnect.

I’ve tried specifying different dns servers - Plusnet’s, OpenDNS (they’re the ones I normally use), 1.1.1.1 and 8.8.8.8 but it makes no difference.

What could be causing this? I'm wondering if I'm triggering some sort of alert which then blocks traffic?

Thanks for any light you can shed on this.

David

Townman · ‎22-08-2007

Is this a simple matter of a loss of line sync?

What connection technology is involved here? ADSL / VDSL / FTTP?

In another browser tab, login into the Plusnet user portal BEFORE clicking the fault & ticket links

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

If this post helped, please click the Thumbs Up and if it fixed your issue, please click the This fixed my problem green button below.

corringham · ‎25-09-2015

I'd consider setting up a ThinkBroadBand (TBB) Broadband Quality Monitor - that will give you an idea whether the problem is just at the times you notice, or whether there is more frequent packet loss.

Townman · ‎22-08-2007

Also I explore setting up router stats. There’s a possibility of REIN / SHINE causing high transmission errors … but not causing a loss of sync.

In another browser tab, login into the Plusnet user portal BEFORE clicking the fault & ticket links

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

If this post helped, please click the Thumbs Up and if it fixed your issue, please click the This fixed my problem green button below.

DavidJ · ‎30-07-2007

Thanks for the replies.

We’re talking VDSL here. I have a Draytek Vigor 166 modem which shows the line has been up for more than 1000 hours. The line stats look good and it hardly ever drops. When I spoke to Plusnet’s 1st line tech support they tested the line and found no problem. All they could see was the repeated pppoe session disconnects/reconnects.

Let me add a few additional facts that might help (or possibly confuse further!)

If I swap out my router for a more common modem/router such as a BT Hub, the problem doesn’t occur.
If I update my router to the latest firmware the problem happens all the time.
If I stick with my current firmware with a fresh boot-up of the router then everything is fine - it’ll run for weeks with no problem.

This is where it gets weird...

If I then make a configuration change to the router (and it can be something that should have no impact on the wan connection, such as turning the status LED on/off or adding a mac address to the list of permitted WiFi clients) the problem will start.

So the router is definitely a factor in this.

However, when the problem is happening, the router’s logs show repeated dns response timeouts and this is what leads to the router deciding to drop and restart the pppoe session. Everything then works fine for about 20 minutes before the problem recurs and so the cycle continues.

And if I look at a wan packet capture taken at that time, I can see those repeated dns requests and then pings (without replies) to those dns servers and then the request to terminate the session. But I also notice that it’s not just dns and ping responses that aren’t arriving; suddenly there’s virtually no incoming traffic at all.

So while the router is definitely a factor, it can’t be responsible for the absence of dns responses and other incoming traffic. It seems as if there’s something that the router is doing that’s triggering a block on traffic. I know that’s an assumption but the above facts would suggest it’s not solely a router or Plusnet thing.

If some alert or block is being triggered I’m assuming this would be logged. I can provide exact dates/times of when the problem occurs if this would help.

Does this help (or confuse further)?

Again, thanks for any light you can shed on this.

MisterW · ‎30-07-2007

I can see those repeated dns requests and then pings (without replies) to those dns servers and then the request to terminate the session.

@DavidJ is your router trying to check the internet connection by pinging the gateway or dns servers ?

Plusnet's DNS servers wont respond to pings.

Can you disable the checks ?

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

DavidJ · ‎30-07-2007

It pings a specific server - ping.ui.com (although this can be changed but not disabled) - to test the internet connection.

However, what I'm seeing in the packet capture is something else. When responses to the dns requests time out, it will repeat those requests a number of times before trying other dns servers - 1.1.1.1 and 8.8.8.8. If it still gets no responses it starts pinging the servers. I guess this makes sense - it will retry the specified servers before trying alternative servers and then try to determine if there's any life out there at all before giving up and restarting the pppoe session.

I use the OpenDNS servers - 208.67.222.222 & 208.67.220.220 - and they do respond to pings.

MisterW · ‎30-07-2007

However, what I'm seeing in the packet capture is something else.

What are you actually capturing ? the logical pppoe interface or the physical eth port ?

I wonder if the UDM is just dropping the pppoe connection for some reason ?

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

Dan_the_Van · ‎25-06-2007

Couple of basic checks,

 nslookup ping.ui.com
Server:  UnKnown
Address:  192.168.10.1

Non-authoritative answer:
Name:    ping2.ui.com
Addresses:  1.1.1.1
          8.8.8.8
Aliases:  ping.ui.com

Maybe a traceroute is in order before the PPPoE connection gives up, using mtr ping.ui.com

 mtr -c 1 -w -z ping.ui.com
Start: 2024-03-26T13:30:14+0000
HOST: wireguard                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    192.168.10.1                          0.0%     1    1.1   1.1   1.1   1.1   0.0
  2. AS6871   195.166.130.249                       0.0%     1    7.4   7.4   7.4   7.4   0.0
  3. AS6871   gi2-23-442.ptn-gw02.plus.net          0.0%     1    8.0   8.0   8.0   8.0   0.0
  4. AS2856   core1-BE1.colindale.ukcore.bt.net     0.0%     1    8.2   8.2   8.2   8.2   0.0
  5. AS2856   peer3-et-0-0-1.redbus.ukcore.bt.net   0.0%     1    8.3   8.3   8.3   8.3   0.0
  6. AS2856   109.159.253.3                         0.0%     1   10.9  10.9  10.9  10.9   0.0
  7. AS15169  209.85.255.77                         0.0%     1    9.4   9.4   9.4   9.4   0.0
  8. AS15169  172.253.65.211                        0.0%     1    9.8   9.8   9.8   9.8   0.0
  9. AS15169  dns.google                            0.0%     1    9.0   9.0   9.0   9.0   0.0

mtr -c 1 -w -z ping.ui.com
Start: 2024-03-26T13:31:50+0000
HOST: wireguard                                       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS???    192.168.10.1                             0.0%     1    1.0   1.0   1.0   1.0   0.0
  2. AS6871   195.166.130.249                          0.0%     1    7.6   7.6   7.6   7.6   0.0
  3. AS6871   gi2-23-442.ptn-gw02.plus.net             0.0%     1    8.0   8.0   8.0   8.0   0.0
  4. AS2856   core1-BE1.colindale.ukcore.bt.net        0.0%     1    8.2   8.2   8.2   8.2   0.0
  5. AS2856   peer7-et-4-1-1.telehouse.ukcore.bt.net   0.0%     1    8.8   8.8   8.8   8.8   0.0
  6. AS???                                         100.0     1    0.0   0.0   0.0   0.0   0.0
  7. AS13335  141.101.71.47                            0.0%     1   13.2  13.2  13.2  13.2   0.0
  8. AS13335  one.one.one.one                          0.0%     1    8.6   8.6   8.6   8.6   0.0

DavidJ · ‎30-07-2007

What are you actually capturing ? the logical pppoe interface or the physical eth port ?

I'm capturing the router's physical wan ethernet port using tcpdump. I also temporarily put a managed switch between the router and the modem, mirrored the port to which the router was connected and did a capture from that port, just to verify what I was getting from tcpdump. The captures matched.

I wonder if the UDM is just dropping the pppoe connection for some reason ?

The UDM is intentionally dropping and restarting the pppoe session (I can see the Terminate packet being sent after all the dns requests and pings). I've spoken to Ubiquiti support and they confirmed this is what the UDM will do when it determines the internet connection is broken.

MisterW · ‎30-07-2007

Thanks, fairly conclusive then!

When you swapped out the UDM for the BT Hub , was the Vigor 166 still being used ? i.e the BT Hub was connected. via its WAN port.

Just trying to elimate the 166...

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

DavidJ · ‎30-07-2007

When you swapped out the UDM for the BT Hub , was the Vigor 166 still being used ? i.e the BT Hub was connected. via its WAN port.

No. The BT Hub I've got is a combined modem/router and doesn't seem to have any setting to change this. The cable from the phone socket was moved from the 166 to the BT Hub.

The Vigor 166 hasn't been updated or changed in any way since way before this problem started. When it's connected to the UDM it's in bridge mode so it shouldn't be doing any filtering/blocking. Also, things will work fine for weeks until I make a configuration change to the UDM which then triggers the problem. I can't prove it's not the 166 but it's always changes made to the UDM that trigger the problem.

MisterW · ‎30-07-2007

Could it be the UDM is triggering flow control on the ethernet link to the 166 ?

You would only expect it to be short lived though...

Superusers are not staff, but they do have a direct line of communication into the business in order to raise issues, concerns and feedback from the community.

Dan_the_Van · ‎25-06-2007

No. The BT Hub I've got is a combined modem/router and doesn't seem to have any setting to change this.

Is your BT Hub is a HH5?

When I connected my plusnet Hub One which is a rebadge BT HH5 to a PPPoE server I needed to factory reset it for it to switch to a router only device. I connected the WAN cable prior to the restart.

This may the case for other BT Hubs with a DSL and WAN port

HTH

DavidJ · ‎30-07-2007

Could it be the UDM is triggering flow control on the ethernet link to the 166 ?

Interesting. Hadn't thought of that.

The link between the 166 and UDM is a 1Gb full duplex link and my broadband download speed is about 70mbps at best so there shouldn't be any need for flow control. Doesn't mean it's not happening, though.

I've never had to play with flow control before. I'm assuming that if the UDM is telling the 166 to pause, then that should have been captured in the packet capture on the UDM's wan port. Is that correct?

I'll have a look when I get home and report back.