Started getting lots of PPPoE disconnects lately

[#1]

I will start by prefacing this with my setup - VDSL 78Mb/20Mb service with an Openreach branded Huawei HG612 DSL modem and a Pfsense 2.8.1 router running on a Fujitsu Futro S920 with 2nd PCI NIC. This setup has worked without issue for several years.

I've been trying to diagnose a problem that started just over a week ago. My PPP connection was dropping about every 1-2 minutes, then reconnecting again a few seconds later. After calling Plusnet they couldn't identify any issues but sent out a new Plusnet hub 2 as my original Plusnet hub didn't work any more for whatever reason.

I spent a lot of time last weekend trying to troubleshoot what the cause of the disconnects was and I'll try and keep this short. I eliminated any ppp daemon crashing on the pfsense router and took some tcpdump captures and anaylsed with wireshark to try to find what initiated the drops.

What I found seems to be packets flowing normally, then there's an Active Discovery Terminate (PADT) packet with source of 40:55:82:da:a9:48 (Nokia according to mac vendor lookups) and a destination of 00:19:99:b8:85:b5 (Fujitsu Tehcnology) followed by presumably an acknowledgement of that back from my router to the Nokia device then 3 seconds later an Active Discovery Initiation (PADI)  where the reconnect sequence starts. I've pasted the packet sequence below:

No. Time Source Destination Protocol Length Info
81051 64.190099 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPPoED 60 Active Discovery Terminate (PADT)
81052 64.190145 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPPoED 38 Active Discovery Terminate (PADT)
81053 67.889037 FujitsuTechn_b8:85:b5 Broadcast PPPoED 36 Active Discovery Initiation (PADI)
81054 67.901248 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPPoED 72 Active Discovery Offer (PADO) AC-Name='acc-aln12.bz'
81055 67.901273 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPPoED 72 Active Discovery Request (PADR) AC-Name='acc-aln12.bz'
81056 67.913243 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPPoED 60 Active Discovery Session-confirmation (PADS)
81057 67.913545 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 38 Configuration Request
81058 67.957000 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Configuration Request
81059 67.957265 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Configuration Reject
81060 67.957385 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 41 Configuration Ack
81061 67.957568 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 36 Configuration Request
81062 67.962014 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Configuration Ack
81063 67.962026 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP CHAP 95 Challenge (NAME='acc-aln12.bz', VALUE=0x47397cf7304e2c74c0964f32f287f36a5c2b221428f195c00e8ea68091905dd8c9d95079…)

Then response with my plusnet username and then a CHAP authentication success.

So it looks like the telcoms equipment at the other end of the ADSL line is initiating a terminate for some reason. Sometimes the drops happen every few minutes, other times it'll go hours or even a day without a drop.

The new Plusnet hub 2 arrived. So on Friday afternoon I put it in bridge mode and connected it up - same problem. Spoke to Plusnet again, no issues they can see. Tried it in normal (non bridge mode) and no drops for 10minutes (just before in bridge mode it was dropping about every minute). Now I didn't really have any devices connected in this mode (may need further tests). But as it didn't drop in normal mode there wasn't much Plusnet support could do. I put it back in bridge mode to connect back up to the rest of my network again. It's been dropping sporadically without any discernable pattern.

Yesterday I started wondering if it was dropping during high bandwidth usage but started a game update which maxed out the link for around 45 minutes and no drops.

I'm utterly stumped at this point what is going on and hoping someone somewhere might have an idea, or can confirm it's the other end dropping my connection rather than something on my end. But noone seems to be able to find any issues. I know we can eliminate the Huawei HG612 DSL modem as the same problem exists with the plusnet hub 2. I can't rule out something on my pfsense router but I haven't been able to find any evidence yet despite quite a bit of digging.

I really don't want to have to use the Plusnet hub 2 in regular configuration and migrate my whole network over to that (it's not a straightforward/simple setup).

[#2]

@jbeeston welcome to the community

The Hub two has a event log which would be worth looking at.

You need to connect a PC to a LAN port on the Hub two with an IP Address between 192.168.1.1 to .253 Class C mask.

The navigate from ->Home -->Advanced settings --->Technical log ---->Event log.

Look for DSL events, note the log file will not be correctly time stamped as there is no NTP client in bridge mode

Is the Hub light staying a pink/purple colour or is it flashing orange or solid orange. This would indicate a line issue.

[#3]

@jbeeston have you checked the line stats on either the HG612 or the HUb 2 when in modem mode ?

My guess is that you have an underlying line problem and its causing LCP timeouts on the PPPoE link.

Its possible that the Hub 2 in router mode has different LCP timeout parameters to your PfSense router and is therefore more resilient. Problem is that the LCP parameters are not visible on the Hub 2. They should be visible on PfSense , usually called LCP Echo failure & LCP Echo Interval.

[#4]

---

@Dan_the_Van wrote:

@jbeeston welcome to the community

The Hub two has a event log which would be worth looking at.

You need to connect a PC to a LAN port on the Hub two with an IP Address between 192.168.1.1 to .253 Class C mask.

The navigate from ->Home -->Advanced settings --->Technical log ---->Event log.

Look for DSL events, note the log file will not be correctly time stamped as there is no NTP client in bridge mode

Is the Hub light staying a pink/purple colour or is it flashing orange or solid orange. This would indicate a line issue.

---

I've looked through the event log but it has nothing other than lots of NTP sync failures as expected in bridge mode and some events whenever I connect a laptop to one of the plunet hub ports to access the admin interface.

The hub light is pink for bridge mode. I haven't seen it during a disconnect event to see if it changes but I highly doubt it. I watched the HG612 during disconnect events and there was no flashing of the DSL sync light so I don't believe the DSL connection is dropping.

I've attached the details from the hub's info page with stats.

[#5]

---

@MisterW wrote:

@jbeeston have you checked the line stats on either the HG612 or the HUb 2 when in modem mode ?

My guess is that you have an underlying line problem and its causing LCP timeouts on the PPPoE link.

Its possible that the Hub 2 in router mode has different LCP timeout parameters to your PfSense router and is therefore more resilient. Problem is that the LCP parameters are not visible on the Hub 2. They should be visible on PfSense , usually called LCP Echo failure & LCP Echo Interval.

---

I don't see obvious issues with timeouts. I have this filtered (pppoe.code in {0x09,0x07,0x19,0x65,0xa7} || (ppp && lcp)) section from the wireshark trace which shoes the LCP echo request and replies seem to be every 10 seconds. It looks likey an LCP Echo request comes every 10 seconds followed by my reply fractionally later back. Now the timing does show an echo request is expected close to the PADT but I don't know if it's misisng because it dropped just before the next scheduled echo request or the echo request was lost somewhere and therefore no reply was received (I don't know how long the timeout is set at other end of the DSL line but it would have to be sub-second for it to drop the line if no reply).

No. Time Source Destination Protocol Length Info
70 0.786625 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Echo Request
71 0.786749 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 30 Echo Reply
4560 10.986672 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Echo Request
4561 10.986786 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 30 Echo Reply
35053 21.487802 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Echo Request
35055 21.487942 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 30 Echo Reply
51888 31.486575 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Echo Request
51889 31.486711 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 30 Echo Reply
73091 42.286613 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Echo Request
73092 42.286777 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 30 Echo Reply
80346 53.386685 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPP LCP 60 Echo Request
80347 53.386825 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPP LCP 30 Echo Reply
81051 64.190099 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPPoED 60 Active Discovery Terminate (PADT)
81052 64.190145 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPPoED 38 Active Discovery Terminate (PADT)
81053 67.889037 FujitsuTechn_b8:85:b5 Broadcast PPPoED 36 Active Discovery Initiation (PADI)
81054 67.901248 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPPoED 72 Active Discovery Offer (PADO) AC-Name='acc-aln12.bz'
81055 67.901273 FujitsuTechn_b8:85:b5 Nokia_da:a9:48 PPPoED 72 Active Discovery Request (PADR) AC-Name='acc-aln12.bz'
81056 67.913243 Nokia_da:a9:48 FujitsuTechn_b8:85:b5 PPPoED 60 Active Discovery Session-confirmation (PADS)

I've yet to find where the LCP parameters are buried in pfsense. I don't see anything in Interfaces, WAN, Advanced.

[#6]

It looks likey an LCP Echo request comes every 10 seconds followed by my reply fractionally later back. Now the timing does show an echo request is expected close to the PADT but I don't know if it's misisng because it dropped just before the next scheduled echo request or the echo request was lost somewhere and therefore no reply was received

It does look as though the other end is not seeing an LCP echo reply and thus terminating the PPPoE link. As you say, we dont know what the timeout parameters are at the other end. If thats the case, then adjusting your parameters wont have any effect.

Looking at your line stats, the downstream SNR is pretty low. It looks like the DSLAM has set a 3db target, which whilst it will allow a higher speed to be negotiated, it will increase the likelyhood of errors. Plusnet do tend to provision VDSL lines on a 'speed' profile, so it might be worth asking for your line to be re-provisoned on a 'stable' profile. You will lose a bit of speed but might give more stability.

[#7]

---

@MisterW wrote:

It looks likey an LCP Echo request comes every 10 seconds followed by my reply fractionally later back. Now the timing does show an echo request is expected close to the PADT but I don't know if it's misisng because it dropped just before the next scheduled echo request or the echo request was lost somewhere and therefore no reply was received

It does look as though the other end is not seeing an LCP echo reply and thus terminating the PPPoE link. As you say, we dont know what the timeout parameters are at the other end. If thats the case, then adjusting your parameters wont have any effect.

Looking at your line stats, the downstream SNR is pretty low. It looks like the DSLAM has set a 3db target, which whilst it will allow a higher speed to be negotiated, it will increase the likelyhood of errors. Plusnet do tend to provision VDSL lines on a 'speed' profile, so it might be worth asking for your line to be re-provisoned on a 'stable' profile. You will lose a bit of speed but might give more stability.

---

I can't imagine they drop the connection after just one failed LCP reply but it's possible that whilst I see replies being sent my end they don't make it to the other end resulting in a number of missed replies being heard but were sent.

Interesting on the SNR and profiles. I did note this started happening after the strong winds and rain on the Thursday just before this issue started - might be coincidence and seems a bit of a stretch but it could affect signal quality I suppose.

Speed wise (and I don't know if this implies anything useful at all) at the moment I get 67Mbps down/18.5Mbps up. Which doesn't seem too bad to me if the line is poor.

Any idea what sort of difference in speed an adjustment from speed profile to stable profile is likely to get?

Can you help me interpret the SNR numbers from the hub2 - what's the first number and what's the number after the / ? Are you saying that for noise margin the line target is 3.3 but I'm nearly double that at 6.8?

Appreciate the replies from everyone so far - this has been driving me somewhat mad, not to mention the amount of time trying to troubleshoot it.

[#8]

Can you help me interpret the SNR numbers from the hub2 - what's the first number and what's the number after the / ?

The first number is the upstream SNR and after the / is the downstream

Are you saying that for noise margin the line target is 3.3 but I'm nearly double that at 6.8?

By default , the downstream target SNR is 6db so you would expect an actual SNR of around that sort of value. If over a period of time, the DSLAM sees low error counts, then in an attempt to increase speed, it can lower the target to 3db. With you actual SNR showing as 3.3, that would seem to be the case. If errors then start to increase the DSLAM SHOULD raise the target back to 6db but it can take a while for this to happen.

Interesting on the SNR and profiles. I did note this started happening after the strong winds and rain on the Thursday just before this issue started - might be coincidence and seems a bit of a stretch but it could affect signal quality I suppose.

That could well be significant, maybe the winds and rain have affected a joint somewhere and with the low SNR , the error counts are increasing. SNR does vary over the day, IIRC its worse in the evening. It might be worth keeping an eye on the value and then if/when it gets to its lowest, then try a resync. That will then put you back at 3db in the worst case.  Maybe worth putting the HG612 back as it can give better stats (I think?)

TBH its a long time since I was on a VDSL connection and so have forgotten about much of this stuff. FTTP just doesnt have these problems, it either works at the provisioned speed or not at all😀

[#9]

@jbeeston 

Maybe worth putting the HG612 back as it can give better stats (I think?)

The Hub two does not support any router stats app, not sure if you are aware, take a look at this link

https://kitz.co.uk/routers/hg612unlock.htm 

The Hub two only useful event log messages are DSL link down and up in bridge mode.

[#10]

@Dan_the_Van  @MisterW 

Good idea, I think I'll put the HG612 back in and then see what I can get stats wise out of it. I've got the unlocked firmware so with some luck I can get some useful stats out of it. That link on HG612 looks like it might be interesting if I can get some software to regularly grab line stats over time. I'm also tempted to try running the Plusnet hub 2 in regular mode for a few hours at least and leave the rest of my network internetless in the name of science - it would be nice to see it happen with the standard Plusnet setup (where I might get a bit more help, they say bridge mode not really supported).

I keep waiting for FTTP to arrive here (North Basingstoke) but it seems Openreach keep messing around! About 3 years ago it was "building in this area" then it went to something equivalent of "we're planning this area" then a year or so ago to "we're building in this area" which it remained in for I guess over a year. Checking today I see "We're planning to build in this area". Seriously........ It's not even like it's out in the middle of nowhere. On a separate subject, the dl/up ratio on openreach fttp is disappointing to say the least. What's even more annoying is my parents live out in the sticks in france and have symetric 1gbps fibre. Rant over.