Looks like my problem is back with a vengeance - REIN / SHINE

[#1]

After a few quiet months I started to notice odd drops again on the line (see here for the first sad tale)

So firing up a syslog server I decided to check tonight as its been getting worse and found:

Nov  1 03:01:20 192.168.1.1 DrayTek: WAN 1 is down.

Nov  1 03:38:13 192.168.1.1 DrayTek: WAN 1 is down.

Nov  1 11:09:30 192.168.1.1 DrayTek: WAN 1 is down.

Nov  1 12:07:36 192.168.1.1 DrayTek: WAN 1 is down.

Nov  1 13:25:49 192.168.1.1 DrayTek: WAN 1 is down.

Nov  1 23:57:51 192.168.1.1 DrayTek: WAN 1 is down.

Nov  2 01:58:52 192.168.1.1 DrayTek: WAN 1 is down.

Nov  2 18:40:06 192.168.1.1 DrayTek: WAN 1 is down.

Nov  2 18:44:17 192.168.1.1 DrayTek: WAN 1 is down.

Nov  3 03:29:02 192.168.1.1 DrayTek: WAN 1 is down.

Nov  4 09:01:24 192.168.1.1 DrayTek: WAN 1 is down.

Nov  5 03:17:39 192.168.1.1 DrayTek: WAN 1 is down.

Nov  7 04:34:36 192.168.1.1 DrayTek: WAN 1 is down.

Nov  8 03:33:55 192.168.1.1 DrayTek: WAN 1 is down.

Nov 10 03:06:34 192.168.1.1 DrayTek: WAN 1 is down.

Nov 15 02:31:33 192.168.1.1 DrayTek: WAN 1 is down.

Nov 15 08:16:50 192.168.1.1 DrayTek: WAN 1 is down.

Nov 15 09:34:41 192.168.1.1 DrayTek: WAN 1 is down.

Nov 15 13:49:22 192.168.1.1 DrayTek: WAN 1 is down.

Nov 16 00:48:57 192.168.1.1 DrayTek: WAN 1 is down.

Nov 16 01:44:29 192.168.1.1 DrayTek: WAN 1 is down.

Nov 17 02:25:25 192.168.1.1 DrayTek: WAN 1 is down.

Nov 17 02:42:18 192.168.1.1 DrayTek: WAN 1 is down.

Nov 17 17:03:09 192.168.1.1 DrayTek: WAN 1 is down.

Obviously it's time to talk to support again and see if I can get O.R. to follow up with the REIN issue they identified and 'went away with and then closed a day later'.

My guess is that its tied into someones heating turning on as the colder days see more issues...

I know it is not ours (as we have only just got it working again since the 'smart meter' install - never use So.Energy) and the new UPS has not helped so I'm thinking line pick up rather than power...

Would not be so bad but my days are nights so I get caught often...

Anyone have any tips on handling REIN with support and O.R.?

[#2]

Hi there, I've just been ,taking a look into this for you and our testing isn't showing signs of any obvious causes for the issue but it would definitely be worth investigating. We'd need to arrange an engineer visit as the next step so please let us know your availability in these slot times:

AM = 08:00 - 13:00

PM = 13:00 - 18:00

[#3]

Hi @adbeasley 

sorry to see you're back! My concern is the Openreach engineer might have used REIN and SHINE as a get out of jail card, did he/she provide any evidence to support this prognosis?

What modem and router are you currently using? I see you've used many over the years

Do you have a NTE5c master socket with a MK4 faceplate? The MK4 faceplate has a REIN filter built in.

The master socket 5c and it's associated faceplates (with built in DSL filter or without filter) do suffer from poor connections, between the faceplate and hidden test socket with in. I suggest reseating the faceplate.

Regardless of the type of master socket you have if possible I recommend connecting your modem directly to the test socket and see if this improves the connection.

It is still worth openreach visiting your property.

HTH

[#4]

Sorry the family has had bad news today so not able to ring in to organise visits and only just catching a breath.

Faceplate seems solid and is well out of the way of being caught by feet / shelf etc.

I will try the test socket when I can (guess things will settle in a day or so).

You are correct I have had a fair few routers in the five years we have been here... From memory the bare minimum has been:

3 EE routers and about 5 or 6 firmware updates though these where the same model

2 PlusNet routers with at least 1 firmware update again these where the same model (maybe the same as the EE one just different software)

1 TP-Link router with 1 firmware update but I've had to retire this as it did not support vlans for the IoT bits I want to put in next year.

Currently running a Draytek 2766 with the latest firmware straight off the new master socket installed by OR when they did the move for me a few months ago. The Draytek is configured out of the box with:

Only odd bit on the setup is the MTU size - that's smaller than the TP-Link from memory.

As for how the engineer decided on the fault cause - it was the area specialist who came and had access to his internal tracking system that showed more detail than the normal testing and it was reporting a major issue with one house in the area and a big issue with mine. He would not say where the other house was and the screen for my account did not have name / location details so I had to take it that it was my account!

As he left I asked what the plan was and he said he would investigate but a day or so later I checked and the call was closed. The error did clear up for a while but it was a lot warmer than it is now.

Anyway I am going to have to hold off things for a day or two while real world bits are settled...

[#5]

Sorry to hear that you've had some bad news today, give me a shout on here any time when you're ready to go ahead and I'll pick this back up for you. 

[#6]

Surely the IP should be set to dynamic, even if PN have given you a fixed IP?

[#7]

I made time today to call the helpline and get the problem logged.

They kindly offered me a new router (Hub 2) but TBH I declined 🙂

The current drop list is a pain to pull as the logs are getting archived / deleted but my best guess is attached. I have tracked down why the log files get very very full - I have 119K+ lines of 'Dialing ISP' so need to filter them out on-going.

Two odd entries (Dated Jan 1) are due to a config change (trying without the static IP settings) and a reboot today - neither helped.

The MTU setting looks OK - the router has an option to test the max MTU (like traceroute testing I assuming) and drops the packet size a max of three times. Trying this against different external IP addresses (e.g 8.8.8.8) consistently reports the same value so that's also fine.

As the idea above - I have removed the front panel of the master socket and now using the internal socket via an ADSL filter (was really surprised I could find it given the chaos going on here) and this looks like a fix as we have not had a drop in the last two hours... Have to say I'm not impressed by the build of the socket - the filter feels very very loose and does not clip in.

It is possible the noise generator was turned off around the same time but I have till Thursday morning to monitor the line again as that's the visit time by OR (I have things I cannot change this week).

Fingers crossed its going to be a short visit and I have a spare front panel at the end of it...

[#8]

Well it lasted till just now (02:15)

Better than yesterday though...

[#9]

@adbeasley 

Being between 00:01 and 05:59 this drop could be caused by the DLM adjusting your line speed, so I'd just monitor it for now.

[#10]

It's definitely trying something - looking at the router log I get a regular entry for speed (about every 26 seconds) that reads

ADSL_Status:[Mode=17A States=SHOWTIME UpSpeed=9592000 DownSpeed=28681000 SNR=16 Atten=22 ]

Graphing these gives:

Before that we where very stable from what I could tell (around 38Mb/s) but did not collect the stats (as I thought things settled down) - fortunately (I think that's the word).

Folk are getting used to me dropping during video overnight now - if I can make a full one without a drop it even gets a laugh and round of applause (blush).

Need to look at the logging options to keep an eye on usage head room - I've found a graph that shows outgoing bandwidth interactively:

but not found out how to get the raw data without a Netflow server. There is an undocumented web hook but a small test shows it only gives device name, mac address and a very basic UP or DOWN status.

To quote Bamber Gascoigne - your starter for ten is 'When was Andrew working yesterday?' and the bonus question - 'why?' 🙂

[#11]

The line speed and bandwidth utilisation are not overly helpful here.  The variable line speed is indicative of multiple drops in the presence (or absence) of noise, as indicated by the 16dB SNR figure.  That probably infers that the line is under heavy DLM management.

Take a look at router stats and see if it will work with one of your many routers.  Right now I suggest that you need a router which allows you to monitor / plot the dynamic SNRM (rather than the best performer / configurable router) to determine if there is REIN / SHINE, what it looks like and when it happens.  Time and shape of the interference can help identify the source.

One activity I investigated resulted in me being able to monitor nearby passing trains ... and which direction they were travelling in.  I recommend a 10 second sampling interval with 720 plots per graph (2 hourds).  Note that a Plusnet Hub One is a BTHH5a for which there is a custom version of Router Stats which works exceptionally well...

[#12]

Thought I was doing well since I pulled the faceplate - a quick check today only shows :

Nov 28 02:13:07 192.168.1.1 DrayTek: WAN 1 is down.
Nov 30 04:29:39 192.168.1.1 DrayTek: WAN 1 is down.

Interestingly, these are about the times when the car chargers normally switch on / off according to my electrician but I do not know of one locally.

@Townman 

The router stats package is X86 and Windows only - I struggled previously to get it running stably on WINE and failed under ARM - I do not have a 'spare' x86 box I can use here (development and my own Macs are all ARM) let alone a Windows licence  - maybe I can get something working on my Synology 920+ NAS!

I did find this site that says RouterStats can 'tweak' the SNR but I do not know if that's disabled on the Draytek or if it would make the matters worse.

I know the router can report both of these (it is on the list of supported routers) and it shows up in the diagnostics screen:

so I shall see if I can get it logged. SNMP / Telnet guides are none existent in the manual!

The syslog text 'SNR=17 Atten=21' is from the downstream near end SNR Margin while the upstream / far end does not show even on the inbuilt viewer. Running a graph off on the near end gives:

The two big and one smaller spikes down are at time of solid drops.

So current plan:

1) OR visiting tomorrow

2) See if I can scrounge up an x86 box and Windows licence

3) See if I can cobble up a VM on my NAS and Windows licence

4) See if I can get the data via SNMP / Telnet script - I'll log a call with Synology but I honestly cannot see any more logging data options without spending big bucks on Netflow etc. Hmm - Zabbix possibly???

Maybe I need to 'justify' a 2018 Intel Mac Mini and run Bootcamp 🙂

Anyone know what the minimum RAM / Disk needs are for RouterStats and if it runs under Windows 10 or 11? Thinking small NUC box if I cannot get anything sorted so I can tuck it away out of site of the good lady 😱

[#13]

Well a little digging about shows two ways:

1) SSH / Telnet (see here for details) gives me this under ads / vdsl:

  ---------------------- ATU-R Info (hw: annex A, f/w: annex A/B/C) -----------
   Running Mode            :      17A       State                : SHOWTIME
   DS Actual Rate          : 28681000 bps   US Actual Rate       :  9888000 bps
   DS Attainable Rate      : 57531650 bps   US Attainable Rate   :  9897000 bps
   DS Path Mode            :        Fast    US Path Mode         :        Fast 
   DS Interleave Depth     :        4       US Interleave Depth  :        1 
   NE Current Attenuation  :       21 dB    Cur SNR Margin       :       17  dB
   DS actual PSD           :    11. 3 dB    US actual PSD        :     7. 4  dB
   NE CRC Count            :       12       FE CRC Count         :       25
   NE ES Count             :        0       FE  ES Count         :        0
   Xdsl Reset Times        :        0       Xdsl Link  Times     :        6
   ITU Version[0]          : 00000000       ITU Version[1]       : 00000000

   VDSL Firmware Version   : 12-3-2-3-0-5   [with Vectoring support] 
   Power Management Mode   : DSL_G997_PMS_L0 
   Test Mode               : DISABLE 
   nBandPlan               :        0 

  -------------------------------- ATU-C Info ---------------------------------

   Far Current Attenuation :        0 dB    Far SNR Margin       :        6  dB
   CO ITU Version[0]       : b5004244       CO ITU Version[1]    : 434da4a1
   DSLAM CHIPSET VENDOR    : < BDCM >

2) SNMP has adslAtucCurrSnrMgn and adslAturCurrSnrMgn (nothing under VDSL) but I cannot enable SNMP without a router reboot so I'll do that when OR is changing the faceplate and has the line down...

Guess I could pull this SNMP data via Node-Red and dump it to a log file rather than messing in Python etc. as my mind is yuk - hate switching from nights to days - wish OR worked nights (then again family would be upset 🙄).

[#14]

Router stats runs find on Win XP / 7 / 8 / 10 / 11.

Resource usage is not excessive.

That said I think you already have enough evidence to suggest that you are blighted with REIN / SHINE.  The question is can you identify the source of the noise at the point of disconnection / when the SNRM rises.

Personally, I would the link when the SNRM is at its highest / around the middle of the day and then see what happens.  You need to look at the SNRM plot as an inverse - when the figure is high, there is less noise, when it is lower there is more noise, so look at what makes it go down (noise) and what makes it go up (the abatement of noise).

Try to identify day to day time correlations.

And yes, fiddling with the SNRM is only going to complicate matters!

[#15]

@Townman Thanks for that - I have managed to get the SNR stats out of the router using Node-Red in a read only mode (so no fiddling for me) this morning.

Turns out the Draytek document is a little off:

1.3.6.1.2.1.10.94.1.1.3.1.4.4 is the near end SNR and not 1.3.6.1.2.1.10.94.1.1.3.1.4

That had me chasing my tail for 30 minutes or so till I looked at the output from the command line walk...

So a very rough graph is:

Need to dump to a database as a restart of the machine will loose the data (and the computer is not even on the UPS 😮) etc but its a starting point.

---

Anyway OR will be here in ten mins or so as they have just called.