Looks like my problem is back with a vengeance - REIN / SHINE

[#1]

Hi @adbeasley, I've just picked this up for you so I'll give it an hour or two for the engineer to finish their work and I'll get in touch to follow things up with you. 

[#2]

Nothing common about the drop days or times (Time vertical / DOW Mon-Sun horizontal):

OR Engineer has just been. Turns out I'm not the only one here in the village with problems - he has moved us off the card (DSLAM?) rather than a port move and the impact on the SNR is visible.

I also have a new front to the master socket though he thinks the ADSL filters are more reliable some days 🙂

Adam - if you see this please call any time after 14:00 - just need to settle my mind a bit...

[#3]

Cheers for taking my call just now, nice chatting with you! I'm glad they replaced the socket and requested a card change for you. 

Thinks are looking more stable so far but it's obviously too soon to make a call on things so let's monitor it for a few days and I'll check in with you again on Monday. 

Do shout up if there's anything in the meantime though. 

<img src="http://visualradius.plus.net/visualradius/generated/image16699086059770.png"/>

[#4]

Well if he did move us on to another card the fix did not work:

Dec  2 18:00:43 192.168.1.1 DrayTek: WAN 1 is down.
Dec  3 02:24:50 192.168.1.1 DrayTek: WAN 1 is down.

I did manage to get a SNR graph at the time it dropped and it does not show any significant change on the SNR before the drop:

but it seems to be changing between 12 dB and 13dB at my end before it dropped.

I may have sourced a Win 10 machine - need to sort a disk / network port for it but I understand the licence is valid and the machine works 🙂 Wonder where my PC keyboard is - if anyone sees it around can you tell me where...

[#5]

Well that change did not help:

Dec  2 18:00:43 192.168.1.1 DrayTek: WAN 1 is down.
Dec  3 02:24:50 192.168.1.1 DrayTek: WAN 1 is down.
Dec  3 03:00:44 192.168.1.1 DrayTek: WAN 1 is down.
Dec  3 16:00:27 192.168.1.1 DrayTek: WAN 1 is down.
Dec  4 02:00:19 192.168.1.1 DrayTek: WAN 1 is down.
Dec  4 18:18:57 192.168.1.1 DrayTek: WAN 1 is down.
Dec  5 00:14:39 192.168.1.1 DrayTek: WAN 1 is down.
Dec  5 03:37:26 192.168.1.1 DrayTek: WAN 1 is down.

No significant changes in SNR reported by the router with 10 second samples:

 Any ideas?

I know if OR comes again they will just drink my tea, stick their line tester on - find nothing, grumble at another visit and go...

[#6]

Hi @adbeasley, sorry to see the connection is still dropping. It's less frequent from what I can see which could be a step in the right direction:

<img src="http://visualradius.plus.net/visualradius/generated/image16702414717257.png"/>

I'll get in touch a bit later today to follow things up with you.

[#7]

Well this morning was painful - just catching up with MasterChief and they just announced the first leaver and then this happened:

Not seen it do that since I got the graphing running - wonder if this was the exchange as the SNR went up but the speed dropped!

Anyway need to see if Nikita gets through...

[#8]

Are you also doing QLTs?

[#9]

Good speaking with you again! as promised I'll check in with you after Thursdays engineer visit and as mentioned that should allow us to escalate things with Openreach if we're no further forward.

[#10]

Snoozing to follow up after OP's engineer visit.

[#11]

Thursday:

Well OR have said that it is REIN but it may take two weeks to action the internal call.

Friday:

With no notice OR turn up (I'm 30+ miles away) and wonder around the house with a meter and confirm that we are not generating anything to cause it (as if I do not know what we have in the house and when it's on!).

The visit seems to have made it worse (not having anything on the 9th):

Dec 10 03:07:49 192.168.1.1 DrayTek: WAN 1 is down.
Dec 10 03:09:34 192.168.1.1 DrayTek: WAN 1 is up.
Dec 10 12:34:16 192.168.1.1 DrayTek: WAN 1 is down.
Dec 10 12:36:52 192.168.1.1 DrayTek: WAN 1 is up.
Dec 10 12:38:22 192.168.1.1 DrayTek: WAN 1 is down.
Dec 10 12:48:30 192.168.1.1 DrayTek: WAN 1 is up.
Dec 10 13:22:13 192.168.1.1 DrayTek: WAN 1 is down.
Dec 10 13:24:39 192.168.1.1 DrayTek: WAN 1 is up.
Dec 10 13:27:35 192.168.1.1 DrayTek: WAN 1 is down.
Dec 10 13:29:50 192.168.1.1 DrayTek: WAN 1 is up.
Dec 10 13:31:14 192.168.1.1 DrayTek: WAN 1 is down.
Dec 10 13:38:10 192.168.1.1 DrayTek: WAN 1 is up.
Dec 10 14:40:53 192.168.1.1 DrayTek: WAN 1 is down.
Dec 10 14:43:38 192.168.1.1 DrayTek: WAN 1 is up.
Dec 11 00:32:47 192.168.1.1 DrayTek: WAN 1 is down.
Jan  1 00:06:01 192.168.1.1 DrayTek: WAN 1 is up.  <== Infers reboot or NTP error
Dec 11 02:57:48 192.168.1.1 DrayTek: WAN 1 is down.
Dec 11 02:59:32 192.168.1.1 DrayTek: WAN 1 is up.
Dec 11 20:22:36 192.168.1.1 DrayTek: WAN 1 is down.
Dec 11 20:30:17 192.168.1.1 DrayTek: WAN 1 is up.

Downtimes are getting longer - we are seeing 6+ minutes now compared to the normal 2+

Talking with the engineer on Thursday, he said to keep an eye on the CRC errors (they where 18000+ at the time we looked - yes over 18 thousand) but I was not sure if the number resets at connection or at reboot.

I need to look at tracking this as it does not get reported in the syslog that I can see and after the drop just now they sit at:

CRC

5 Downstream

80 Upstream

Is there any guide as to what is 'normal' or 'acceptable'?

[#12]

This is a step in the right direction.  BTOR accept this is a RIEN issue - you might not understand just how much of a leap this is.  The BTOR observation needs to be read as being very positive for you - they have confirmed the presence of REIN (so often they are not there when it happens) AND that it does not emanate from your domain.  That is double good news.

Can you please keep the SNRM plots running 24x7 and watch for patterns - if you are lucky you will see repeating shapes, possibly at the same (or similar) time of day.  I once managed to correlate my REIN with the train timetable.

[#13]

Due to the amount of data I'm collecting the SNR stats are just about impossible to graph - Excel is giving some very very weird results I may end up dropping them to file and ignoring the time stamp - that is really throwing the graphing into a mess.

There does not seem to be any correlation to day / time of the drops going by the graph from a couple of weeks ago and the trains around here are over a mile away and very regular 😞

I cannot find a SNMP way of monitoring the CRC values but they are increasing:

CRC

62 Downstream

507 Upstream

The Draytek site has gone to pot - you either get a grey block or a 404 now - support call here we come...

Problem is my mind is beginning to get muddy (sign of anxiety attacks / depression on-set) over this so I do not know how much more I can do at the mo. as I have got zero way with the Windows machine and things are beginning to build up.

[#14]

When you say that the Excel data is impossible to graph, do you mean that there is too much data to view sensibly?

I might have a helpful solution.  I have a spreadsheet somewhere with some magic sliders (VBA macros) on a graph which allows you to zoom in / along a time line.  If I can find it, I will give you a link to it, so that you can pinch the logic.

[#15]

---

@Townman wrote:

When you say that the Excel data is impossible to graph, do you mean that there is too much data to view sensibly?

I might have a helpful solution.  I have a spreadsheet somewhere with some magic sliders (VBA macros) on a graph which allows you to zoom in / along a time line.  If I can find it, I will give you a link to it, so that you can pinch the logic.

---

That's most kind and would be helpful if you can find it. I have been thinking about checking the data on a rolling average to help identify massive changes in SNR but I've tweaked the node-red to sample every minute and output 12 hrs and that helps a bit with the visualisation:

but it does require me to capture the screen shot each day at the minute so analysis of the drop pattern is going to be based on the text system log rather than the SNR records. The one minute sample window is the max I would go for as drops last around 2 minutes on a good day (not the right term but you know what I mean).

Draytek have asked me to upgrade the router software to their new RC1 codebase - not sure if it is for the SNMP or to help with the drops or just a standard reply before the code is pushed for general availability...

I did get the head straight enough to look at the telnet / shh access - you can only do it via the main admin user and it does not support SSH keys so I would have to hardcode the router admin password into the script! Putting that onto the back burner this week.

I did find that the Disney app is hopeless at recovery - the 4AM drop this morning was while I was watch a film and it just sat there in a 'retry circle' despite the connection being made again and the internet up and running!