Millions of CRC Errors on TG582N Router

[#1]

Has anyone else seen this?
Ever since I received this router a couple of months ago, I have seen situations where I look at the router stats in the morning and there have been literally millions of CRC errors overnight.
If I reboot the router, the stats clear and the number of CRCs per hour goes back down to around 40, i.e. around one CRC per 90 seconds.
This seems to happen nearly every day.
I have been trying to identify when this starts and thereby identify the cause, but it seems random.
As part of this investigation, I have gone back to a Thompson TG585 router which is basically the same as the Technicolor TG582.
I left the old TG585 running for several days and got an average of one CRC per minute overnight and one CRC per 90 seconds during the day.
As soon as I leave the new TG582 running overnight it gets stuck generating around 50 to 60 CRC errors per second (To confirm, around 3500 per minute!).
As of this morning when I looked at the router stats it was still generating 3500 CRC errors per minute.
I rebooted the router and the CRC count immediately went back down to one every 90 seconds.
So this new TG582 goes from generating 210,000 CRCs per minute to 0.67 CRCs per minute just by rebooting.
As of 11:15, I have just checked the stats after rebooting and I am currently getting 0.50 CRCs per minute or 1 CRC per 120 seconds.
This is hurting my line speed because the line now syncs on an SNRM of 6, whereas for literally months it was on an SNRM of 3. As a consequence, my sync speed is over half a meg. less than it was previously.
Just to re-iterate, I do not get any problems of this nature on the Thompson TG585 router. I would be happy to keep using the TG585, but it doesn't support wireless-N.
It seems to me like a duff router, but I was wondering if anyone else has seen similar problems, even with a dirrerent router?

[#2]

Hi,
I have seen similar... which TG582 have you got - with / without USB and firmware version no would be useful.  One TG585 I had got so busy dealing with FEC / CRC (not an issue to other routers) that it could not respond to web admin page requests.  PN replaced that with a TG582n.  One wonders if there is a router / line compatibility issue here.
Monitoring with router stats would be useful, if only to see what the over night SNRM / noise / tone / bit swap looks like.

[#3]

Over a period of 5hrs 20mins, it has averaged one CRC every 96.2 seconds, which is pretty good.
The old TG585 would allow bit swaps data to be produced, but the newer TG582N is so locked down, only basic info. can be extracted.
The firmware is 10.2.2.B.
It hasn't got a USB port.
Strangely, the older TG585 doesn't produce CRC errors, but the newer TG582N does. They are both broadcom.
I'll investigate routerstats.
This is a snapshot of some of the data I have at present :-
Up time (Days hh:mm:ss): 0 days, 5:20:40
xDSL Standard: ITU-T G.992.5
xDSL Annex: Annex A
Channel Mode: Interleaved
Bearers generic info DS        US
Payload rate [Kbps]: 4939 444
Attenuation [dB]: 52.5          32.4
Margins [dB]: 5.9            25.0
G.997.1 Statistics (last 24 hours):
Line failures Near end
Loss of signal (LOS): 0
Loss of frame (LOF): 0
Loss of power (LPR): 0
Performance monitoring:
Line PM: Near end
Error second (ES): 1242
Code Violation (CRC): 200      14
HEC violation count (HEC): 3108 N/A
The CRC errors don't seem to be occurring yet. I'll keep an eye out for when they start, but either way I'll report the CRC situation tomorrow morning.

[#4]

I've installed and configured Routerstats and it is now running.
Current situation is 282 CRCs in over 7 hours - Averaging at one CRC per 93.2 secs.

[#5]

Quote from: rnewfie

The old TG585 would allow bit swaps data to be produced, but the newer TG582N is so locked down, only basic info. can be extracted. I'll investigate routerstats.

Hi I have the TG682n and router stats - producing bit swap graphs is not an issue.
One thing to look out for though; RS (on Win 7) can get its nickers twisted.  SNRM and synch graphs show zero (just as though there is no synch) when there is.  If you look at the summary tab, you will see the bit per tone data.  RS has lsot internal messages and it looking at the wrong data at the wrong time.  Stop and start the logging (stop and play buttons on the menu bar) to rectify the problem.  I have flagged this error to the owner of RS, but to date, there has been no fixed developed.

[#6]

Typical - When you monitor something, you don't get the problem.
I ran Routerstats for around 17 hours from 17:45 to 10:45.
The CRC count was 858, which works out to be one CRC every 100 seconds and considering that was through the evening and overnight, that's pretty good and normal for this line.
I'll restart Routerstats and continue to monitor.
I have attached the router log from a few minutes ago. There's no point in attaching the other graphs as they are pretty clean.

[#7]

This is getting to be a real pile of rubbish now.
Have reported this to Plusnet and the estimated response time is 44 hours.
The router has re-booted itself now twice today and the connection speed has dropped by 25%.
Strangely though - virtually no CRC errors reported.
It is now down below my already pathetic FTR of 4096.

[#8]

There was an unexpected re-boot of the router at 11:33 today, which I only noticed a couple of hours later.
The router seemed to get itself back to normal, inasmuch as the SNRM went back down to 3.3 which is usual on this very clean line.
The sync speed also went from 3909 to 5410, again around normal.
I started Routerstats at around 13:00 where there were 40 CRCs recorded between 11:33 and 13:00 (one CRC per 130 seconds).
By about 15:15 there were 115 CRCs recorded (one CRC per 116 seconds).
Then at around 15:15 for no apparent reason, there were approx 3500 CRCs being recorded per minute, i.e. around 60 per second.
As of 00:48, the router was still recording around 3500 CRCs per minute and I stopped it at just over 2 million CRC errors.
I have attached the CRC graph, the router stats and the bitswaps graph.
Strangely, the router seemed fine in operation with no noticeable delays, so is it possible that these reported CRC errors weren't actually occurring, but the router was counting something else?

[#9]

I did a speed test just to see if the router was being affected by the millions of CRC errors and it seems that it isn't. I used Thinkbroadband and the results came back as normal, i.e. just over 4 meg actual download speed.
About 10 mins ago, (after the speed test), the router re-booted after recording over 4,400,000 CRC errors in less than 24 hours (that's an average of 50 per second!).
The SNRM has increased from 3.3 to 6.0 and sync speed has correspondingly dropped around 1/2 meg to 4900.

[#10]

By way of an update to this, Plusnet have supplied a replacement router, which does exactly the same thing, i.e. it averages around one CRC per 90 seconds, then all of a sudden produces around 3,500 per minute. This continues for many hours until the router is re-booted, when the errors go back down to one every 90 seconds.
Thinking that this must be a line problem, I have re-installed an old TG585 router, which runs for days averaging one CRC every 2 minutes.

[#11]

Hi,
The bit swap graph is horrendous!  The CRC errors look poor too.  Change the graph plot to be "change per interval" - this will more clearly show where / when the error bursts are happening.  The sudden step change looks like something being switched on.
Need to keep graphing and I think raise a fault report at failures.plus.net
Note the long term CRC error rate is not overly useful here.  What is more relevant is the error rate when the errors are occurring.  Changing the graph as suggested will point to the problem periods.

[#12]

The graph above (CRCs.jpg on 30/11/2013, 01:10) shows that CRCs were occurring at around 0.56 per minute prior to the start of the burst, calculated as 118 at around 15:18, minus 40 at around 13:00, i.e. 78, divided by the time difference (138 minutes). This calculates to around one CRC every 1.77 minutes. At around 15:18, the CRC level shot up to 3,500 per minute and continued overnight until the router was re-booted. If it were noise causing this, then it would not last for 18 hours at a consistent rate of 3,500 per minute.
Change the graph plot to be "change per interval" - I can't see where that is set on RouterStats. Please advise.

[#13]

Quote from: rnewfie

By way of an update to this, Plusnet have supplied a replacement router, which does exactly the same thing, i.e. it averages around one CRC per 90 seconds, then all of a sudden produces around 3,500 per minute. This continues for many hours until the router is re-booted, when the errors go back down to one every 90 seconds. Thinking that this must be a line problem, I have re-installed an old TG585 router, which runs for days averaging one CRC every 2 minutes.

After extensive testing using 5 different routers, I have now established that the cause of these excessive CRC errors (exactly 60 per second), is running in ADSL2+ mode.
I have connected a D-Link 2640B, a TG585v6, a TG585v8 and 2 different TG582N routers.
The only time I get literally millions of CRCs generated is when the TG582N routers are run in ADSL2+ mode.
None of the other routers have problems in ADSL2+ mode.
Neither of the 2 TG582N routers have a problem in ADSL2 mode.
There is clearly a firmware problem with these TG582N routers in ADSL2+ mode on this line.

[#14]

Has anyone any thoughts on this?
I have reported my findings to Plusnet who have replied by stating that they are going to issue an SNR reset because my SNRM must be too low, which is causing the CRC errrors.
I cannot see how this can be the case when the 3 non-TG582N routers will happily run on an SNRM of 3.0 for literally years (The DSL2640B had been running on an SNRM of 3.0 since January 2012 with a sync speed of around 5200 in ADSL2+ mode).
The TG582N routers will only sync at a maximum of around 4900, but that is because they only connect on an SNRM of 6.0, which never drops to 3.0 as the other routers do.
If I run either of my Plusnet TG582N routers in ADSL2+ mode, they generate EXACTLY 3600 CRC errors per minute!
[Moderator's note by Dick (Strat):  Full quote of preceding post removed, as per Forum Rule]