Why is a "degraded" line not a faulty line?

[#1]

Radio 4 

[#2]

These are the current middle of the days statistics and graphs:

DOWNSTREAM (Rx)
Noise Margin:     6.2   dB
Connection Rate:  4967  Kbps
Line Attenuation: 52.5  dB
Power:            0.0   dBm
Max Rate:         5756  Kbps
 
SuperFrames:      4787565
SF (CRC) Errors:  409
Reed Solomon:     311191762
RS Corrected:     113882
RS Un-Corrected:  5943
HEC:              364
Errored Seconds:  254
Severe ES:        20
Interleave Depth: 32
Bitswaps:         13690

UPSTREAM (Tx)
Noise Margin:     6.1   dB
Connection Rate:  851   Kbps
Line Attenuation: 32.8  dB
Power:            13.0  dBm
Max Rate:         864   Kbps
 
SuperFrames:      589888
SF (CRC) Errors:  3
Reed Solomon:     684390
RS Corrected:     1342
RS Un-Corrected:  0
HEC:              6
Errored Seconds:  1
Severe ES:        0
Interleave Depth: 8
Bitswaps:         1387

TOTAL
Total Uptimes (From SF counts):
 WAN:   0 days, 22:37:31
 LAN:   0 days, 00:00:00
 
 
CRC:            1398            3
ES:             254             1
SES:            20              0
UAS:            1525            1525
LOS:            2               0
LOF:            17              0

[#3]

Hi John,
Well the SNRM graph is looking more like the sort of pattern I'd expect to see from night to day, if not a little more than one would have hoped.
The Bits/Tone is showing slightly more bits allocated in the daytime as one might expect, but surprisingly we're not seeing any of the ones around ~ 196-209 & ~ 214-220. You aren't using any extension leads from your new socket are you?

[#4]

In the photo below you should be able to see my phone cable is plugged into the 2nd ADSL filter and my modem cable is plugged into the 1st ADSL filter.
However I will shortly be adding an extension cable to get the phone to a more convenient part of my office.
(When I took the below photo, I happened to line the camera up exactly with the hole and you can see a chink of daylight. However I checked and the engineer did use clear silicon to fill the round the cable.)

[#5]

Well that all looks fine, I hope you've got some thermal insulation in that hole before the snow comes  ;D.
I'd put the phone extension lead between the two filters    and keep the phone as far as practical from the modem/router.

[#6]

Just trying to review all the info and apart from the comment about the increased attenuation resulting from the 'E' side swap, the 'D' side swap initially didn't appear to have much effect. However having reviewed everything I see there's one other bit of info mentioned in a PM about the engineers comments which he didn't put in the notes to Plusnet -

Quote

Something else he observed when changing the connections at the pole was that one of the wires of my old pair (to the cabinet) had a couple of spots of corrosion, which he said may not have caused the current situation and fixing that probably only prevent a future fault. He showed me the relevant bit of wire and I could see that about 10mm of insulation was missing from one side in two places, exposing the copper wires and there was green corrosion present at both points.

So whether it's the 'D' side pair with an issue, the connections at the top of your pole may well be worth a second look in case there is some corrosion on the pair you were put on.

[#7]

I took a look at my phone connection on Saturday. I wanted to try a different phone and also add an extension cable while monitoring the SNRM, just to give me something to easily demonstrate if a further engineer visit is arranged.
I got time to type this up this morning, but I also had a resync at 05:38 this morning and my connection is currently 5277kbps @ 4.8dB.
On Friday I had reset my connection and the sync speed rose from around 4750 kbps to around 4990 kbps, but it had dropped again around 1am to 4754kbps.
I tried various scenarios from nothing plugged in to either phone plugged in with or without a 2nd ADSL filter. I did not disturb the ADSL modem connection and did no more than handle the main ADSL filter. However the SNRM showed great variations. There were even jumps in the SNRM for a period while I was not connecting and disconnection phone equipment. My ADSL connection dropped 5 times during the exercise.
I took detailed notes about the various combinations of connections at each stage, but did not identify any relationship between the presence or absence of any item and a rise or fall in SNRM.
The last combination was to connect a 5m CAT5 extension cable and my original cordless phone without a 2nd ADSL filter, which was my preferred end setup. This seemed to result in a further resync and my connection ended up as below:
Downstream router stats at 17:08 - 175 minutes after connection up:

Noise Margin:     5.6   dB
Connection Rate:  4978  Kbps
Line Attenuation: 52.5  dB
Power:            0.0   dBm
Max Rate:         5696  Kbps
 
SuperFrames:      620455
SF (CRC) Errors:  31
Reed Solomon:     40329626
RS Corrected:     14141
RS Un-Corrected:  292
HEC:              25
Errored Seconds:  1150
Severe ES:        818
Interleave Depth: 32
Bitswaps:         1725

Below I've attached the Noise, sync speed and Bits/tome graphs covering the testing.

[#8]

Could you post the current Bits/Tone please. Is it still the same filter plugged into the NTE5a as was plugged in yesterday through all the tests?

[#9]

I can confirm that the same filter has been plugged into the NTE5a since the last engineer visit, through out the tests (the day before yesterday)  and is still plugged in.
I've attached the latest Bits/Tone graph and SNR graph for the last 24 hours.
Below is the current downstream router stats.

Noise Margin:     2.5   dB
Connection Rate:  5227  Kbps
Line Attenuation: 52.5  dB
Power:            0.0   dBm
Max Rate:         5968  Kbps
 
SuperFrames:      8736960
SF (CRC) Errors:  16791
Reed Solomon:     567902414
RS Corrected:     2112777
RS Un-Corrected:  123771
HEC:              13485
Errored Seconds:  2149
Severe ES:        1117
Interleave Depth: 32
Bitswaps:         15981

The increase in the above CRC error total was about 500 for the past 24 hours.
[Edit (first line read that same filter was in place at last engineer's visit).]

[#10]

I'll just try and summarise - current Bits/tone looking quite good, considering the Target SNRM at 3db, the increased allocation of bits is obvious, however the disappearance of the higher tones is again evident, maybe because it's after dark. The SNRM variation over the 24 hours is good. Note, these comments are in consideration of the length of your line. Now if the 'E' side attenuation can be brought back down, then I think you can see the potential of you line and it should give you similar performance to that before all your problems started some 18 months ago - IF - the stability can be maintained. That IF is a big one at the moment taking account of all the variation on Saturday.
Looking at the Bits/Tone plots on Saturday, the sync speeds achieved at the various times is totally consistent along with the SNRM when there was some stability, noting the then 6db Target SNRM.
What is of concern is some of the wild swings and irregular behaviour of the SNRM and what's bugging me in particular is the significant changes in the Bits/tone allocations. It's all consistent with some sort of interference - this could be something such as RF pickup resulting from a bad joint (top of the pole is a possibility for example bearing in mind the 2nd engineers remarks) or possibly the "REIN" type of interference that you seemed to be picking up last week, or some other interference.
At least after the last resync on Saturday, the performance looked a bit more like that which might be expected.
I think we really need some further tests, to eliminate a number of possibilities if nothing else, I'll drop you a PM with some further ideas.
Edit: Forgot to say that the increase in CRC errors of 500 over the 24hours is also good.

[#11]

An engineer visit had been arranged for 14/12/12 so I tried to do some testing before the visit.
However testing time was very limited, due to electrical work being done in my home (ie no electricity) and the lack of daylight hours.
The testing that Anotherone advised involved switching filters and phones and with the above limitations that testing was not very successful.
So, the engineer visit went ahead on 14/11/12. The aim was to fully test the line, including assessing the original pairs on the d & e sides, and to co-op with Plusnet. As the previous engineer did not know about contacting Plusnet, I printed the Plusnet engineer instructions and showed them to him. However I don't think he contacted Plusnet. However shortly after arrival, he received a call from the previous engineer advising what he had done. I also described the actions of the previous 3 visits.
The engineer ran tests at the master socket and, as per his predecessors, said that there were lots of errors. He left to carry out a comparison test from the cabinet. He returned about 90 minutes later to say that he had disconnected connections at the cabinet and tried to run tests, but had been unable to connect any of his test numbers.His manager confirmed that this was not an isolated problem and that broadband tests were not available at that time. I demonstrated the SNRM jump when making a call. It happened the first time I tried it, but not the second time. There may have been some improvement?  The engineer left to write up his notes. He commented that the next stage would probably be a new equipment order in the exchange and another engineer visit to carry on.

---

16/12/12 10:00 - 13:52
Secondary ADSL filters
I wanted to get back to doing the testing suggested by Anotherone, so in preparation I did some testing of the requirement for a second ADSL filter.
When I had added my CAT5 extension wiring last week, I had omitted the secondary ADSL filter. So, I ran a few comparisons by adding a secondary filter then switching between secondary filters (I found that I have 5 in stock, now all permanently marked with a number).
I saved Bits/Tone graphs before starting and after changing each filter and also waited between each change to see if there was any affect on noise margin.  There was no noticeable difference after changing each filter.
See below for one of the Bits/Tone graphs during the testing and the noise margin graph saved shortly after midnight on 17/12/12 showing the whole of 16/12/12.
ADSL2+
I had a little daylight time left, so I thought I would see if I could force my router to connect with ADSL2+ instead of ADSL2.
13:57 - Router Stats on ADSL2

Noise Margin:     5.6   dB
Connection Rate:  4943  Kbps
Line Attenuation: 52.5  dB
Power:            0.0   dBm
Max Rate:         5724  Kbps
 
SuperFrames:      10784553
SF (CRC) Errors:  7477
Reed Solomon:     0
RS Corrected:     0
RS Un-Corrected:  0
HEC:              5122
Errored Seconds:  9322
Severe ES:        43
Interleave Depth: 1
Bitswaps:         33130
Noise Margin:     6.3   dB
Connection Rate:  888   Kbps
Line Attenuation: 32.9  dB
Power:            13.0  dBm
Max Rate:         904   Kbps
 
SuperFrames:      384181
SF (CRC) Errors:  124
Reed Solomon:     0
RS Corrected:     0
RS Un-Corrected:  0
HEC:              78
Errored Seconds:  184
Severe ES:        0
Interleave Depth: 1
Bitswaps:         2440

It took two attempts of router settings to get the connection to switch to ADSL2+
14:03 - Router Stats on ADSL2+

Noise Margin:     6.3   dB
Connection Rate:  4975  Kbps
Line Attenuation: 55.5  dB
Power:            0.0   dBm
Max Rate:         5708  Kbps
 
SuperFrames:      928
SF (CRC) Errors:  0
Reed Solomon:     0
RS Corrected:     0
RS Un-Corrected:  0
HEC:              0
Errored Seconds:  9329
Severe ES:        43
Interleave Depth: 1
Bitswaps:         1

Noise Margin:     6.0   dB
Connection Rate:  923   Kbps
Line Attenuation: 32.9  dB
Power:            12.9  dBm
Max Rate:         924   Kbps
 
SuperFrames:      840
SF (CRC) Errors:  0
Reed Solomon:     0
RS Corrected:     0
RS Un-Corrected:  0
HEC:              0
Errored Seconds:  184
Severe ES:        0
Interleave Depth: 1
Bitswaps:         0
Total Uptimes (From SF counts):
 WAN:   0 days, 00:00:15
 LAN:   0 days, 00:00:00

I saved the Bits/Tone on ADSL2+ immediately on connection and then again at 15:34, as I'd noticed that the Bits/Tone seemed to change slightly (settle?) a few minutes after connection, as per graphs 3 & 4 below:
These were the Router Stats on ADSL2+, at 15:34:

Noise Margin:     5.8   dB
Connection Rate:  4975  Kbps
Line Attenuation: 55.5  dB
Power:            0.0   dBm
Max Rate:         5892  Kbps
 
SuperFrames:      337964
SF (CRC) Errors:  495
Reed Solomon:     0
RS Corrected:     0
RS Un-Corrected:  0
HEC:              211
Errored Seconds:  9522
Severe ES:        46
Interleave Depth: 1
Bitswaps:         996
Noise Margin:     6.2   dB
Connection Rate:  923   Kbps
Line Attenuation: 32.9  dB
Power:            12.9  dBm
Max Rate:         928   Kbps
 
SuperFrames:      305406
SF (CRC) Errors:  0
Reed Solomon:     0
RS Corrected:     0
RS Un-Corrected:  0
HEC:              0
Errored Seconds:  184
Severe ES:        0
Interleave Depth: 1
Bitswaps:         195
Total Uptimes (From SF counts):
 WAN:   0 days, 01:35:45
 LAN:   0 days, 00:00:00
 

I left things like this overnight, there was no drop out or other noticeable difference between ADSL2 and ADSL2+
However I wanted to more tests today, so changed the router settings to allow whichever standard the router negotiated on connection (it returned to ADSL2).

---

This morning I carried out some comparison tests by switching the primary ADSL filter and will post details next.

[#12]

This morning I switched ADSL filters and compared the results. I have 5 ADSL filters, but in the end only compared 3 of them.
No1 - BT ADSL MF50, Item Code: 011249
No2 - BT ADSL Filter, Irtem Code: 034738 (NA30513)
No3 - Netgear ACC-107-012
The conclusion of the testing was that I now do not trust filter No 3. 
Before starting I carried out a resync (to allow the return to reconnect at ADSL2 from ADSL2+).
The test sequence in each swap was:
1. Disconnect the PPP session
2. Wait 1 minute
3. Power down router
4. Wait 1 minute
5. Disconnect ADSL filter from Master socket
6. Power up router
7. Swap ADSL filter, by moving phone and modem cable from previous filter
8. Wait 10 minutes after step 5
9. Connect ADSL filter to Master socket
10. Collect Statistics
11. Phone off hook for 30 seconds
12. Collect Statistics
13. Phone on hook
14. Collect Statistics
I powered up the router immediately after disconnecting the filter from the master socket, as it simplified the number of steps required after reconnecting the ADSL filter to the master socket.
Filter No 1 in place.
10:13 - reconnect from ADSL2+ to ADSL2
10:17 - SNRM = 6.4 dB, Sync Speed = 4903 kbps
10:18 - 10:31 disconnect/reconnect sequence, including changing filter No 1 to filter no 2
10:32 - SNRM = 6.1 dB, Sync Speed = 4947 kbps
10:33 - Phone off hook - SNRM = 6.1 dB, Sync Speed = 4947 kbps
10:34 - Phone on hook - SNRM = 6.2 dB, Sync Speed = 4947 kbps
10:36 - 10:48 disconnect/reconnect sequence, including changing filter No 2 to filter no 3
10:49 - SNRM = 6.9 dB, Sync Speed = 4059 kbps
10:50 - Phnoe off hook - SNRM = 9.9 dB, Sync Speed = 4059 kbps
10:51 - Phone on hook - SNRM = 6.9 dB, Sync Speed = 4059 kbps
[I could see that this filter was obviously poorer than the other two filters, so I decided to switch back to filter No 1.]
10:53 - 11:05 disconnect/reconnect sequence, including changing filter No 3 to filter no 1
11:06 - SNRM = 6.0 dB, Sync Speed = 4879 kbps
The poorer performance of filter No 3 was also obvious in the Bits/Tone graphs below:
Note, each graph was saved at about the same time after reconnection as there seems to be a pattern where the higher frequency tones disappear after a few minutes. I also didn't bother uploading graphs showing the Bits/Tone while phone was off hook as they looked the same as when phone was on hook.

[#13]

Hi John,
Yes quite right not to trust Filter 3, I'd say it was either a bin job or secondary filter only, not a primary one.
There seems little to choose between Filters 1 & 2 from a Frequency response point of view, but I thought from a previous test Filter 1 had shown jumps in SNRM with the phone off/on hook. I note that test isn't mentioned in those above, but if that remains the case then Filter 2 would be a the better one.
As far as the ADSL2/2+ tests go, I'm not really surprised that you noticed little difference as no tones above 255 (I presume) are used. I'm not sure I would place too much emphasis on the "settling" between graphs 3 & 4 in reply #85 as it must have been getting pretty dark by 1534 and propagation would be increasing..

[#14]

The latest Openreach engineer reported to Plusnet that a fault has been found in the "Hand Over Distribution Frame". Plusnet are co-ordinating with BT to arrange a related Lift and Shift.
For a few minutes, shortly before 6pm today, there were some disconnections on my line.
These are the Telnet Stats from about an hour after the disconnections.

DOWNSTREAM (Rx)
Noise Margin: 6.3 dB
Connection Rate: 5056 Kbps
Line Attenuation: 52.0 dB
Power: 18.8 dBm
Max Rate: 5088 Kbps
SuperFrames: 206145
SF (CRC) Errors: 275
Reed Solomon: 0
RS Corrected: 0
RS Un-Corrected: 0
HEC: 202
Errored Seconds: 6993
Severe ES: 38
Interleave Depth: 1
Bitswaps: 1024
UPSTREAM (Tx)
Noise Margin: 7.0 dB
Connection Rate: 768 Kbps
Line Attenuation: 31.5 dB
Power: 12.9 dBm
Max Rate: 896 Kbps
SuperFrames: 206146
SF (CRC) Errors: 13
Reed Solomon: 0
RS Corrected: 0
RS Un-Corrected: 0
HEC: 4
Errored Seconds: 367
Severe ES: 0
Interleave Depth: 1
Bitswaps: 42

Also...

> adsl info --stats
adsl: ADSL driver and PHY status
Status: Showtime
Retrain Reason: 1
Max:    Upstream rate = 896 Kbps, Downstream rate = 5088 Kbps
Channel:        FAST, Upstream rate = 768 Kbps, Downstream rate = 5056 Kbps
Link Power State:       L0
Mode:                   G.DMT 
TPS-TC:                 ATM Mode
Trellis:                ON
Line Status:            No Defect
Training Status:        Showtime
                Down            Up
SNR (dB):        6.2             7.0
Attn(dB):        52.0            31.5
Pwr(dBm):        18.8            12.9

So it looks like I am on ADSL1 again.

[#15]

Hmm, any further update on the ticket yet? Attenuation hasn't improved that's for sure!