Seems fixed - Ive kicked off my large download that was failing
Transcript of BT fault report.
Received Issue Update
15/09/2015 13:37 FINAL Service fully restored at 11:32. Following the replacement of theOpenreach Network facing card, services restored for a short time, before the Openreach switch crashed. Remote and on site actions were able to bring the switch back up, and the new Network facing card was replaced with the original card at 11:30, ensuring full customer connectivity. Monitoring and testing have taken place for 90 minutes, by operations, Technical services, Openreach and vendors. All are in agreement that full service is restored, and working as expected. Customers have also confirmed that full service has restored. Enhanced monitoring of the service will be undertaken by operations, and of the switch by Openreach. As a precautionary measure, further hardware will be ordered to site by openreach, should there by any subsequent problems identified. BT regrets any inconvenience this may have caused.
15/09/2015 11:57 12 Openreach have replaced the Network facing card on the layer 2 switch in Tranent which was seen to initially increase traffic levels significantly across all ports, indicating that the issue had resolved. Shortly afterwards, the layer 2 switch was seen to crash, resulting in a loss of service to all customers. Following a number of switch card reseats, and a card replacement, all service was restored at 11:30. Monitoring is under way to ensure that all errors are now clear. Next update will be issued upon service restoration unless there should be a significant development.
15/09/2015 09:50 11 The technical bridge has reconvened with all relevant parties, including Operations, Technical Services, Openreach and vendor all represented. An engineer is on site, standing by to assist where required. The controller card at the Openreach Head end has been switched, but initial indications are that the issues seen are persisting. Next update will be issued upon service restoration unless there should be a significant development.
15/09/2015 04:01 10 The CPM card changed to no avail. A reload was completed on this network element at 02:00 and the tech files of before and after have been passed to the Vendor. They have identified no issues with the network element. Checks have also been performed on the transmission paths to Falkirk and Edinburgh and no issues have been identified. Traffic levels have died down so investigations are being hampered. The technical call will reconvene at 09:00 to investigate further. Root cause and estimated restoration time unknown at present. Next update will be issued upon service restoration unless there should be a significant development.
14/09/2015 22:55 9 Traffic has been switched between the two CPM cards and tech log files have been taken to assist in investigation. The core network has been discounted from investigations. We are now in process of changing the CPM card in the A slot and eliminating the hardware. Vendor is also engaging their 3rd line to assist with investigation. Root cause and estimated restoration time unknown at present. Next update will be issued upon service restoration unless there should be a significant development.
14/09/2015 21:40 8 The bridge reconvened at 21:00 with all parties attending now including IPTV. Interrogations are taking place with the 7750 cards at Tranent, the TSO engineer is onsite and has located the spare card. The TSO engineer is co-oping with Technical Services and is currently removing Central Processing Module (CPM). Investigations are continuing. Next update will be issued upon service restoration unless there should be a significant development.
14/09/2015 19:52 7 This is running as a Parent case P1 incident IMT43198/15. Headline changed from Glasgow to Tranent. Spares (both of the same type and may not be required) are now on route with ETA?s of 21:15 & 23:15 to Tranent. A TSO engineer has also been task to site with an ETA 20:50 to co-op with the ADSL operations team and receive the spare. The bridge will reconvene at 21:00 with all parties lined up attend. Next update will be issued upon service restoration unless there should be a significant development.
14/09/2015 18:02 6 This is running as a Parent case P1 incident IMT43198/15. The TSO engineer at Falkirk has cleaned fibres and re-seated cards after which the traffic was routed back to the original route. Traffic can be seen flowing but the customer is still seeing issues, 3rd line support are still monitoring the quality of the service as the customer can download files but the data is showing as corrupt. As a parallel action the operations team are ordering spare cards for the end of the routes (Edinburgh and Tranent) TSO engineers will be tasked once ordered. The vendor is now on the bridge providing support. Next update will be issued upon service restoration unless there should be a significant development.
14/09/2015 15:35 5 This is running as a Parent case P1 incident IMT43198/15. Having identified a number of port errors, the Technical Service team have costed out a link from a Falkirk access router into the 21c Core. Operations still are working to establish whether this has had any positive impact on affected customers. As a paralell action, a TSO engineer is being tasked to Falkirk as a precaution with a ETA 15:40. Next update will be issued upon service restoration or sooner should there be a change in impact.
14/09/2015 14:10 4 Having identified a number of port errors, the Technical Service team have costed out a link from a Falkirk access router into the 21c Core. Operations are working to establish whether this has had any positive impact on affected customers. As a paralell action, a TSO engineer is being tasked to Falkirk as a precaution. Next update will be issued upon service restoration or sooner should there be a change in impact.
14/09/2015 13:40 3 Investigations are ongoing to localise the fault. Technical Support are working with the operations teams, continuing with their investigations.
14/09/2015 12:09 2 Customer impact: Degraded internet services to {1632} Broadband Customers Current Situation: Completing initial diagnostics. Root cause and estimated time of repair are not known at present. Next update will be issued upon service restoration unless there should be a significant development.
14/09/2015 12:07 1 Initial diagnostics taking place
