Tiscali Timeline

[#1]

LLU Disconnections

On Saturday Plusnet (with the help of contributors on our portal forums and Usenet) identified a common pattern  for the frequent disconnects being reported by some of our customers who's service is provided by Tiscali.   Problem number 34944 was raised at 2pm on Saturday 26/Aug/2006 and assigned priority one, Plusnet's highest classification. After eight days of investigation no definite cause has been found, but I'm happy to update you about what is known, the steps that have been taken and how its being progressed.

The problem

The symptoms of this problem are that customer are reguarly disconnected and that during the disconnection the 'sync' light on their broadband modem/router remains solidly lit. The effected customers find they can reconnect instantaneously or within minutes. If these customer's connect via the Tiscali test login (testing@dslconnect.co.uk)  they receive a stable connection although this login doesn't allow them to browse the Internet.

Situations where the sync light flickers, is not lit, where authentication fails consistently or where connecting to the test login fails to provide a stable connection are unrelated and should be raised by the normal fault procedures.

The fact the connections are was stable whilst connected to the Tiscali's test line proves the fault to lies not on the customers line but in Plusnet or its supplier's networks. Our observations suggest that up to one thousand customers are effected, just over five and a half percent of the approximately Eighteen thousand customers who's services we provide by Tiscali.  The disconnects occur between 4 and 40 times per day.  Customers are disconnected simultaneously but not all affected customers are disconnected each time.

Timeline

Saturday (26^th)

On the day the problem was reported the on duty network engineer (Adeyinka Makinde) was called out. He inspected the network monitoring in place and quickly ruled an equipment failure in our equipment or in Tiscali's equipment at the point they hand customer connections over to ourselves. Reviewing our internal 'users connected via Tiscali' graphs Adeyinka identified a common 'blip' the last of which corresponded to the last drop reported by customers. Tiscali's first line support were alerted at 15:03.

At 18:00 the problem was escalated to Mark Lovely, the main engineer responsible for our Broadband interconnects and (fortuitously) second line network support that weekend. Mark began investigating the problem using the diagnostic facilities of our ERX devices, which are the first item of Plusnet's equipment  that receives data from Tiscali's network.

He quickly identified the users affected, gained confidence that the cause was within the Tiscali network.  Mark passed his finding and extensive debugging information to Tiscali's engineers. We were asked to review the affected users and identify any geographic, product or customer equipment pattern. No such pattern was identified. At the end of the Saturday our supplier was unable to confirm ownership of the fault.

Sunday (27^th) and Monday (28^th)

Tiscali escalated the report to their second line support. Mark continued to monitor the disconnections and work with Tiscali's engineers. Together they were able to watch several failures as they occurred. By analysing the network path of customer disconnections Mark was also able to rule out a intermittent failure within one of our ERX's line cards (input modules) the Tiscali equipment they connect to or the fibre optical cables with interconnect them.

Tuesday (29^th)

The problem was internally escalated to Philip Webb, Plusnet's highest ranking network engineer and our final technical authority on network infrastructure.  Mark and Phil jointly reviewed the configuration of the ERX's and ruled out and misconfiguration of  this equipment.  Following further discussions and reviewing Tiscali's second line engineers were able to accept ownership of the fault and escalate the problem to third their line support.

Wednesday (30^th)

A major failure of another part of the Tiscali network took priority and occupied their third line support. We continued contact their network operation staff to indicate further disconnection events.

Thursday (31^st)

The problem was formally escalated to the Plusnet CEO (Lee Strafford) on Thursday. Mr Strafford and his team of directors made representations to their contacts within Tiscali.  They will continue to pursue the matter at this level until a resolution is achieved.

Friday (1^st Sept)

Tiscali's technical account manager for Plusnet working with their third line support identified the source of the authentication system errors and these were corrected. Tiscali identified the disconnections as originating from a section of their network infrastructure which is leased from NTL. Tiscali have raised the issue with NTL and are progressing the investigation  with them.

Further Work

We continue to run our ERX devices with diagnostic logging enabled. The information is reviewed and summarised daily. The results are shared with Tiscali.

Plusnet's network team contact their opposite numbers at least Tiscali twice a day.  The services of our network engineers have been placed at their disposal and on Monday we will reiterate our offer to source external expertise or diagnostic equipment to help identify the cause.