cancel
Showing results for 
Search instead for 
Did you mean: 

More Lip Service, Anyone?

N/A

More Lip Service, Anyone?

I see our emails are gone forever:

Quote
Our data recovery specialists have been able to provide a partial file list of the email data, but it has since become clear that we will not be able to recover the directory structure. Without the directory structure we cannot recover any meaningful data, due to complexity of associating the data with the relevant customer accounts.

I would like to take this opportunity to once again extend our sincerest apologies for the inconvenience caused by the loss of any customer emails and in particular the unread emails. This has been taken very seriously within PlusNet. An extensive analysis of how this happened has been carried out and working practices implemented to prevent it happening again.


Not good. I think we are entitled to know:

i) Who are these 'recovery specialists' who can't recover?
ii) What does it actually mean if something is taken very seriously within PlusNet?
iii) What are the working practices that have been implemented?
iv) What guarantee is there that they will prevent it from happening again?

Frankly I have very little confidence in anything coming from this organisation at present. I believe it's all just hollow words and promises.

I've been with PlusNet since October 2001 and in all honesty I have never known the service to be as bad as it is now and I seem to be one of the lucky ones as my connection is at least fairly stable.

I have since moved my domains and hosting to a different ISP, my important email is now with a different ISP and I am certainly looking at jumping ship completely.

I wonder - what would happen if we all cancelled our direct debits for one month in protest?
14 REPLIES
James
Grafter
Posts: 21,036
Registered: 04-04-2007

More Lip Service, Anyone?

Hi Bullwinkle,

Have you also read Simon's posting here?

http://usergroup.plus.net/forum/index.php/topic,2753.0.html
N/A

More Lip Service, Anyone?

I have tried but I am asked to log in and then told "That username does not exist".

Do I use my normal userid, ie 'bullwinkle'?
James
Grafter
Posts: 21,036
Registered: 04-04-2007

More Lip Service, Anyone?

No, you actually need to register to the site first.

The PlusNet UserGroup is not actually part of the PlusNet Portal. It's a site run by the PUGgers separately.
gofaster
Rising Star
Posts: 364
Thanks: 11
Registered: 01-08-2007

More Lip Service, Anyone?

I do not wish to register with PNUG - in fact I am unable to register with PNUG (I am not currently using my PN connection). Why on earth can't Simon's post be put HERE where all PN users can see it?

Surely an open and frank supplier like PlusNet isn't too embarrassed to give full details of this fiasco?
channel
Grafter
Posts: 697
Registered: 03-09-2007

More Lip Service, Anyone?

For the benefit of anyone who doesn't want to register on the PUNG site, but still wants to have the fullest information on how their email was destroyed, here is a copy of the post from Simon Day, which jamesbailey linked to earlier on:

As with my initial posting on the loss of customer email data - http://usergroup.plus.net/forum/index.php/topic,2472.0.html, what follows is an open and honest account of the events surrounding the attempt to recover the data.

Firstly you need to understand that in an attempt to recover the data swiftly, the engineer who deleted the 3 volumes in the first place swiftly followed up his error by immediately trying to create a volume of the same size as the 1st of the volumes in the same place. This is an old sysadm "trick" that on some file systems could have revealed the lost data, however in this case, it did not work, and in fact caused us more problems, as you will see later on in this account.

Within 2 hours of the data being deleted, a data recovery company had been contacted and within 3 hours the NAS was in transit to them. By 14:00 on that day the specialists were racking the NAS and began the process of copying all the 1's and 0's from our equipment to their own. This is standard operating procedure for anyone working in the field of data recovery, and is simply about ensuring that there is always an untouched copy of the information in case something further goes wrong while working on the recovery. Due to the volume of data that was being dealt with, the copy took until the early hours of the following morning.

At that point, based on their initial investigations, the data recovery specialists set the expectation with us, that we would recover some of the data, possibly not all of it though, and that it could take 4-5 days. From that point forward we have a tale of increasing woe as each new deadline set by the data recovery people was broken as they discovered more and more problems. In the following paragraphs I will briefly cover off the main problems that have been encountered.

The Sun NAS that we had selected for the mail storage platform is the first series of products to emerge from Sun since their purchase of StorageTek, and as such does not run the usual Sun OS of Solaris. It uses StorageTek's own proprietary OS which is a heavily modified FFS2 (Fast File System 2). The modifications are all about increasing the performance of the system to ensure enterprise level performance.

As the kit is fairly new to market, the data recovery specialists had not worked on this specific OS before, though they do have a lot of experience with NAS's in general, Therefore they had to significantly rewrite the tools that they use for analysing and recovering data. They utilised their engineering departments in both the UK and the US to work around the clock to achieve a re-worked set of tools.

Apart from the tools issue, the proprietary OS, uses the 1st volume it has access to, to store the master inode table. For more information about inodes, take a look at the Wikipedia article http://en.wikipedia.org/wiki/Inode. Essentially this is the table that tells the system where all the other files on the system are. As I mentioned earlier, the PlusNet engineer involved, had attempted to recover the data by creating a volume of the same size in the same place as the 1st volume. That action more than any other has caused us the most issues. By creating a new volume, the existing inodes were wiped and all data that was on that volume was essentially gone. Without that master inode table and with no knowledge of where the system stored it's back-up copy of this table, it has proved very difficult to work out what the data on the relatively undamaged 2nd and 3rd volumes actually is.

We have received a partial file list from the 2nd and 3rd volumes. This list amounts to a list of inodes and the data in them, not the list of complete files. Without even a partial directory structure it becomes vastly more complex to work out which inodes are associated with which other inodes and therefore piece together the complete files. Without the data on the 1st volume we do not believe we are ever going to get the directory structure. Without the directory structure it becomes vastly more complex trying to work out which file from the partial list belongs to which user.

So, here we are, it is almost a month since the 700GB of email and mailing lists were lost and we still have no recovered data to return to you. This is of course upsetting for us, and even more so for the customers whose data has been affected. The longer we wait for the equipment to be returned to us the greater the risk we run of hitting other capacity issues that we know are ahead of us, and we do not feel that we can justify any longer a wait, and still be taking the appropriate action for our customers.

When it is implemented the new platform will provide us and you with a vastly scalable, site resilient mail storage set-up, with 6 hourly checkpoints to ensure we can roll back the majority of the changes that happen on the system within a four hour period.

What this really means is that we are currently arranging for the return to us, of the Sun NAS head unit and Disk Arrays, so that we can push forward with the implementation of the new email storage platform.




--------------------------------------------------------------------------------
Simon Day
Network Improvement Consultant
PlusNet Plc
N/A

More Lip Service, Anyone?

Thanks, channel.

If I read this correctly, nothing has actually been implemented but *will* be implemented despite the email from Alistair Wyse in my original post stating the contrary.

I can fully understand an engineer making a mistake, but what I don't understand is what happened to the backup strategy to mean there was no offline backup of the storage. Why no tape backup?

The article still doesn't say who the recoveryless specialists are. How do we know that these weren't a two-bit company operating out of a PO Box No?

Also - I do find it extremely disappointing that there is not even the slightest offer of goodwill to those whose email has been destroyed.
Grff
Grafter
Posts: 351
Registered: 02-08-2007

More Lip Service, Anyone?

Quote

The article still doesn't say who the recoveryless specialists are. How do we know that these weren't a two-bit company operating out of a PO Box No?


What difference would it make if you knew which company has looked at the data anyway? It's been said a lot of work from both the UK and US has looked at it so it doesn't sound like a small 2-bit company to me.

I'd have thought users would have some sense to download mail into a mail program locally anyway. Maybe also setting the option to 'leave a copy on the server' if they wanted to view it from somewhere else too.

If plusnet don't know who the bits of files they can recover belong to, can they actually tell who is missing mail to compensate them anyway.
Community Veteran
Posts: 1,656
Registered: 13-06-2007

More Lip Service, Anyone?

I assume it was someone local meaning that it was probably the 1st/2nd result (recover.dat) from yell here

As it states "no fix, no fee" then PN may have saved a fortune lol
N/A

More Lip Service, Anyone?

What would be wrong with taking the recovered data without the file system structure, analysing the data to find the email headers and retransmitting the data following data to the address(es) found in the headers?

Sure, this will cause more work for the developers, and it's a question of whether it is actually worth the labour, or to just forget about it and try to ensure it doesn't happen again.

Luckily, I don't actually use my plusnet account for anything but email from Plusnet.
Community Veteran
Posts: 3,069
Thanks: 201
Fixes: 3
Registered: 05-04-2007

More Lip Service, Anyone?

It sure hasn't taken The Register long to get wind of it.
Though I am sure their version won't be quite so detailed as what we've just had!
KevinHyland
Grafter
Posts: 61
Registered: 20-08-2007

More Lip Service, Anyone?

Quote
Thanks, channel.

...I can fully understand an engineer making a mistake, but what I don't understand is what happened to the backup strategy to mean there was no offline backup of the storage. Why no tape backup?


Also - I do find it extremely disappointing that there is not even the slightest offer of goodwill to those whose email has been destroyed.


I too would like to know why there was no off-line storage? Surely everything should have been backed up, with the tapes being stored off-site.

What disaster recovery procedures does Plusnet have in place? What if the servers were (God forbid) caught in a fire or other catastrophic event?

Rule 1 of backing up!! Keep copies in a secure, off-site location!!!

Incidentally, I believe I am currently short of about 600+ e-mails. Admittedly most were already opened and there was a certain amount of rubbish too. (Poor housekeeping on my part I know!) Nevertheless I feel extremely let down by Plusnet. Some form of compensation is definitely due - and there must be many people hit far harder than I am.

Half price ADSL for a few months anyone??

P.S. I notice the Service Status light for e-mail is now green. Personally, as the issue can never and will never be properly resolved, I believe it should stay amber permanently as a mark of respect to our deceased e-mails...
kosh2
Grafter
Posts: 211
Registered: 04-08-2007

More Lip Service, Anyone?

Is it practical to keep backups of such fluid and ever changing data? (Thats an honest question and not sarcasm by the way.) And what back-up policy do Plusnet have?
N/A

More Lip Service, Anyone?

Quote
... I do find it extremely disappointing that there is not even the slightest offer of goodwill to those whose email has been destroyed.


Quote
Half price ADSL for a few months anyone?Huh

I agree.
What did we say here and hereHuh

We've been given the run-around for over 3 weeks and now they just say they're sorry - it's just not good enough :!: I've lost a fortnight's emails because I had the misfortune to be on holiday while this bunch of dimwits fouled things up.
What are PlusNet's proposals for compensation? That's one way they can show just how sorry and sincere they are. Evil
N/A

More Lip Service, Anyone?

Quote
Is it practical to keep backups of such fluid and ever changing data?

I would agree if it was POP3 / IMAP only, where there will only ever be transient data. When offering a webmail service it's reasonable to assume people will store mail there and you would expect some level of off-site backup.

I wonder if PNs backup policy extends to their ftp and web servers too?