Website "Forbidden"???

[#1]

Quote

means that your DR plans were woefully inadequate.

I know this isn't what you want to hear, but it really was an unfortunate set of circumstances that caused this. The replication chain to the secondary storage cluster failed on Monday, while this was being repaired there was a disk failure on the primary cluster. Due to the replication chain breaking the secondary cluster didn't have full copies of all customers' content which is where the problem came from.
We're all very sorry this happened, but obviously happier now since the information on the latest service status came to light.

Quote

This should never of hapand.

Agreed, but it did happen and we worked hard to resolve it.

[#2]

If it was replicating, then surely the worst situation could have been that the downlevel copy would have simply had older copies of content on it?
ie, copies from the previous successful replication.
It should have been possible to have restored everybody's sites up to the point they were on Monday at the point of last replication, without having to wait for the volume to be fixed.
You should have been in a position very quickly to have reverted to Mondays last copy.
The restoration could have been configured to only restore files where newer versions weren't present, and you should have been back up and running within a few hours.
The fact that its taken so long, shows serious holes in the design of critical parts of your network. (I accept homepages may not be mission critical to you, but they are to many of your customers)
Its fine to have something go wrong - its not fine to make a cludge of fixing it.

[#3]

Just a further update on this, now we have moved the data over to the more resilient storage (NetApp),  failing to DR will be very quick and can now happen with minutes of us getting on it.

[#4]

Quote from: Midnight

Quote from: katie [big snip]  I cannot remember how I even did them in the first place. Katie why own why havn't got a back up of your web sites (s)?  I have a number of sites (s) totaling  other 1,000 pages all backed up &  backed up again.  F9/PlusNet I am not impressed, why own why havn't got a back up Server for this kind of thing, get one! This Is a mes, and you are becoming the biggest Joke.

I do have a copy of all my files for both websites, it's my memory on how to upload them again which is failing. I'm getting to the age where the memory cells are being lost more quickly than I like. I will get it sorted eventually, I'd just rather not if I can help it.
Katie

[#5]

If you want some technical clarification, I'm happy to give it.
Our homepages storage system has existed pretty much unchanged for a number of years... finger in the air guess with my bad memory, Jan 2003. 
The resilience and replication was at the time on par with the other data we had.  The replication method used was SNDR and this provided an asynchronous, constantly streaming, copy of the data from one site to our DR site.
Web storage from a resource perspective is not IO intensive, so the platform has sat barely breaking a sweat since its inception.
As part of a program of continuous improvement of our systems, some time ago it was deemed that the underlying storage for customer websites was not at a level we would like given advances in technology and other systems we have in house here.  A plan was drawn up to migrate this storage to a new platform which would address many of the shortcomings of the system we had identified.
The migration from the old storage to the new storage was scheduled to begin next week.
Our current DR webstorage suffered a massive failure late friday last week.  This was a simultaneous double-disk failure.  The decision was then taken to back the data up elsewhere at the same time as bringing the DR site back online.  SNDR does take a long time to resynchronise data from a source, so we were increasing our chances of data resilience in the case of an outage.
Because of the failure in the DR storage, the work to migrate to the new system was brought forward.  The migration began in earnest on Tuesday.
On Wednesday, we checked the progress of the migration.  We were around 50% over onto the new storage, which was faster than we anticipated.
Wednesday lunchtime, with about 20 hours left on the migration, the primary site suffered a similar failure to the DR site.  This was extremely frustrating for us, having been over halfway through a migration and having two geographically diverse sites suffer such a failure.
This is the reason the data was damaged.  For this we can only apologise.  A large majority of our team has been working around the clock to restore data and get things back to normal.
As of today our engineers were able to bring the old primary storage back online, however this was no longer being 'pointed at' by our web and FTP servers.  Work is ongoing to recover the gap of data from what we had in backup and what was missing overall.
Now onto the future...
The new storage has resilience for both a failure within a site, as well as an offsite system that can be failed over to.  Point-in-time near-instant  recovery of the whole filesystem is possible.  Point-in-time "not-near-instant" recovery of individual users files is possible.
We can fail over the storage from one node in the site to the other without a disruption in service.  This means that we could take half the system offline (say to replace a motherboard) without even causing a blip in availability.  We are protected in all cases from a double-disk failure (these don't happen very often!).
We can fail over the storage by pointing our web and ftp servers to our DR site, which contains all the data with a lag of up to one hour.  This DR copy also has the point-in-time recovery features of the live site.
As for performance of the new storage, the utilisation of the storage is also around 15% allowing for plenty of growth.  The platform is also upgradable in both capacity and throughput without much trouble (and in many cases without any downtime at all).
In summary, from an engineers perspective, it's been an extremely unlucky and frustrating week for us, we are very sorry we have let you down as our goal is to keep you all happy the whole time, and I can assure you that now we are on the new storage this particular incident can simply never happen again.
Wishing you all a good weekend and many happy websites,
Gricey.

[#6]

Quote from: Gricey

...As for performance of the new storage, the utilisation of the storage is also around 15% allowing for plenty of growth.  The platform is also upgradable in both capacity and throughput without much trouble (and in many cases without any downtime at all). Wishing you all a good weekend and many happy websites, Gricey.

What about passing on extra capacity to our websites up from 250MB to say 500MB? I am sure our weekends would possibly be better after something like that.

[#7]

Quote from: geezer

What about passing on extra capacity to our websites up from 250MB to say 500MB? I am sure our weekends would possibly be better after something like that.

Yes, now! My ccgi is otherwise soon full. I could give up on www for that. Deal possible? Would solve a lot of headaches (and it is not my fault that symlinks don't work on ccgi).

[#8]

Just one slight problem with that: not only are homepages and ccgi different servers, but homepages is in Sheffield and ccgi is in London!

[#9]

I just wanted to say that my websites are back in place as promised;. Thank You, but can you give me notice next time, so I can get a supply of Vallium in.
Katie

[#10]

Well I've got a ticket still open for a friends webspace which still hasn't been restored.
The ticket keeps being responded to but noone will answer the question as to when exactly the content will be restored.
If anyone who has access to the appropriate tools to do so could take a look at restoring the content for www.oakescottages.plus.com I would be extremely grateful.
I'm not used to this unhelpful response from PlusNet, usually whenever a problem has been reported on the home accounts and the business accounts I look after the op has been able to resolve or at least give informative response as to when a problem will be fixed.
Thanks in anticipation
Roly.

[#11]

Quote from: roly

If anyone who has access to the appropriate tools to do so could take a look at restoring the content for www.oakescottages.plus.com I would be extremely grateful.

It's all there, however it's within a folder on their space. If you ask them to log in via FTP they can see the folder on there and confirm all the data is there.
*edit*  I've looked a bit further into this and it appears the script missed this site, the networks guys are having a look into this.

Quote

I'm not used to this unhelpful response from PlusNet, usually whenever a problem has been reported on the home accounts and the business accounts I look after the op has been able to resolve or at least give informative response as to when a problem will be fixed.

We've kept customers up to date via the service status tool on a regular basis for this problem with as much information as was available each time.

[#12]

Quote from: Chris

[It's all there, however it's within a folder on their space.

How from the service status announcements etc. was somebody supposed to figure that out?
Why haven't people been emailed to tell them what's been done?
Edit: I've just checked my own webspace and found an htdocs.20090119010320 folder - as my websites came back pretty quickly it hadn't occurred to me to check!
Edit 2: ... and now I've found a difference with some extra files which I added to the website on 18th Jan.

[#13]

Roly - I've answered your ticket.

[#14]

Quote from: jelv

How from the service status announcements etc. was somebody supposed to figure that out?

Figure what out?

Quote

Why haven't people been emailed to tell them what's been done?

We've kept everyone up to date via service status, customers can subscribe and receive emails from there if they wish.

Quote

Edit: I've just checked my own webspace and found an htdocs.20090119010320 folder - as my websites came back pretty quickly it hadn't occurred to me to check!

We've raised a problem on these directories this morning.

[#15]

Quote

This is an update to the previously reported problem preventing some customers from being able to access their hosted websites, or connect to our FTP servers. A copy of the previous announcement can be found here:- http://usertools.plus.net/status/archive/1200679722.htm Our network engineers have advised that all data has now been successfully copied back to customers directories and all site access and content should now be restored.

"Restored" means put back to as it was - not a copy of the website put in to a new folder that can only be accessed via FTP.