SSI - Server Side Includes - stopped working

[#1]

I see two of the webservers' Apache software was updated today.  Maybe this is connected or not, but SSI have stopped working on my website http://www.thechurchinbinleywoods.org.uk/
The "Question" input page says new incidents are waiting over 29 hours before they are looked at at the moment - this will be a long time to wait.
Has anyone else got the same fault?
Thanks!
Edit: My ticket is 39861785

[#2]

Oh **** !
[me=jelv]goes to check![/me]

Mine's broken too!
Edit: ticket 39865902 raised

[#3]

Thank for checking, jelv!  It added some weight to my phone call just now.
I just got through to Support on the phone, and they said that fixing it is a low priority because not many people have raised tickets for this problem.
I suggested that people don't keep checking their websites every day - I only checked mine because I had an update.
So if you are reading this, please check your website and raise a ticket if you are affected, and if you can afford it, call on their revenue-sharing phone number, and they might fix it this week.

[#4]

I've put a notice on the front of my site: http://www.pdacontact.org.uk/
I suggest anyone affected does something similar.
This could not have come at a worse time. The National Autistic Society has just held a major conference on PDA and there is a great deal of interest at the present time. All pages bar the front page of the site use SSI so it is totally wrecked.
I've been [Censored] off with Plusnet before, but

Quote from: jws

they said that fixing it is a low priority because not many people have raised tickets for this problem.

has really wound me up!

[#5]

I've just checked our website stats. As I expected February is showing the highest average daily hit count ever!

[#6]

Hey Guys,
I'll make sure that someone is aware of this when I get into the office in the morning.
Sorry for any inconvenience this may have/be causing you.
Regards,
Matt

[#7]

Morning guys, a problem has been raised (ref: 65797) and we've just re-added the include module that controls SSI. Can people try again and report back here if it's still not working please. Apologies for the time taken to pick this up, should have been looked at yesterday considering it's almost certainly this that broke it

[#8]

I confirm that it is now working.
What is annoying is that immediately after I raised the ticket I rang in to try to make sure it got immediate attention and suggested that it should be a P1. Obviously that suggestion wasn't taken up.
I may be teaching my granny to suck eggs, but I hope a check has now been done to look at all the modules previously loaded and ensure they are on the new setup! (or you know why they have been dropped)

[#9]

Note: When I checked my site I found that I needed to clear the browser cache before it displayed correctly.

[#10]

Yes, my site is working too, and I also had to force a full refresh to clear the cache.
There must be lots of ways to test upgrades to avoid this sort of thing such as testing before deployment on a test environment and then verifying after deployment to production, for example by using a reference website that exercises all the expected modules and facilities.
If you can't be confident in your testing, PlusNet, then you should really do one server first instead of two, and notify your customers who have websites on that server in advance, asking them to check their websites, and provide a hot line for them to call in with any issues...
.. and then take even the first call seriously!
I have to say I really was astounded by the approach that PlusNet would need maybe 10,000 calls before they gave priority to an error reported immediately after a change (such as the Apache upgrade).  The 10,000 number was stated by the agent who took my call yesterday evening.  Our discussion was about the low priority PlusNet was giving to the SSI issue because they had received only a few calls.  I was saying that I have seen PlusNet Status messages that mentioned priority being given to issues because "a few" customers were experiencing some problem.  The agent replied that in those cases the "few" meant 10,000 customers, as opposed to the handful who had reported the SSI issue.

[#11]

Quote from: jws

If you can't be confident in your testing, PlusNet, then you should really do one server first instead of two, and notify your customers who have websites on that server in advance, asking them to check their websites, and provide a hot line for them to call in with any issues...

To be blunt, I don't think we've the ability to test in this capacity when we make the number of changes we do. To give you an idea, I've just checked the list of change controls for this month and the number is approaching 100!

Quote from: jws

I have to say I really was astounded by the approach that PlusNet would need maybe 10,000 calls before they gave priority to an error reported immediately after a change (such as the Apache upgrade).  The 10,000 number was stated by the agent who took my call yesterday evening.

That's not accurate information and I strongly suspect it was a relatively inexperienced agent you spoke to. I'll find out who it is and have a word with them.
In other news, the Apache upgrade has been rolled back now due to some other problems we've come across. I'll post to our Service Status feed when we're due to roll it again...

[#12]

In this case, surely the desk checking of the proposed change control should have included looking at which additional modules were to be loaded compared with the old version.

[#13]

Bit more info here.

[#14]

Heads up that we upgraded the first of the six servers again yesterday afternoon.