Webstats not generated (6th April)

[#1]

Can someone give then a kick again.
Tony

[#2]

If this script fails for the same reason each time it fails then it should have been fixed long ago. If it fails for a different reason each time then it must be very fragile and needs to be made more robust. Surely a job that has run every night for many years ought by now to have evolved into something more reliable.
Keith

[#3]

Given it's Saturday I suggest you ring in - posts here may not be spotted in time.

[#4]

Quote from: chrispurvey

We do now have in-place a more vigilant monitoring process for when/if this occurs.

That statement was made about 24 hours ago, after yesterday's webstats failure. No sign of a Service Status announcement yet.

[#5]

Hi all,
Really sorry about this, especially two days running. We're looking into it now, I'll post a service status shortly.

[#6]

Problem's been raised and we've got a service status up: http://usertools.plus.net/status/archive/1365248944.htm
We'll make sure we check into why it's failed two days in a row now, and get that sorted. Really sorry about this guys.

[#7]

I'd like to acknowledge the courteous and helpful manner in which Mark and his team leader dealt with my call and took action to move this forward. #67973801
David

[#8]

Is it just me or are we coming up to 7.30pm with the stats still not generated?
Actually, looking at the service status it says the next update is due '08/04/2013 @ 11:00', so that it it for the weekend.
Tony

[#9]

Quote from: Tony

Actually, looking at the service status it says the next update is due '08/04/2013 @ 11:00', so that it it for the weekend.

... just seen your post, with some degree of dismay.  I'd seen the thread earlier, but hadn't added to it as it looked as though everything was in hand.

Quote from: kdiment

If this script fails for the same reason each time it fails then it should have been fixed long ago. If it fails for a different reason each time then it must be very fragile and needs to be made more robust. Surely a job that has run every night for many years ought by now to have evolved into something more reliable.

... yeah : )  you're probably right.
Have to admit to being a bit concerned at the prospect of losing two days' worth of stats (primarily because it skews the monthly and annual cumulative figures).
More worried though that there's been a significant failure again, hot on the heels of the one a few weeks back where two consecutive runs failed and stats were lost.  I read the update (other thread, yesterday I think) but I don't think it actually said what *caused* the failure on that occasion.  And now a fairly swift recurrence, seemingly.
Whatever it is, I hope it's sort-able.
Regards,
Penny.

[#10]

Sorry it's so late (thanks spraxyt) - had a chat with Kelly and called the networks on call, we're trying to get them run for today so the stats aren't lost.

[#11]

Too late.

[#12]

Quote from: Matt

We're looking into it now, I'll post a service status shortly.

Quote from: Matt

had a chat with Kelly and called the networks on call, we're trying to get them run for today so the stats aren't lost.

Did it really take 11 hours to decide to call someone to try to fix the problem?
Today's log file was generated about 2 hours earlier than normal and only contains the data for yesterday (6 April).
All of the data for 5 April has been lost despite Plus Net knowing about the problem since at least 12:29 yesterday.

[#13]

My guess is that the late call out did result in the stats run being manually started but unfortunately by then the midnight switchover had caused the previous day's files to be overwritten. The normal run for today, which would have produced the stats later, probably failed (exited) because the stats script was already running.
From my conversations last night I'm aware of the procedural reason why action was not taken earlier (when the problem was raised). That is something that will need sorting out on Monday. Unfortunately the lack of feedback did not help to get those procedures challenged until it was too late.
David

[#14]

The web stats have been cocked up so many times in the past that there is absolutely no excuse whatsoever for everybody not knowing that time is of the essence!

[#15]

We've now closed the service status. Unfortunately Friday's webstats are no longer retrievable as the scripts hadn't completed before the next days scripts were due to start.
We've flagged this up to the relevant team.
Sorry for any inconvenience caused.
Chris