cancel
Showing results for 
Search instead for 
Did you mean: 

Usergroup/Usertools

dvorak
Moderator
Moderator
Posts: 29,473
Thanks: 6,623
Fixes: 1,482
Registered: ‎11-01-2008

Re: Usergroup/Usertools

No emails come through here either..
Customer / Moderator
If it helped click the thumb
If it fixed it click 'This fixed my problem'
woldsend
Dabbler
Posts: 21
Registered: ‎13-11-2008

Re: Usergroup/Usertools

I'm subscribed but haven't received any notices yet.
Peter_Vaughan
Grafter
Posts: 14,469
Registered: ‎30-07-2007

Re: Usergroup/Usertools

nothing here either so it looks like that part is not working yet.
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,869
Thanks: 4,950
Fixes: 315
Registered: ‎04-04-2007

Re: Usergroup/Usertools

My RSS feeds aren't updating, neither have I received any email updates to an email addy that I have subscribed. Looks like there's going to be some residual issues to iron out Sad
I'll update Service Status to that effect however it would be interesting to know which of the following services aren't back to full operation. Does anybody know of any issues aside from the Service Status feed? Some scripts were missing from the box and I'm guessing it's one of these that's causing the problems.

  • Usergroup site
  • Usergroup forums
  • Issue tracker
  • Service Status RSS/Email feed
  • Exchange checker
  • Usertools site
  • IRC
  • Anything else I've missed

Quote from: dhookham
Surely the service status notification facility alone ought to have warranted it being a P1, with the various other elements such as exchange checker, PUG website and forums, etc only increasing the importance of a swift recovery.

Yes, as mentioned in one of my replies to Liam, I would agree. This has been raised internally and we'll be much more vigilant in the event that a similar situation arises in the future (which  hopefully it won't).
Quote from: James_G
I struggle to see how its taking so long.
If it took me so long to restore onto new hardware from a backup, the powers that be would be seriously displeased - why is a company whose entire business model is based around having computers that work[!] taking so long to restore one little box?

On Thursday when we realised the box went down an engineer was sent to one of our data centres as we couldn't access the machine remotely. We tried rebooting but encountered a shed-load of read errors on bootup. We tried numerous boot discs with a view to fsck the hard disk but to no avail. At this point the decision was made to pick things up first thing in the morning (probably a mistake in retrospect). The next day we connected another disk and attempted to sync this and then fsck the drive. This worked on the second attempt but wouldn't boot due to OS errors on bootup. We then mounted the HDD on another box, grabbed the data off that and started rebuilding a replacement server. The data was left copying to the new server over the weekend and a decision made to resume the rebuild first thing this morning.
Quote from: James_G
It smacks of lousy [non-existent?] DR planning.

Which is something we need to address. The Usertools platform doesn't have the resiliency of other value-added/core services. If we want to treat it with the priority it deserves then we need a plan of action to address this - another thing that is being discussed internally. I think we probably need to get stuff across to the Community servers as has been talked about before (although I'm not sure where we got with that as I've not really been involved in much of the discussion).

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Usergroup/Usertools

It appears to me that a significant part of the problem (and probably something that was frustrating you Bob in your attempts to get some action) was that this is the usertools and usergroup server and was perceived by some people internally to therefore to a certain extent to be the responsibility of the Usergroup. I am certain that a very major part of the load on this server is the service status emails and RSS feed, the Exchange checker and the gateway checker. Moving those functions on to a fully supported main server such as Community is years overdue with clear responsibility for ensuring that updates are done being given to an appropriate department. It seems that in particular the exchange checker is being largely maintained by an ex-user now employed by Plusnet in his spare time.
Edit: I think the statement about usertools support which appears on the website says it all:
Quote
Note: PlusNet Support can not provide any support for the tools offered on this website. Questions regarding these tools should be sent to the Author using the Contact the Author link on the appropriate tool.
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Usergroup/Usertools

Useful links is giving an error:
Quote
PlusNet Links Database
Warning: mysql_connect() [function.mysql-connect]: Access denied for user 'links'@'localhost' (using password: YES) in /var/www/html/links/config.php on line 96
PlusNet Links Tracker
This is a collection of useful, and sometimes not so useful (but fun), links relating to PlusNet
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
Lurker
Grafter
Posts: 1,867
Registered: ‎23-10-2008

Re: Usergroup/Usertools

Quote from: Bob
On Thursday when we realised the box went down an engineer was sent to one of our data centres as we couldn't access the machine remotely. We tried rebooting but encountered a shed-load of read errors on bootup. We tried numerous boot discs with a view to fsck the hard disk but to no avail. At this point the decision was made to pick things up first thing in the morning (probably a mistake in retrospect). The next day we connected another disk and attempted to sync this and then fsck the drive. This worked on the second attempt but wouldn't boot due to OS errors on bootup. We then mounted the HDD on another box, grabbed the data off that and started rebuilding a replacement server. The data was left copying to the new server over the weekend and a decision made to resume the rebuild first thing this morning.

I accept what you are saying as a representation of what happened, but seriously, a restore to replacement hardware should have taken very little time, and although the numbers look large in terms of the number of visits, its not a huge amount for a webserver to cope with, and just about any off-the-shelf box would have done the job if there wasn't some spare kit lying around the place to use on a short term basis.
This isn't the first time your engineers have made a hash of executing a recovery [whether it was caused by blindly following a flawed plan or not] - I hope you are better prepared if a core service fails.
Colin
Grafter
Posts: 1,264
Registered: ‎04-04-2007

Re: Usergroup/Usertools

Quote from: Bob
My RSS feeds aren't updating, neither have I received any email updates to an email addy that I have subscribed. Looks like there's going to be some residual issues to iron out Sad

That'll be deliberate... because the stuff isn't back in Cron yet.
Colin
Grafter
Posts: 1,264
Registered: ‎04-04-2007

Re: Usergroup/Usertools

Quote from: jelv
Useful links is giving an error:

Someone uses that?? Fixed.
jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Usergroup/Usertools

It gets worse! If a cron hadn't been set up why was the following service status post made:


09/02/2009 @ 16:56Usertools/Usergroup Accessibility (55128) - UPDATE
This is an update to the previously reported problem with access to the Usertools and Usergroup websites.
Our engineers have now successfully resolved the problems with the Usergroup & Usertools websites. The Usergroup site and forums have been restored and the RSS Feeds and Exchange Checker are now working.
Please accept our sincere apologies for the period of downtime.
We'll leave this status posting set to Amber overnight to ensure that there are no further issues.
Kind Regards,
James Bailey
Customer Support


James will have been told that everything was working before he made that post (I don't believe for one second that if he had been told that RSS and service status emails were not working he would not said that in the service status).
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
Lurker
Grafter
Posts: 1,867
Registered: ‎23-10-2008

Re: Usergroup/Usertools

Left hand - meet right hand...
jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Usergroup/Usertools

Quote from: Colin
Someone uses that?? Fixed.

I don't, but I did what someone at your end should have done and went down the menu clicking each in turn to check if it was working. Did someone check quicklinks as that aslo uses a database I presume.
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)
James
Grafter
Posts: 21,036
Thanks: 5
Registered: ‎04-04-2007

Re: Usergroup/Usertools

Nope, I didn't check.
I tried out a few links and saw a test post that Colin had put into the forums.  Probably should have checked with Colin but at the time it seemed to be working fine.  I kept the status set to amber just incase there were any issues.
Colin
Grafter
Posts: 1,264
Registered: ‎04-04-2007

Re: Usergroup/Usertools

RSS should now be back and generating, emails will need to wait a bit until I've had a chance to babysit it through the first couple of runs.
Colin
Grafter
Posts: 1,264
Registered: ‎04-04-2007

Re: Usergroup/Usertools

I've just run the email script manually... can someone confirm if they get the email(s)?