Critical Path Anti-spam Maintenance

[#1]

Yeah, t'will be the same thing Jelv. Should be fixed pretty soon.

[#2]

I guessed it would be - I just wanted to stop the conspiracy theories that it was just the CP graphs that had been taken off because it was going wrong!

[#3]

Now fixed, and guess what...
Somebody knocked a cable!
*points*

[#4]

giggle.
An easy mistake to make...

B.

[#5]

*Trips on a cable*.
"Whoops... there goes all the customers' connections".
🙂

[#6]

Watt is that Barry?  

[#7]

Quote from: tdadyslexia

Watt is that Barry?

http://www.sysadminday.com/horrors.html
A fantastic site of networking horror stories.
That's what a comms room can end up looking like if it's not looked after
B.

[#8]

Quote from: Barry

That's what a comms room can end up looking like if it's not looked after

Is that your comms room Barry? 
Runs off and hides.

[#9]

The CP queue looks pretty weird here
Either it is broken or it is showing 0.0 items queued.

[#10]

I think that's actually correct - the CP boxes have been outperforming the mail servers in processing the mail
B.

[#11]

Hi Barry,
They didn't originally - when they were only connected to one or two mail servers - so why are they doing it now? Is the mouse running faster on the wheel?

[#12]

I wonder if it was that the mail servers behind the CP servers were not taking the mail fast enough (i.e. it was the queue of emails to be passed on); now there are more behind, the CP servers can shift it on as fast as they get it.

[#13]

That's a fair assumption. There have been a number of tweaks to the number fo connections the CP boxes and our mail servers will accept. It was always going to be a case of achieving a fine balance, however we're now in a position where our mail servers are happily dealing with the email being handed over by the CP boxes. The graphs aren't broken, it's just that there genuinely isn't any email queued.

[#14]

Today's update.
Things are on hold at the moment. sunmxcore 09, 10, 11, 12 & 13 are still behind the Critical Path boxes.
The reason we've not resumed the roll-out is due to an issue we've identified with the disk swap on one of the boxes. This isn't causing any customer facing problems at the moment but it's something we need to investigate as it could easily affect performance once more servers are added.
We'll be restarting some of the services on the affected box shortly. This is likely to result in a blip on one of the Critical Path graphs as emails are backed up slightly.
Edit: The restart seems to have sorted it and the problem is now resolved. We're back on track to place another server (sunxcore14) behind the Critical Path boxes tomorrow morning. We'll then probably work at the rate of one a day.

[#15]

After a few days of being on hold, the trial resumed yesterday morning.
We now have 7 servers behind the Critical Path boxes (9, 10, 11, 12, 13, 14 & 15)
Load and CPU usage on these devices are currently less that 40%, so there is now nothing which should stop us completing the roll out early next week as per the Service Status update I posted yesterday.