Critical Path Anti-spam Maintenance

[#1]

The increasing queues show up well on the Mail Queues graphs. I see the mxlast servers are also affected.

[#2]

It does look like queues are rising across the entire platform.  My gut reaction would be that it's genuine load across the platform.  My assumption is that no maintenance was applied to mxcore09 and mxcore10 and they're also showing the same load increase so it's possibly not caused by the maintenance itself.
However, I'm sure Bob has his eagle eye on it.
B.

[#3]

I've got my eagle eye on it and have already asked the networks guys, they say it's nothing to worry about at the moment and is just load. I'll keep an eye on it and prod them if they continue to increase.

[#4]

Yeah, seems like the lasts are getting hit pretty hard (which would normally be suggestive of a surge in spam). We've got all hands on deck at the moment, taking a look at this.

[#5]

Looking at the graphs now they are starting to come down too

[#6]

Yesterday I didn't receive a single spam message (tagged or untagged), most unusual and raised a niggling doubt that all was well (???).  However normal service resumed this morning.
Perhaps the spammers had a day off for Halloween and that accounts for some of the heavier load today. 

[#7]

The Critical Path boxes aren't rejecting anything our normal platform wouldn't. Neither are they tagging the subject line of emails just yet.

[#8]

Quote from: Bob

Quote Anyhow, so far it's looking good.  I reckon Bob should be popping the kettle on in celebration You've obviously not heard about my reluctance to make tea for the team

TWO! How long did it take you to recover?

[#9]

We're currently looking set for 3 in 2 days.
Clearly Bob is now demonstrating excellence in delivery, which is one of our internal "cornerstones".

[#10]

Some may notice the blip in the Critical Path graphs over the last hour. This is because I was getting forced into making tea and left my monitoring post to look for a teaspoon!
Only kidding! It was caused by an increase in load across the platform and we've tweaked the connection settings on the CP boxes which seems to have done the trick.

[#11]

Bob - put that kettle down and kick the graphs.  They seem to have stopped dead
B.

[#12]

The monitoring on the servers has decided to stop temporarily, it's being looked at and should start again.
Trust me Barry, Bob doesn't have the kettle in his hands, it's the lack of tea that gives it away.

[#13]

Looks like monitoring at one of our Sheffield data centres is down. We've just been over there to do some work so I'm wondering if somebody's knocked a cable somewhere!

[#14]

No one spilt tea on one of the servers did they!?! 
Phil

[#15]

I see some of the webspace server graphs have stopped as well.