Evening all,

A quick heads up that I've just posted the following Service Status and blog posts to the Usertools Site and Comms blog regarding a significant piece of work we'll be carrying out on the mail platform next week.

Basically, we will be deploying our Critical Path anti-spam solution in front of the live customer servers again. I'll not go into too much detail here at a risk of repeating myself, but I'm sure I'm right in assuming that some of you may be slightly concerned by this prospect - Especially given the email delays/loss that occurred last time.

We're fully confident we've covered all eventualities this time round and have a strict roll-back plan which can be implemented quickly at the sign of any trouble. I've spent some time explaining the rationale behind this decision, and have attempted to allay many of the concerns customers may have in the Service Status and blog post I have made.

If you have any feedback, concerns, questions or worries about this work then please feel free to post them here and we'll do our best to address them. I'd strongly suggest reading the infromation I have linked to before doing so as there's a good chance I may have already covered the question you're wanting to ask.

There's always going to be a risk associated with any work we do, however we feel that we're very well prepared for this work and have gone a long way to ensuring that all eventuatlities we can think of have been covered.

Needless to say, many of you have been very helpful in the past in quickly identifying problems both during and following a roll-out and we'll especially be looking for your feedback during and after this work. Those involved with the roll-out will be monitoring this thread closely and we'll be on hand to quickly identify and react to any issues should they occur.

The benefits of this work should it be successful are very promising and this should hopefully see a large step towards overcoming the reoccurring issues (notably email delays) that keep on popping up to bite us.

Incredibly verbose description Bob, for which I thank you.

At least by posting this then the community as a whole will be able to assist in any issues that may arise.

I look forward to seeing the performance graphs over the coming weeks to see how the new configuration performs in a live environment.

Good luck!

B.

You're welcome!


Which is certainly the idea.


Thanks, we moved two central pipes the other day without hitch so here's hoping!

Subtle reminder that this work will be commencing tomorrow morning.

After the first server goes live you'll be able to see graphs showing the latency and queues on the Critical Path devices compared to the queues on the sunmxcores. This will allow you to see exactly what's happening in real-time.

They're not available at the moment, however once the roll-out commences you'll be able to see the graphs here.

Lets hope it stops the latest batch of stock market related spam

Quick heads up that sunmxcore09 is now behind one of the Critical Path boxes. The only thing that it's not doing is tagging the subject line of emails. This means there shouldn't be a change to detection rates as originally announced.

You can monitor the Critical Path graphs here.

This doesn't quite look to be true this morning, although I admit that the sample size is a bit small so I'm not going to go claiming that PN have broken the spam system yet. But I was very surprised to turn on my computer a few minutes ago and find six pieces of spam e-mail since midnight today, whereas I'd normally get about two in a whole day. I can't really spot a pattern; the e-mails have arrived from several sunmxores and there's none of that '0.4000' stuff going on in the Dspam factors. I do note, though, that four of the six e-mails show a 'X-Dspam-Probability' of 0.0000 or 0.0001 so I don't know if that's hinting at anything.

I'm not going to be around for much of the rest of the day to monitor things, so my main purpose in posting this is to encourage other people to check to see if they've also been seeing considerably more spam arrive since the early hours of this morning... if they haven't then I guess it was just a little coincidence; if they have then maybe something is broken after all.

Just a quickie..

You mentioned you deployed sunmxcore-09 *in front* of the CP boxes?  I thought the idea was to do it the other way round?

Also, there are two CP boxes - is the eventual idea to have a pair of CP boxes filtering and feeding each mxcore?

Anyway, I'll be F5-ing the graphs every 15 mins to see how it goes

B.

My bad Barry, I've edited my post.

The arrangement is designed to work as follows:

load balancer > CP Box > load balancer > mail server

Rollback already?

The graphs are fairly fun to watch (ok, ok.  I'm waiting for a large recompilation that usually takes about 25 mins )

Are pih-criticalpath01 and pih-criticalpath02 both "feeding" sunmxcore09 ?

Also, out of interest why are the CP boxes feeding back through the load balancers?  I tried to figure it out from the blog post but I'm assuming it's in case you need to drain the queues on the CP boxes in case of a full rollback?

B.

It looks like they triggered the
rollback condition.

Ah well, better luck next time.

It was rolled back temporarily while some teething problems were ironed out and the queue decreased again (this decreased almost immediately). Some tweaks were made and now the appliance is back on again performing happily.

It's to ensure load across the 22 mail servers is handled efficiently and helps prevent the CP boxes flooding a particular mail delivery server. If there was a CP box for each server then we wouldn't need to do this but that would be overkill.

Well, it certainly looks like it - the queues on cp01 and cp02 are around nil!

* Scuttles off to do some other type of work before getting hooked into refreshing the status graphs *

B.

OT: I'm intrigued by the pattern that can be seen on some of the servers - particularly mxcore 17. Why does that happen?

Below are the headers from a recently missed spam which came through the CP boxes. Is the X-MAA line added by CP? - if so it was right!

X-Daemon-Classification: INNOCENT
Envelope-to: xxx@yyy.plus.com
Delivery-date: Tue, 30 Oct 2007 12:14:57 +0000
Received: from pih-criticalpath01.plus.net ([84.92.7.52] helo=cp3a.criticalpath.priv)
     by pih-sunmxcore09.plus.net with esmtp (PlusNet MXCore v2.00) id 1Impzk-0006H0-JF 
     for xxx@yyy.plus.com; Tue, 30 Oct 2007 12:14:56 +0000
X-MAA: Suspected Spam
Received: from gatov3fiduyiff (87.218.78.168) by cp3a.criticalpath.priv (7.3.118.15)
        id 472061DE001CB741 for xxx@yyy.plus.com; Tue, 30 Oct 2007 12:14:56 +0000
Message-ID: 30fab01c81aee$73890f80$0264641b@gatov3fiduyiff
From: "Aurora Sylvester" <ChrisdemountableMarcum@linksys.com>
To: <xxx@yyy.plus.com>
Subject: stock breaker report
Date: Tue, 30 Oct 2007 13:14:09 -0100
MIME-Version: 1.0
Content-Type: text/plain;
   format=flowed;
   charset="iso-8859-1";
   reply-type=original
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2900.2180
X-MIMEOLE: Produced By Microsoft MimeOLE V6.00.2900.2180
X-PN-Spam-Filtered: by PlusNet MXCore (v4.00)
X-DSPAM-Result: Innocent
X-DSPAM-Processed: Tue Oct 30 12:14:57 2007
X-DSPAM-Confidence: 0.6992
X-DSPAM-Improbability: 1 in 233 chance of being spam
X-DSPAM-Probability: 0.0000
X-DSPAM-Factors: 27,
   Delivery-date*Tue+30, 0.00553,
   cocktail+in, 0.01000,
   single+serving, 0.99000,
   go+If, 0.02530,
   read+their, 0.03021,
   and+bar, 0.03021,
   is+lighter, 0.03021,
   a+50, 0.03419,
   bar, 0.04727,
   Received*30, 0.05150,
   Received*30, 0.05150,
   look+into, 0.06401,
   and+innovative, 0.06852,
   as+there, 0.06933,
   To*<john, 0.07226,
   pouch, 0.07226,
   present+a, 0.07397,
   Whats+the, 0.07577,
   convenient+to, 0.08367,
   an+active, 0.08745,
   the+go, 0.08795,
   Subject*report, 0.09306,
   X-MAA*Suspected, 0.89956,
   X-MAA*Spam, 0.89956,
   X-MAA*Suspected+Spam, 0.89956,
   releases+and, 0.10840,
   portable, 0.11711