cancel
Showing results for 
Search instead for 
Did you mean: 

Critical Path Anti-spam Maintenance

bobpullen
Community Gaffer
Community Gaffer
Posts: 16,887
Thanks: 4,979
Fixes: 316
Registered: ‎04-04-2007

Critical Path Anti-spam Maintenance

Evening all,
A quick heads up that I've just posted the following Service Status and blog posts to the Usertools Site and Comms blog regarding a significant piece of work we'll be carrying out on the mail platform next week.
Basically, we will be deploying our Critical Path anti-spam solution in front of the live customer servers again. I'll not go into too much detail here at a risk of repeating myself, but I'm sure I'm right in assuming that some of you may be slightly concerned by this prospect - Especially given the email delays/loss that occurred last time.
We're fully confident we've covered all eventualities this time round and have a strict roll-back plan which can be implemented quickly at the sign of any trouble. I've spent some time explaining the rationale behind this decision, and have attempted to allay many of the concerns customers may have in the Service Status and blog post I have made.
If you have any feedback, concerns, questions or worries about this work then please feel free to post them here and we'll do our best to address them. I'd strongly suggest reading the infromation I have linked to before doing so as there's a good chance I may have already covered the question you're wanting to ask.
There's always going to be a risk associated with any work we do, however we feel that we're very well prepared for this work and have gone a long way to ensuring that all eventuatlities we can think of have been covered.
Needless to say, many of you have been very helpful in the past in quickly identifying problems both during and following a roll-out and we'll especially be looking for your feedback during and after this work. Those involved with the roll-out will be monitoring this thread closely and we'll be on hand to quickly identify and react to any issues should they occur.
The benefits of this work should it be successful are very promising and this should hopefully see a large step towards overcoming the reoccurring issues (notably email delays) that keep on popping up to bite us.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

70 REPLIES 70
zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Critical Path Anti-spam Maintenance

Incredibly verbose description Bob, for which I thank you.
At least by posting this then the community as a whole will be able to assist in any issues that may arise.
I look forward to seeing the performance graphs over the coming weeks to see how the new configuration performs in a live environment.
Good luck! Wink
B.
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,887
Thanks: 4,979
Fixes: 316
Registered: ‎04-04-2007

Re: Critical Path Anti-spam Maintenance

Quote from: Barry
Incredibly verbose description Bob, for which I thank you.

You're welcome!
Quote
At least by posting this then the community as a whole will be able to assist in any issues that may arise.

Which is certainly the idea.
Quote
I look forward to seeing the performance graphs over the coming weeks to see how the new configuration performs in a live environment.
Good luck! Wink

Thanks, we moved two central pipes the other day without hitch so here's hoping! Wink

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

bobpullen
Community Gaffer
Community Gaffer
Posts: 16,887
Thanks: 4,979
Fixes: 316
Registered: ‎04-04-2007

Re: Critical Path Anti-spam Maintenance

Subtle reminder that this work will be commencing tomorrow morning.
After the first server goes live you'll be able to see graphs showing the latency and queues on the Critical Path devices compared to the queues on the sunmxcores. This will allow you to see exactly what's happening in real-time.
They're not available at the moment, however once the roll-out commences you'll be able to see the graphs here.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

Oldjim
Resting Legend
Posts: 38,460
Thanks: 787
Fixes: 63
Registered: ‎15-06-2007

Re: Critical Path Anti-spam Maintenance

Lets hope it stops the latest batch of stock market related spam
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,887
Thanks: 4,979
Fixes: 316
Registered: ‎04-04-2007

Re: Critical Path Anti-spam Maintenance

Quick heads up that sunmxcore09 is now behind one of the Critical Path boxes. The only thing that it's not doing is tagging the subject line of emails. This means there shouldn't be a change to detection rates as originally announced.
You can monitor the Critical Path graphs here.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

Be3G
Grafter
Posts: 6,111
Thanks: 1
Registered: ‎05-04-2007

Re: Critical Path Anti-spam Maintenance

Quote from: Bob
This means there shouldn't be a change to detection rates as originally announced.

This doesn't quite look to be true this morning, although I admit that the sample size is a bit small so I'm not going to go claiming that PN have broken the spam system yet. But I was very surprised to turn on my computer a few minutes ago and find six pieces of spam e-mail since midnight today, whereas I'd normally get about two in a whole day. I can't really spot a pattern; the e-mails have arrived from several sunmxores and there's none of that '0.4000' stuff going on in the Dspam factors. I do note, though, that four of the six e-mails show a 'X-Dspam-Probability' of 0.0000 or 0.0001 so I don't know if that's hinting at anything.
I'm not going to be around for much of the rest of the day to monitor things, so my main purpose in posting this is to encourage other people to check to see if they've also been seeing considerably more spam arrive since the early hours of this morning... if they haven't then I guess it was just a little coincidence; if they have then maybe something is broken after all.
zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Critical Path Anti-spam Maintenance

Just a quickie..
You mentioned you deployed sunmxcore-09 *in front* of the CP boxes?  I thought the idea was to do it the other way round?
Also, there are two CP boxes - is the eventual idea to have a pair of CP boxes filtering and feeding each mxcore?
Anyway, I'll be F5-ing the graphs every 15 mins to see how it goes Wink
B.
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,887
Thanks: 4,979
Fixes: 316
Registered: ‎04-04-2007

Re: Critical Path Anti-spam Maintenance

My bad Barry, I've edited my post.
The arrangement is designed to work as follows:
load balancer > CP Box > load balancer > mail server

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Critical Path Anti-spam Maintenance

Rollback already? Wink
The graphs are fairly fun to watch (ok, ok.  I'm waiting for a large recompilation that usually takes about 25 mins :P)
Are pih-criticalpath01 and pih-criticalpath02 both "feeding" sunmxcore09 ?
Also, out of interest why are the CP boxes feeding back through the load balancers?  I tried to figure it out from the blog post but I'm assuming it's in case you need to drain the queues on the CP boxes in case of a full rollback?
B.
gofaster
Rising Star
Posts: 369
Thanks: 16
Registered: ‎01-08-2007

Re: Critical Path Anti-spam Maintenance

It looks like they triggered the
Quote
The pending queue on the Critical Path appliance is greater than 10,000, and increases by more than 1,000 every 5 minutes.

rollback condition.
Ah well, better luck next time. Roll_eyes
Chris
Legend
Posts: 17,724
Thanks: 600
Fixes: 169
Registered: ‎05-04-2007

Re: Critical Path Anti-spam Maintenance

It was rolled back temporarily while some teething problems were ironed out and the queue decreased again (this decreased almost immediately). Some tweaks were made and now the appliance is back on again performing happily.
Former Plusnet Staff member. Posts after 31st Jan 2020 are not on behalf of Plusnet.
bobpullen
Community Gaffer
Community Gaffer
Posts: 16,887
Thanks: 4,979
Fixes: 316
Registered: ‎04-04-2007

Re: Critical Path Anti-spam Maintenance

Quote from: Barry
Also, out of interest why are the CP boxes feeding back through the load balancers?  I tried to figure it out from the blog post but I'm assuming it's in case you need to drain the queues on the CP boxes in case of a full rollback?

It's to ensure load across the 22 mail servers is handled efficiently and helps prevent the CP boxes flooding a particular mail delivery server. If there was a CP box for each server then we wouldn't need to do this but that would be overkill.

Bob Pullen
Plusnet Product Team
If I've been helpful then please give thanks ⤵

zubel
Community Veteran
Posts: 3,793
Thanks: 4
Registered: ‎08-06-2007

Re: Critical Path Anti-spam Maintenance

Quote from: Chris
performing happily.

Well, it certainly looks like it - the queues on cp01 and cp02 are around nil!
* Scuttles off to do some other type of work before getting hooked into refreshing the status graphs *
B.
jelv
Seasoned Hero
Posts: 26,785
Thanks: 971
Fixes: 10
Registered: ‎10-04-2007

Re: Critical Path Anti-spam Maintenance

OT: I'm intrigued by the pattern that can be seen on some of the servers - particularly mxcore 17. Why does that happen?
jelv (a.k.a Spoon Whittler)
   Why I have left Plusnet (warning: long post!)   
Broadband: Andrews & Arnold Home::1 (FTTC 80/20)
Line rental: Pulse 8 Home Line Rental (£14.40/month)
Mobile: iD mobile (£4/month)