E-mail delays
Morning all,
Apologies for the delay getting back with an explanation.
It looks like there are two separate issues here.
The first issue is due to work that was conducted yesterday. Apologies ithat this was not announced via planned maintenance however the Comms team were not made aware that this had been done until today.
Turns out that one of the Critical Path boxes was deployed alongside sunmxcore18 for an hour or so yesterday (between 2:00pm-3:00pm). This was done for debugging purposes (there were a couple of engineers from Critical Path here).
Unfortunately this has resulted in some of the email destined for sunmxcore18 (approximately 1 in 11 emails will have gone to this server) to become delayed
It seems that the CP box accumulated a queue of around 150,000 emails during it's hour on the network. We've been investigating why this happened. These messages have been dequeuing at a rate of about 1500-3500 emails per hour - This should be a lot quicker and (overnight especially) all email should have been dequeued. Sadly it hasn't been and there are still 20,000 emails or so on the box. Most of this will be spam but there are likely to be genuine emails in the midst so chances are there'll be more reports of delays. We anticipate that the queues will have cleared within a couple of hours.
Whilst the correct change control/peer review avenues were followed, omitting to notify Comms meant that we weren't pro-active in our messaging and no planned maintenance went out. Formal changes have been proposed for the Change Control procedure to always involve the Comms Team.
The second issue (The one that the OP experienced) has been raised internally as problem 46136 and relates to delays on the incoming mx.core mail delivery servers. Initial /suspicions/ are that this is something to do with one of the spam processes not retrying if it fails to run first time round (normally when it's busy). We're still looking into this but thankfully reports don't appear to be too widespread.
Sorry for the inconvenience being caused by these issues ans for the lack of notification. I'll update Service Status and change the traffic light to amber.
Apologies for the delay getting back with an explanation.
It looks like there are two separate issues here.
The first issue is due to work that was conducted yesterday. Apologies ithat this was not announced via planned maintenance however the Comms team were not made aware that this had been done until today.
Turns out that one of the Critical Path boxes was deployed alongside sunmxcore18 for an hour or so yesterday (between 2:00pm-3:00pm). This was done for debugging purposes (there were a couple of engineers from Critical Path here).
Unfortunately this has resulted in some of the email destined for sunmxcore18 (approximately 1 in 11 emails will have gone to this server) to become delayed
It seems that the CP box accumulated a queue of around 150,000 emails during it's hour on the network. We've been investigating why this happened. These messages have been dequeuing at a rate of about 1500-3500 emails per hour - This should be a lot quicker and (overnight especially) all email should have been dequeued. Sadly it hasn't been and there are still 20,000 emails or so on the box. Most of this will be spam but there are likely to be genuine emails in the midst so chances are there'll be more reports of delays. We anticipate that the queues will have cleared within a couple of hours.
Whilst the correct change control/peer review avenues were followed, omitting to notify Comms meant that we weren't pro-active in our messaging and no planned maintenance went out. Formal changes have been proposed for the Change Control procedure to always involve the Comms Team.
The second issue (The one that the OP experienced) has been raised internally as problem 46136 and relates to delays on the incoming mx.core mail delivery servers. Initial /suspicions/ are that this is something to do with one of the spam processes not retrying if it fails to run first time round (normally when it's busy). We're still looking into this but thankfully reports don't appear to be too widespread.
Sorry for the inconvenience being caused by these issues ans for the lack of notification. I'll update Service Status and change the traffic light to amber.
Poacher turned Gamekeeper - or is that the other way round 