Route Loops causing delivery failure when using own SMTP server

[#1]

Hi, I have mail delivered to my own SMTP server, which isn't online all the time. This has never normally been a problem.
I'm getting mail frequently bounced before it even gets to my mail server (e.g. if it's off) - the mail headers indicate it's bouncing around inside PlusNet's mail servers.
Is anyone else having this problem?
I raised a support issue on the 28th, and this has been escalated to gain a ticket ID on the 6th, but doesn't appear to be being looked at.

Matt
e.g. [mail addresses redacted]
SMTP error from remote mail server after end of data:
  host relay.plus.net [212.159.8.107]: 550 To many recieved headers. Rejected due to route loop.
------ This is a copy of the message, including all the headers. ------
Return-path: <a.user@gmail.com>
Received: from [212.159.7.35] (helo=relay.ptn-ipout01.plus.net)
        by pih-inmx05.plus.net with esmtp (PlusNet MXCore v2.00) id 1OWmbW-00003f-79
        for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:37:10 +0100
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: Aq8EAAsrNUzUnw4S/2dsb2JhbACBQ55Sea4fggyFei6IUwEBAwWFIASIPw
Received: from outmx04.plus.net (HELO outmx06.plus.net) ([212.159.14.18])
by relay.ptn-ipout01.plus.net with ESMTP; 08 Jul 2010 09:36:10 +0100
Received: from pih-inmx05.plus.net ([212.159.10.6])
        by outmx06.plus.net with esmtp (Exim) id 1OWmaX-0000qJ-KQ
      for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:36:09 +0100
Received: from [212.159.7.99] (helo=relay.pcl-ipout01.plus.net)
        by pih-inmx05.plus.net with esmtp (PlusNet MXCore v2.00) id 1OWmaX-0007T4-Ay
        for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:36:09 +0100
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AhAFANgqNUzUnw4R/2dsb2JhbACBQ55SCHGuGoIMhXouiFMBAQMFhSAEiD8
Received: from outmx02.plus.net ([212.159.14.17])
by relay.pcl-ipout01.plus.net with ESMTP; 08 Jul 2010 09:35:09 +0100
Received: from pih-inmx03.plus.net ([212.159.10.4])
        by outmx02.plus.net with esmtp (Exim) id 1OWmZY-0000dk-Ml
      for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:35:08 +0100
Received: from [212.159.7.35] (helo=relay.ptn-ipout01.plus.net)
        by pih-inmx03.plus.net with esmtp (PlusNet MXCore v2.00) id 1OWmZY-00023d-DX
        for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:35:08 +0100
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AhAFAN4pNUxUXeb6/2dsb2JhbACBQ55QCHGuKIIMhXkuiFMBAQMFhSAEiD8
Received: from outmx05.plus.net ([84.93.230.250])
by relay.ptn-ipout01.plus.net with ESMTP; 08 Jul 2010 09:34:08 +0100
Received: from inmx22.plus.net ([84.93.229.196])
        by outmx05.plus.net with esmtp (Exim) id 1OWmYZ-0000j7-Lv
      for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:34:07 +0100
Received: from [212.159.7.99] (helo=relay.pcl-ipout01.plus.net)
        by inmx22.plus.net with esmtp (PlusNet MXCore v2.00) id 1OWmYZ-00039t-Bo
        for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:34:07 +0100
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: AhAFAKMpNUxUXebr/2dsb2JhbACBQ55QCHGuKIIMhXkuiFMBAQMFhSAEiD8
Received: from outmx07.plus.net ([84.93.230.235])
by relay.pcl-ipout01.plus.net with ESMTP; 08 Jul 2010 09:33:07 +0100
Received: from inmx06.plus.net ([84.93.229.130])
        by outmx07.plus.net with esmtp (Exim) id 1OWmXa-0006hm-Ko
      for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:33:06 +0100
Received: from [74.125.82.179] (helo=mail-wy0-f179.google.com)
        by inmx06.plus.net with esmtp (PlusNet MXCore v2.00) id 1OWmXa-0002Mc-C2
        for matt@mydomain.me.uk; Thu, 08 Jul 2010 09:33:06 +0100
Received: by wyf19 with SMTP id 19so324955wyf.10
      for <matt@mydomain.me.uk>; Thu, 08 Jul 2010 01:32:06 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
      d=gmail.com; s=gamma;
      h=domainkey-signature:mime-version:received:received:date:message-id
        :subject:from:to:content-type;
      bh=EERsaiwWnBWID5jew7egDylH3ov3tfLJJk26o6963W0=;
      b=YklCw9f++vVlcsukzWVOyq7q+6U6LLSDkyOCNL4HB06/5dxv77Fsut9cdkdfJtNiBH
        g15kCICSSEp9KvbAayBGt5dad+ez7+eiyxOed2z6EviBNksGNoI8BxDxq5lZXN/8mzjZ
        qxEyFcB5GU41ZvOfeU/Zq5buylGHI/WQDLwm4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
      d=gmail.com; s=gamma;
      h=mime-version:date:message-id:subject:from:to:content-type;
      b=V25v3GRXi7RKCJK73JMz/GtJ31GnYM+EXdoyOPvKxfhM8K8vwmyffGak02ms3Qa2Ni
        acXyJfckh+kvZYqW3y3lrMsYtRjBiwnpnqxQ6mNIFAEmn0dWTnoTE6FUt7qwZPcK3R0L
        ThSBNKJxjl+GYidKbbTdE5LSr8YixBe99y0w0=
MIME-Version: 1.0
Received: by 10.216.236.146 with SMTP id w18mr2202469weq.19.1278577925982;
      Thu, 08 Jul 2010 01:32:05 -0700 (PDT)
Received: by 10.216.20.83 with HTTP; Thu, 8 Jul 2010 01:32:05 -0700 (PDT)
Date: Thu, 8 Jul 2010 09:32:05 +0100
Message-ID: <AANLkTikCuis9mRtTHC3AP1tey3JGlZi5nKEC3rVenz_0@mail.gmail.com>
From: John Doe <a.user@gmail.com>
To: Me <matt@mydomain.me.uk>
Content-Type: multipart/alternative; boundary=000e0cd5272e188c92048adc1fd1
X-PN-Virus-Filtered: by PlusNet MXCore (v5.00)

[#2]

I thought all mail that was direct delivered used the autoturn system.  The Ironports shouldn't get involved at all.
I would check your MX records for the domain.  They should look similar to:

domain.com.            14400  IN      MX      5 mail.domain.com.
domain.com.            14400  IN      MX      10 autoturn.plus.net.uk.
domain.com.            14400  IN      MX      20 mx.last.plus.net.
It sounds like your domain records may refer to:
domain.com.            14400  IN      MX      10 mx-ironport.core.plus.net.
domain.com.            14400  IN      MX      20 mx-ironport.last.plus.net.
Which means when your server is offline, it's being relayed through the Ironports (which is incorrect)
Cheers
B.

[#3]

Hi Barry,
Many thanks for your swift reply, I thought it would be something similar to this, but my MX records on both ns1.force9.net and ns2.force9.net are as they should be, not mentioning the ironport servers. They are as you suggest:
5: mydomain.me.uk
10: autoturn.plus.net.uk
20: mx.last.plus.net.
Could anyone from PlusNet comment on this please?
Thanks,
Matt

[#4]

I'll see if I can nudgette one of the relevant peeps to have a look.
Cheers
B.

[#5]

Wonder if it's anything to do with this, will investigate further and see what I can find out...
Edit: reckon it is related. The mail is attempting to get sent to your server which is offline/unavailable. It's then trying autoturn I think but isn't able to connect to it because it's busy/not available, it then tries the mxlast record which accepts the mail and tries delivering to your server again. Your server is then either still offline or you have it configured to bounce/reject mail from mxlast. The change I linked to altered the way we deal with bounced emails, so the rejected/bounced message then goes to ipouts via the outmx servers and the process gets repeated until it gives up. If you've no aversion to it then I think you might be able to sort this out by removing the mx.last record.

[#6]

Barry, thanks for your nudge!
Bob, many thanks for looking into this!
I haven't configured my server to bounce/reject from anyone, unless there's something obviously wrong with the mail (as you'd typically see when spammers try to send to you). In this case, my server is offline.
I'm unsure whether removing the mx.last record is going to help - if my server is down or the ADSL connection drops (I have it set to connect on demand), the sender would try autoturn, which you think may be busy; in this case, would the sender give up and return the mail as undeliverable? (Since there's no further MTA in the priority chain?)
Is autoturn regularly busy/not available?
I presume ipouts/outmx are the mail servers that deal with IronPort filtering - since I chose to have SMTP delivery, should the delivery retry be attempted to autoturn?
Are other direct-SMTP users experiencing similar problems? (Just wondering: how many users do you have that use their own SMTP servers?)
Thanks,
Matt

[#7]

Quote from: mattgumbley

I'm unsure whether removing the mx.last record is going to help - if my server is down or the ADSL connection drops (I have it set to connect on demand), the sender would try autoturn, which you think may be busy; in this case, would the sender give up and return the mail as undeliverable? (Since there's no further MTA in the priority chain?)

Depends on the configuration of the sending server. Most should just defer the message and try again at predefined intervals until the email is delivered or the server gives up and returns the message to the sender. You'd like to think that either your server or autoturn would be available for one of these retry attempts though.
I must admit, I'm a little miffed that you're running your own SMTP server when your connection isn't up half of the time, it seems massively impractical to me?

Quote

Is autoturn regularly busy/not available?

Yes it is.

Quote

I presume ipouts/outmx are the mail servers that deal with IronPort filtering...

No. The outmx servers are customer MTA's that we've built ourselves. The ipouts are IronPort appliances however they do very little spam filtering, most of that is handled by the ipins.

Quote

... since I chose to have SMTP delivery, should the delivery retry be attempted to autoturn?

It may well be trying to do this but failing due to how busy the server is.

Quote

Are other direct-SMTP users experiencing similar problems? (Just wondering: how many users do you have that use their own SMTP servers?)

Not 100% sure about this, but this is the first query of this nature that I've seen since the changes we made.

[#8]

Most 'sane' sending MTAs will retry at regular intervals.  I've seen 4min, 10min, 1 hour, 4 hours, 8hours, 16hours and all the way up to around 7 day defers (so it will try each of these in turn) although it really is down to the way the sending MTA is configured, as Bob says.
B.

[#9]

Many thanks for your suggestions - if you don't mind, I'd like to hammer out ideas for moving forward here on the forum, then place the appropriate requests for configuration changes via the support system...
It's not so impractical, it just allows me to control users and aliases easily and locally (I have many aliases: different companies requiring sign up get a new one, when they sell the address to spammers, I close it down), and to back up my mail (seem to recall a certain ISP losing large quantities of mail stored on their mailboxes?).
I've had this kid of mail system since using Demon Internet around '96, I'm just a grizzled UNIX guy, and it's What We Do. 🙂
However, I'm wondering whether it'd be simpler to switch to IMAP. (And probably pull down from PlusNet IMAP mailboxes and push through my SMTP server via fetchmail, so my users have all their mail in the same place - mostly for backup purposes.)
I have some questions about this though:
Been reading through http://www.plus.net/support/email/email_explained.shtml.
1. Number of mailboxes: Aliases won't be a problem; I can see I can have as many as necessary, but how many mailboxes can I set up?
2. Control over domain mailboxes/aliases/catch-all: I don't use the mydomain.plus.com mail, since the Great Webmail Compromise, but I can see from the config pages how mailboxes, aliases, catch all etc. are configured for that.
If I switch to IMAP for my domain mail, are the mailboxes, aliases etc configured in exactly the same way, but for the domain, or do addresses at my domain map straight through to the addresses at plus.com (meaning I'd have to reactivate the mydomain.plus.com).
i.e. do I have control over the mailboxes and aliases @mydomain.me.uk separately, or do I control them via the @mydomain,plus.com settings?
3. Spam. Since the Great Webmail Compromise, I've had to install spamassassin inline locally. This would still be used, as fetchmail would deliver to my local SMTP. However, would mail coming in via IMAP be processed by PlusNet's Spam filters?
Many thanks,
Matt

[#10]

Quote from: mattgumbley

1. Number of mailboxes: Aliases won't be a problem; I can see I can have as many as necessary, but how many mailboxes can I set up?

There's no limit.

Quote from: mattgumbley

2. Control over domain mailboxes/aliases/catch-all: I don't use the mydomain.plus.com mail, since the Great Webmail Compromise, but I can see from the config pages how mailboxes, aliases, catch all etc. are configured for that. If I switch to IMAP for my domain mail, are the mailboxes, aliases etc configured in exactly the same way, but for the domain, or do addresses at my domain map straight through to the addresses at plus.com (meaning I'd have to reactivate the mydomain.plus.com).

The mailbox setup is a global thing and applies to *all* domains hosted against your account, including the username.plus.com virtual domain. You can switch off mail delivery for these domains independently, but if they're all switched on then email to localpart@username.plus.com would go to the same mailbox as messages sent to localpart@yourdomain.tld.

Quote from: mattgumbley

i.e. do I have control over the mailboxes and aliases @mydomain.me.uk separately, or do I control them via the @mydomain,plus.com settings?

As above, they're controlled via the username.plus.com settings.

Quote from: mattgumbley

3. Spam. Since the Great Webmail Compromise, I've had to install spamassassin inline locally. This would still be used, as fetchmail would deliver to my local SMTP. However, would mail coming in via IMAP be processed by PlusNet's Spam filters?

That depends on whether or not you have our spam filtering switched on or off.
By the way, we reckon we can fix the route loop issue, we're going to roll out a patch to each of the delivery servers over the coming days...

[#11]

Many thanks for the info Bob, and thanks for continuing to look into the route loop problem.
I was thinking I'd switch over to pulling mail down via IMAP or POP3 using fetchmail - given that I'm not connected permanently.
However, do you have an ETA for your patch? What changes are you considering? Do you need help in testing it after it's applied (i.e. should I hold off my IMAP changeover?)
Thanks,
Matt

[#12]

I'd be very careful using Fetchmail, if you're considering using 'multi-drop' mode
What fetchmail usually does is retrieve mail from a remote server, then deliver it to the local MTA for local delivery (or relaying).
The problem will occur with mails from mailing lists, where the 'to' field will usually be the mailing list.  When the initial SMTP transaction occurs, the mail is directed towards the correct recipient.  However, when you process via Fetchmail, often the correct deliver-to recipient is no longer available - thus you end up with undeliverable mail.
Read 'man fetchmail' very very carefully.
If you're using it in 'single drop' mode, then it *should* work fine. 
However, removing the mx.last record should (as Bob says above) resolve the issue - at least temporarily.
Cheers
B.

[#13]

We haven't had a change control request through yet so we don't have an eta or a confirmed program of works I'm afraid. We will have a comprehensive testing procedure scheduled too, but customer feedback is always appreciated!

[#14]

Barry,
Many thanks, I hadn't considered that - I was going to use it in single-drop mode, for each mailbox in turn.
Ben, OK thanks, I'll remove the 20 mx.last.plus.net MX record for now via the web UI, and run some tests after it has propagated.
Thanks,
Matt

[#15]

The route loop fix is scheduled to be rolled live to a single server later today. We'll then do the remaining servers at one site on Thursday, and the servers at the second site next Tuesday (20th July).
Edit: The route loop fix was rolled to a single server this morning (didn't get time yesterday), and looks to be serving it's purpose. All being well the remaining servers at the first of the two Sheffield sites will be done tomorrow (15th July).