Duplicate message troubleshooting

Andreas Kostyrka andreas at kostyrka.org
Tue Jul 24 22:42:01 CEST 2007

Hash: SHA1

Ok, putting on my Postmaster hat, could you please provide Message Ids
and headers for messages from heaven.kostyrka.org that were sent
duplicate? Please use private mail.


Marco Barreno wrote:
> Messages to the list are being duplicated again.  Some of the messages
> are from Gmail, but not all (e.g. some are from kostyrka.org and
> cnlohr.com).  I can try to help debug the problem from the Gmail end
> if an admin on the list machine can send me verbose SMTP logs; please
> see below for details.  Harald or Sean, can one of you send detailed
> logs?  Anyone else?
> Some Gmail message IDs with duplicated messages today:
>   e4b4377c0707240911s5d79faa4s48dea3b00a4418c4 at mail.gmail.com
>   eef811f30707240955i4348c71ai16f337968c714e67 at mail.gmail.com
>   562ec500707241157r4205bba7p919a99eb489552c6 at mail.gmail.com
> Thanks,
> Marco
> ----- Forwarded message from Marco Barreno <barreno at CS.Berkeley.EDU> -----
> From: Marco Barreno <barreno at CS.Berkeley.EDU>
> Date: Fri, 13 Jul 2007 14:36:57 -0700
> To: "Wolfgang S. Rupprecht" <wolfgang.rupprecht+gnus200707 at gmail.com>
> Cc: community at lists.openmoko.org
> Subject: Re: gmail.com problems and this list
> Hi everyone,
> Short version:
> It seems the duplicated messages are not just a Gmail problem: mails
> are also being duplicated from starband.net and possibly others as
> well, so it seems the problem is likely something on the openmoko side
> that only happens to cause duplicates with some sending domains and
> not others.  I'ts also not consistent; some messages from Gmail and
> starband.net are duplicated while others aren't.
> I've checked with the Gmail team, and the duplicated messages from
> Gmail are happening because Gmail doesn't receive the SMTP OK from the
> openmoko list server, so it retransmits because it thinks the messages
> don't get delivered.  To debug further, an admin on the list machine
> needs to enable verbose SMTP logging if it's not already enabled and
> provide information about if/when the OK is sent.  Here are some
> example message IDs for which the logs would be useful:
>   2f3aa2770707070010k7407439axfaa269de33395ea2 at mail.gmail.com
>   c914135d0707121458s509d1ee1v3a8fe77101bbcaa2 at mail.gmail.com
>   b829084f0707121328q5e4d0f28me015de4502cffb69 at mail.gmail.com
> If verbose logging isn't already enabled, however, enabling it and
> then sending logs for any future duplicated messages would work too.
> Detailed version:
> On Mon, Jul 09, 2007 at 11:14:03PM -0700, thus spake Wolfgang S. Rupprecht:
>> At least one of the opemoko machines at opens an smtp
>> connection back to the sending domain to verify the sender address of
>> any incoming message.  The smtp-back machine has messed up DNS.  The
>> claimed rDNS for that IP is "openmoko.org" but the forward DNS check
>> for that "openmoko.org" doesn't list as a valid
>> address.  If the sending machine is checking for spammers claiming a
>> bunk DNS name will reject's SMTP verify.  The opemmoko
>> machine will then interpret that failed smtp verify attempt as the
>> verified address not existing and will decline the initial incoming
>> transfer.
> Even ignoring the messed-up DNS for the moment, doing an SMTP callback
> is not a good way to check existence of an email account.  For one
> thing, whenever a spammer spoofs a From address in an innocent domain,
> that domain can get bombarded with callbacks.  For another thing,
> spammers sometimes use RCPT commands themselves to check existence of
> email addresses, so ISPs are likely to consider many such requests to
> be abusive behavior.  In the case of a large ISP (such as Gmail), if a
> lot of email is sent to a destination that does callbacks, the
> frequency of callbacks can trigger abuse prevention measures.  See
> http://en.wikipedia.org/wiki/Callback_verification for more drawbacks.
> (In particular, note: "If a server receives a lot of spam, it will do
> a lot of callbacks and if those addresses are invalid, the server will
> look very similar to a spammer who is doing a dictionary attack to
> harvest addresses. This in turn might get the server blacklisted
> elsewhere.")
> For these reasons, Gmail strongly recommends avoiding callbacks and
> instead checking SPF records (Sender Policy Framework, provided via
> DNS) to verify that the mail is indeed coming from Google (i.e. the
> SMTP connection is coming from an IP authorized to send email on
> behalf of Google).
> However, in this case it does not seem that the problem was caused by
> the callbacks.  The openmoko server accepted the mail (which it
> wouldn't do if the callback failed to verify the sender) and the DATA
> command was sent, but Gmail never received the next SMTP OK
> acknowledgment before timing out.  So one of four things was
> happening:
> 1) openmoko never sent the OK -- problem on openmoko end
> 2) openmoko waited too long before sending the OK and SMTP timed out
>     -- seems to indicate problem on the openmoko end
> 3) the OK was lost in transit -- network difficulties, neither side
>     at fault
> 4) the OK was received by Gmail -- problem at the Gmail end
> Since at least one other domain is showing the same behavior, it seems
> 1 or 2 is most likely, but we can't say for sure without more
> information.
> One interesting fact is that the openmoko server started sending mail
> back to Gmail recipients (after list expansion) before Gmail received
> the SMTP OK.  Perhaps this means the list server waits until after
> list expansion to send the OK?  That could cause it to time out pretty
> easily.
> It would help diagnosis if someone on the list machine could send the
> verbose SMTP logs (if they were recorded).  The logs posted by Harald
> Welte don't give enough information: what we really need to see is
> whether (and if so, when) the SMTP OK was sent for any of the
> failed/duplicated messages.  Here are three example message IDs:
>   2f3aa2770707070010k7407439axfaa269de33395ea2 at mail.gmail.com
>   c914135d0707121458s509d1ee1v3a8fe77101bbcaa2 at mail.gmail.com
>   b829084f0707121328q5e4d0f28me015de4502cffb69 at mail.gmail.com
> If the logging needs to have more verbosity enabled, however, the logs
> for any duplicated message after such enabling would work just as
> well.
> By the way, the Received header chains posted by David Ford don't
> necessarily show that the problem is within Gmail; there could be any
> number of legitimate reasons why the resending isn't originating at
> the last hop out of the Google network.  Yes, Gmail is sending the
> mail multiple times, but that's because it thinks the openmoko server
> did not receive the message on earlier attempts.
> I hope someone with admin access to the list machine can help debug
> this problem so we can get it fixed, wherever the bug is located.
> Thanks,
> Marco
> _______________________________________________
> OpenMoko community mailing list
> community at lists.openmoko.org
> http://lists.openmoko.org/mailman/listinfo/community
> ----- End forwarded message -----
> _______________________________________________
> OpenMoko community mailing list
> community at lists.openmoko.org
> http://lists.openmoko.org/mailman/listinfo/community
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


More information about the community mailing list