Duplicate message troubleshooting

Andreas Kostyrka andreas at kostyrka.org
Tue Jul 24 22:42:01 CEST 2007


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ok, putting on my Postmaster hat, could you please provide Message Ids
and headers for messages from heaven.kostyrka.org that were sent
duplicate? Please use private mail.

Andreas

Marco Barreno wrote:
> Messages to the list are being duplicated again.  Some of the messages
> are from Gmail, but not all (e.g. some are from kostyrka.org and
> cnlohr.com).  I can try to help debug the problem from the Gmail end
> if an admin on the list machine can send me verbose SMTP logs; please
> see below for details.  Harald or Sean, can one of you send detailed
> logs?  Anyone else?
> 
> Some Gmail message IDs with duplicated messages today:
>   e4b4377c0707240911s5d79faa4s48dea3b00a4418c4 at mail.gmail.com
>   eef811f30707240955i4348c71ai16f337968c714e67 at mail.gmail.com
>   562ec500707241157r4205bba7p919a99eb489552c6 at mail.gmail.com
> 
> Thanks,
> Marco
> 
> 
> ----- Forwarded message from Marco Barreno <barreno at CS.Berkeley.EDU> -----
> 
> From: Marco Barreno <barreno at CS.Berkeley.EDU>
> Date: Fri, 13 Jul 2007 14:36:57 -0700
> To: "Wolfgang S. Rupprecht" <wolfgang.rupprecht+gnus200707 at gmail.com>
> Cc: community at lists.openmoko.org
> Subject: Re: gmail.com problems and this list
> 
> Hi everyone,
> 
> 
> Short version:
> 
> It seems the duplicated messages are not just a Gmail problem: mails
> are also being duplicated from starband.net and possibly others as
> well, so it seems the problem is likely something on the openmoko side
> that only happens to cause duplicates with some sending domains and
> not others.  I'ts also not consistent; some messages from Gmail and
> starband.net are duplicated while others aren't.
> 
> I've checked with the Gmail team, and the duplicated messages from
> Gmail are happening because Gmail doesn't receive the SMTP OK from the
> openmoko list server, so it retransmits because it thinks the messages
> don't get delivered.  To debug further, an admin on the list machine
> needs to enable verbose SMTP logging if it's not already enabled and
> provide information about if/when the OK is sent.  Here are some
> example message IDs for which the logs would be useful:
>   2f3aa2770707070010k7407439axfaa269de33395ea2 at mail.gmail.com
>   c914135d0707121458s509d1ee1v3a8fe77101bbcaa2 at mail.gmail.com
>   b829084f0707121328q5e4d0f28me015de4502cffb69 at mail.gmail.com
> If verbose logging isn't already enabled, however, enabling it and
> then sending logs for any future duplicated messages would work too.
> 
> 
> Detailed version:
> 
> On Mon, Jul 09, 2007 at 11:14:03PM -0700, thus spake Wolfgang S. Rupprecht:
>> At least one of the opemoko machines at 88.198.124.203 opens an smtp
>> connection back to the sending domain to verify the sender address of
>> any incoming message.  The smtp-back machine has messed up DNS.  The
>> claimed rDNS for that IP is "openmoko.org" but the forward DNS check
>> for that "openmoko.org" doesn't list 88.198.124.203 as a valid
>> address.  If the sending machine is checking for spammers claiming a
>> bunk DNS name will reject 88.198.124.203's SMTP verify.  The opemmoko
>> machine will then interpret that failed smtp verify attempt as the
>> verified address not existing and will decline the initial incoming
>> transfer.
> 
> Even ignoring the messed-up DNS for the moment, doing an SMTP callback
> is not a good way to check existence of an email account.  For one
> thing, whenever a spammer spoofs a From address in an innocent domain,
> that domain can get bombarded with callbacks.  For another thing,
> spammers sometimes use RCPT commands themselves to check existence of
> email addresses, so ISPs are likely to consider many such requests to
> be abusive behavior.  In the case of a large ISP (such as Gmail), if a
> lot of email is sent to a destination that does callbacks, the
> frequency of callbacks can trigger abuse prevention measures.  See
> http://en.wikipedia.org/wiki/Callback_verification for more drawbacks.
> (In particular, note: "If a server receives a lot of spam, it will do
> a lot of callbacks and if those addresses are invalid, the server will
> look very similar to a spammer who is doing a dictionary attack to
> harvest addresses. This in turn might get the server blacklisted
> elsewhere.")
> 
> For these reasons, Gmail strongly recommends avoiding callbacks and
> instead checking SPF records (Sender Policy Framework, provided via
> DNS) to verify that the mail is indeed coming from Google (i.e. the
> SMTP connection is coming from an IP authorized to send email on
> behalf of Google).
> 
> However, in this case it does not seem that the problem was caused by
> the callbacks.  The openmoko server accepted the mail (which it
> wouldn't do if the callback failed to verify the sender) and the DATA
> command was sent, but Gmail never received the next SMTP OK
> acknowledgment before timing out.  So one of four things was
> happening:
> 
> 1) openmoko never sent the OK -- problem on openmoko end
> 2) openmoko waited too long before sending the OK and SMTP timed out
>     -- seems to indicate problem on the openmoko end
> 3) the OK was lost in transit -- network difficulties, neither side
>     at fault
> 4) the OK was received by Gmail -- problem at the Gmail end
> 
> Since at least one other domain is showing the same behavior, it seems
> 1 or 2 is most likely, but we can't say for sure without more
> information.
> 
> One interesting fact is that the openmoko server started sending mail
> back to Gmail recipients (after list expansion) before Gmail received
> the SMTP OK.  Perhaps this means the list server waits until after
> list expansion to send the OK?  That could cause it to time out pretty
> easily.
> 
> It would help diagnosis if someone on the list machine could send the
> verbose SMTP logs (if they were recorded).  The logs posted by Harald
> Welte don't give enough information: what we really need to see is
> whether (and if so, when) the SMTP OK was sent for any of the
> failed/duplicated messages.  Here are three example message IDs:
>   2f3aa2770707070010k7407439axfaa269de33395ea2 at mail.gmail.com
>   c914135d0707121458s509d1ee1v3a8fe77101bbcaa2 at mail.gmail.com
>   b829084f0707121328q5e4d0f28me015de4502cffb69 at mail.gmail.com
> If the logging needs to have more verbosity enabled, however, the logs
> for any duplicated message after such enabling would work just as
> well.
> 
> By the way, the Received header chains posted by David Ford don't
> necessarily show that the problem is within Gmail; there could be any
> number of legitimate reasons why the resending isn't originating at
> the last hop out of the Google network.  Yes, Gmail is sending the
> mail multiple times, but that's because it thinks the openmoko server
> did not receive the message on earlier attempts.
> 
> I hope someone with admin access to the list machine can help debug
> this problem so we can get it fixed, wherever the bug is located.
> 
> Thanks,
> Marco
> 
> _______________________________________________
> OpenMoko community mailing list
> community at lists.openmoko.org
> http://lists.openmoko.org/mailman/listinfo/community
> 
> ----- End forwarded message -----
> 
> _______________________________________________
> OpenMoko community mailing list
> community at lists.openmoko.org
> http://lists.openmoko.org/mailman/listinfo/community
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGpmQZHJdudm4KnO0RAkh0AKCGoyvG0MxqjbSj4iMvBOv196wR1gCeIkBZ
JOzTSdqXaDakc3qP9Z3fTlQ=
=+pFD
-----END PGP SIGNATURE-----




More information about the community mailing list