gmail.com problems and this list

Marco Barreno barreno at cs.berkeley.edu
Fri Jul 13 23:36:57 CEST 2007


Hi everyone,


Short version:

It seems the duplicated messages are not just a Gmail problem: mails
are also being duplicated from starband.net and possibly others as
well, so it seems the problem is likely something on the openmoko side
that only happens to cause duplicates with some sending domains and
not others.  I'ts also not consistent; some messages from Gmail and
starband.net are duplicated while others aren't.

I've checked with the Gmail team, and the duplicated messages from
Gmail are happening because Gmail doesn't receive the SMTP OK from the
openmoko list server, so it retransmits because it thinks the messages
don't get delivered.  To debug further, an admin on the list machine
needs to enable verbose SMTP logging if it's not already enabled and
provide information about if/when the OK is sent.  Here are some
example message IDs for which the logs would be useful:
  2f3aa2770707070010k7407439axfaa269de33395ea2 at mail.gmail.com
  c914135d0707121458s509d1ee1v3a8fe77101bbcaa2 at mail.gmail.com
  b829084f0707121328q5e4d0f28me015de4502cffb69 at mail.gmail.com
If verbose logging isn't already enabled, however, enabling it and
then sending logs for any future duplicated messages would work too.


Detailed version:

On Mon, Jul 09, 2007 at 11:14:03PM -0700, thus spake Wolfgang S. Rupprecht:
> At least one of the opemoko machines at 88.198.124.203 opens an smtp
> connection back to the sending domain to verify the sender address of
> any incoming message.  The smtp-back machine has messed up DNS.  The
> claimed rDNS for that IP is "openmoko.org" but the forward DNS check
> for that "openmoko.org" doesn't list 88.198.124.203 as a valid
> address.  If the sending machine is checking for spammers claiming a
> bunk DNS name will reject 88.198.124.203's SMTP verify.  The opemmoko
> machine will then interpret that failed smtp verify attempt as the
> verified address not existing and will decline the initial incoming
> transfer.

Even ignoring the messed-up DNS for the moment, doing an SMTP callback
is not a good way to check existence of an email account.  For one
thing, whenever a spammer spoofs a From address in an innocent domain,
that domain can get bombarded with callbacks.  For another thing,
spammers sometimes use RCPT commands themselves to check existence of
email addresses, so ISPs are likely to consider many such requests to
be abusive behavior.  In the case of a large ISP (such as Gmail), if a
lot of email is sent to a destination that does callbacks, the
frequency of callbacks can trigger abuse prevention measures.  See
http://en.wikipedia.org/wiki/Callback_verification for more drawbacks.
(In particular, note: "If a server receives a lot of spam, it will do
a lot of callbacks and if those addresses are invalid, the server will
look very similar to a spammer who is doing a dictionary attack to
harvest addresses. This in turn might get the server blacklisted
elsewhere.")

For these reasons, Gmail strongly recommends avoiding callbacks and
instead checking SPF records (Sender Policy Framework, provided via
DNS) to verify that the mail is indeed coming from Google (i.e. the
SMTP connection is coming from an IP authorized to send email on
behalf of Google).

However, in this case it does not seem that the problem was caused by
the callbacks.  The openmoko server accepted the mail (which it
wouldn't do if the callback failed to verify the sender) and the DATA
command was sent, but Gmail never received the next SMTP OK
acknowledgment before timing out.  So one of four things was
happening:

1) openmoko never sent the OK -- problem on openmoko end
2) openmoko waited too long before sending the OK and SMTP timed out
    -- seems to indicate problem on the openmoko end
3) the OK was lost in transit -- network difficulties, neither side
    at fault
4) the OK was received by Gmail -- problem at the Gmail end

Since at least one other domain is showing the same behavior, it seems
1 or 2 is most likely, but we can't say for sure without more
information.

One interesting fact is that the openmoko server started sending mail
back to Gmail recipients (after list expansion) before Gmail received
the SMTP OK.  Perhaps this means the list server waits until after
list expansion to send the OK?  That could cause it to time out pretty
easily.

It would help diagnosis if someone on the list machine could send the
verbose SMTP logs (if they were recorded).  The logs posted by Harald
Welte don't give enough information: what we really need to see is
whether (and if so, when) the SMTP OK was sent for any of the
failed/duplicated messages.  Here are three example message IDs:
  2f3aa2770707070010k7407439axfaa269de33395ea2 at mail.gmail.com
  c914135d0707121458s509d1ee1v3a8fe77101bbcaa2 at mail.gmail.com
  b829084f0707121328q5e4d0f28me015de4502cffb69 at mail.gmail.com
If the logging needs to have more verbosity enabled, however, the logs
for any duplicated message after such enabling would work just as
well.

By the way, the Received header chains posted by David Ford don't
necessarily show that the problem is within Gmail; there could be any
number of legitimate reasons why the resending isn't originating at
the last hop out of the Google network.  Yes, Gmail is sending the
mail multiple times, but that's because it thinks the openmoko server
did not receive the message on earlier attempts.

I hope someone with admin access to the list machine can help debug
this problem so we can get it fixed, wherever the bug is located.

Thanks,
Marco




More information about the community mailing list