Duplicate message troubleshooting

Marco Barreno barreno at cs.berkeley.edu
Thu Aug 16 20:36:05 CEST 2007 

Good idea.  I'll do that later today if I don't get a direct response
from an admin.

Marco


On Thu, Aug 16, 2007 at 08:25:42PM +0200, thus spake Santiago Crespo:
> Hi Marco,
> 
> I think that the best way to solve this is opening a bug in the bugzilla.openmoko.org system, select "Site Infraestructure" and then "Lists":
> 
> http://bugzilla.openmoko.org/cgi-bin/bugzilla/enter_bug.cgi?product=Site%20Infrastructure
> 
> Nobody has reported this yet:
> 
> http://bugzilla.openmoko.org/cgi-bin/bugzilla/buglist.cgi?short_desc_type=allwordssubstr&short_desc=&product=Site+Infrastructure&component=Lists&long_desc_type=allwordssubstr&long_desc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&bug_status=RESOLVED&bug_status=VERIFIED&bug_status=CLOSED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&votes=&changedin=&chfieldfrom=&chfieldto=Now&chfieldvalue=&cmdtype=doit&newqueryname=&order=Reuse+same+sort+as+last+time&field0-0-0=noop&type0-0-0=noop&value0-0-0=
> 
> Thanks!
> 
> 
> 
> El mar, 24-07-2007 a las 13:14 -0700, Marco Barreno escribi??:
> > Messages to the list are being duplicated again.  Some of the messages
> > are from Gmail, but not all (e.g. some are from kostyrka.org and
> > cnlohr.com).  I can try to help debug the problem from the Gmail end
> > if an admin on the list machine can send me verbose SMTP logs; please
> > see below for details.  Harald or Sean, can one of you send detailed
> > logs?  Anyone else?
> > 
> > Some Gmail message IDs with duplicated messages today:
> >   e4b4377c0707240911s5d79faa4s48dea3b00a4418c4 at mail.gmail.com
> >   eef811f30707240955i4348c71ai16f337968c714e67 at mail.gmail.com
> >   562ec500707241157r4205bba7p919a99eb489552c6 at mail.gmail.com
> > 
> > Thanks,
> > Marco
> > 
> > 
> > ----- Forwarded message from Marco Barreno <barreno at CS.Berkeley.EDU> -----
> > 
> > From: Marco Barreno <barreno at CS.Berkeley.EDU>
> > Date: Fri, 13 Jul 2007 14:36:57 -0700
> > To: "Wolfgang S. Rupprecht" <wolfgang.rupprecht+gnus200707 at gmail.com>
> > Cc: community at lists.openmoko.org
> > Subject: Re: gmail.com problems and this list
> > 
> > Hi everyone,
> > 
> > 
> > Short version:
> > 
> > It seems the duplicated messages are not just a Gmail problem: mails
> > are also being duplicated from starband.net and possibly others as
> > well, so it seems the problem is likely something on the openmoko side
> > that only happens to cause duplicates with some sending domains and
> > not others.  I'ts also not consistent; some messages from Gmail and
> > starband.net are duplicated while others aren't.
> > 
> > I've checked with the Gmail team, and the duplicated messages from
> > Gmail are happening because Gmail doesn't receive the SMTP OK from the
> > openmoko list server, so it retransmits because it thinks the messages
> > don't get delivered.  To debug further, an admin on the list machine
> > needs to enable verbose SMTP logging if it's not already enabled and
> > provide information about if/when the OK is sent.  Here are some
> > example message IDs for which the logs would be useful:
> >   2f3aa2770707070010k7407439axfaa269de33395ea2 at mail.gmail.com
> >   c914135d0707121458s509d1ee1v3a8fe77101bbcaa2 at mail.gmail.com
> >   b829084f0707121328q5e4d0f28me015de4502cffb69 at mail.gmail.com
> > If verbose logging isn't already enabled, however, enabling it and
> > then sending logs for any future duplicated messages would work too.
> > 
> > 
> > Detailed version:
> > 
> > On Mon, Jul 09, 2007 at 11:14:03PM -0700, thus spake Wolfgang S. Rupprecht:
> > > At least one of the opemoko machines at 88.198.124.203 opens an smtp
> > > connection back to the sending domain to verify the sender address of
> > > any incoming message.  The smtp-back machine has messed up DNS.  The
> > > claimed rDNS for that IP is "openmoko.org" but the forward DNS check
> > > for that "openmoko.org" doesn't list 88.198.124.203 as a valid
> > > address.  If the sending machine is checking for spammers claiming a
> > > bunk DNS name will reject 88.198.124.203's SMTP verify.  The opemmoko
> > > machine will then interpret that failed smtp verify attempt as the
> > > verified address not existing and will decline the initial incoming
> > > transfer.
> > 
> > Even ignoring the messed-up DNS for the moment, doing an SMTP callback
> > is not a good way to check existence of an email account.  For one
> > thing, whenever a spammer spoofs a From address in an innocent domain,
> > that domain can get bombarded with callbacks.  For another thing,
> > spammers sometimes use RCPT commands themselves to check existence of
> > email addresses, so ISPs are likely to consider many such requests to
> > be abusive behavior.  In the case of a large ISP (such as Gmail), if a
> > lot of email is sent to a destination that does callbacks, the
> > frequency of callbacks can trigger abuse prevention measures.  See
> > http://en.wikipedia.org/wiki/Callback_verification for more drawbacks.
> > (In particular, note: "If a server receives a lot of spam, it will do
> > a lot of callbacks and if those addresses are invalid, the server will
> > look very similar to a spammer who is doing a dictionary attack to
> > harvest addresses. This in turn might get the server blacklisted
> > elsewhere.")
> > 
> > For these reasons, Gmail strongly recommends avoiding callbacks and
> > instead checking SPF records (Sender Policy Framework, provided via
> > DNS) to verify that the mail is indeed coming from Google (i.e. the
> > SMTP connection is coming from an IP authorized to send email on
> > behalf of Google).
> > 
> > However, in this case it does not seem that the problem was caused by
> > the callbacks.  The openmoko server accepted the mail (which it
> > wouldn't do if the callback failed to verify the sender) and the DATA
> > command was sent, but Gmail never received the next SMTP OK
> > acknowledgment before timing out.  So one of four things was
> > happening:
> > 
> > 1) openmoko never sent the OK -- problem on openmoko end
> > 2) openmoko waited too long before sending the OK and SMTP timed out
> >     -- seems to indicate problem on the openmoko end
> > 3) the OK was lost in transit -- network difficulties, neither side
> >     at fault
> > 4) the OK was received by Gmail -- problem at the Gmail end
> > 
> > Since at least one other domain is showing the same behavior, it seems
> > 1 or 2 is most likely, but we can't say for sure without more
> > information.
> > 
> > One interesting fact is that the openmoko server started sending mail
> > back to Gmail recipients (after list expansion) before Gmail received
> > the SMTP OK.  Perhaps this means the list server waits until after
> > list expansion to send the OK?  That could cause it to time out pretty
> > easily.
> > 
> > It would help diagnosis if someone on the list machine could send the
> > verbose SMTP logs (if they were recorded).  The logs posted by Harald
> > Welte don't give enough information: what we really need to see is
> > whether (and if so, when) the SMTP OK was sent for any of the
> > failed/duplicated messages.  Here are three example message IDs:
> >   2f3aa2770707070010k7407439axfaa269de33395ea2 at mail.gmail.com
> >   c914135d0707121458s509d1ee1v3a8fe77101bbcaa2 at mail.gmail.com
> >   b829084f0707121328q5e4d0f28me015de4502cffb69 at mail.gmail.com
> > If the logging needs to have more verbosity enabled, however, the logs
> > for any duplicated message after such enabling would work just as
> > well.
> > 
> > By the way, the Received header chains posted by David Ford don't
> > necessarily show that the problem is within Gmail; there could be any
> > number of legitimate reasons why the resending isn't originating at
> > the last hop out of the Google network.  Yes, Gmail is sending the
> > mail multiple times, but that's because it thinks the openmoko server
> > did not receive the message on earlier attempts.
> > 
> > I hope someone with admin access to the list machine can help debug
> > this problem so we can get it fixed, wherever the bug is located.
> > 
> > Thanks,
> > Marco
> > 
> > _______________________________________________
> > OpenMoko community mailing list
> > community at lists.openmoko.org
> > http://lists.openmoko.org/mailman/listinfo/community
> > 
> > ----- End forwarded message -----
> > 
> > _______________________________________________
> > OpenMoko community mailing list
> > community at lists.openmoko.org
> > http://lists.openmoko.org/mailman/listinfo/community
> -- 
> Santiago Crespo
> SOC - S21Sec
> 600 409 281 (ext. 262)
> 
> www.s21sec.com
> La seguridad digital del futuro, Hoy.
> 
> La informaci??n contenida en este mail, as?? como los archivos adjuntos,
> es CONFIDENCIAL. Grupo S21sec Gesti??n, S.A. garantiza la adopci??n de las
> medidas necesarias para asegurar el tratamiento confidencial de los
> datos de car??cter personal. En el caso de que el destinatario del correo
> no sea usted, le rogamos env??e una notificaci??n al remitente y lo
> destruya de forma inmediata.
> 
> La lectura y/o manipulaci??n de esta informaci??n en la situaci??n se??alada
> anteriormente ser?? considerada ilegal, permitiendo a la empresa
> remitente realizar acciones legales de diferente envergadura.



> _______________________________________________
> OpenMoko community mailing list
> community at lists.openmoko.org
> http://lists.openmoko.org/mailman/listinfo/community


-- 
"If you want total security, go to prison. There you're fed, clothed,
given medical care and so on. The only thing lacking... is freedom."
    - Dwight D.  Eisenhower, U.S. general and 34th president (1890-1969)

More information about the community mailing list