Duplicates - they won't be fixed on the server, so fix it locallyor live with it

[Joel Newkirk]

On Wed, 10 Sep 2008 21:11:55 +0930, Rod Whitby <rod at whitby.id.au> wrote:
> Q: How long has there been duplicates on the Openmoko mailing lists?
> A: Since the list got so big that sita.openmoko.org can't handle it.
> 
> Q: How long has that been?
> A: At least a year, maybe more.
> 
> Q: Has Openmoko done anything about it?
> A: Doesn't seem so.  Must be a hard problem.
> 
> Q: Is Openmoko likely to fix it?
> A: If they haven't fixed it by now, then chances are they probably won't.
> 
> Q: What should I do about it?
> A: Don't email the list about - we all know already.
> 
> Q: How can I fix it locally for me?
> A: Get a mail reader that understands that a message with a duplicate
>    Message-Id header field doesn't need to be displayed to the user.
> 
> Can we stop now?
> 
> -- Rod

The problem with that approach is that the situation will only get worse -
exponentially.  If the proximate cause is load on the server, then the same
messages being delivered 2-4 times just increases that load, which will
cause more messages to suffer the same effect - increasing the load, et
cetera ad immobilium.  Filtering out dupes at the recipient end is
effective for not seeing the dupes (if available to the subscriber) but the
underlying problem gets continually worse, using up more server and network
resources.  (not even factoring in increasing numbers of subscribers and
posters)

I've built and administered dozens of mailservers at ISPs, processing at
one point over a million SMTP connections a day, so I've got some idea how
'hard' the problem is.  I also know what happens to servers and their
resources when they're overtaxed.  But even from a more general
perspective, hiding a problem never makes it go away.

I realize this thread is utterly off-topic of Openmoko support, and believe
me I was hesitant to bring it up again for that reason.  But it really DOES
need to be fixed, and this list is a reliable way of reaching several
persons at Openmoko, persons who presumably have observed the effect
themselves and can grasp the problem.  

Even with every message in this thread, it's still less than the number of
duplicate messages received in a day...  ;)  And if your mail reader is
advanced enough to offer duplicate-removal, it likely lets you ignore
selected threads as well.

j