[imp] Hack concept - IMAP speedup

[Daniel Eckl]

There are some post already for cyrus. I's like to add some details:

Cyrus has indexes for headers by default and you can add an index for the 
whole mailbody, too.

Sorting within a single folder containing 17000 (reads seventeenthousand) 
mails is done with no noticable delay using cyrus 2.2.3 on a dual XEON 2,4 
Ghz machine.

Using the SQUAT body index, searching in the body of those 17000 mails is done 
in about 3 seconds.

Daniel

Am Mittwoch, 8. September 2004 23:59 schrieb Ross Becker:
> Just throwing this out, I'm really not certain as to which organizations
> I should be talking to about this, but Horde/IMP seems as good as any.
>
> Currently, free IMAP servers which are capable of webmail backend is bad
> for any organization which can allow large folders. The primary reason
> is that maildir (and maildir++) has no indexing at all. This means that
> to sort a folder, you have to open every message, read the headers, suck
> out the one(s) you need, and transform them into sortable format.  Once
> you've read every message, then you can sort.  This leads to woeful
> performance.  On a Pentium 4 with 800mhz memory bus, I clocked
> Courier-IMAP taking ~14 seconds to sort a 12.5k message folder by date.
>
> Clearly, for a webmail system, you want the server to be able to sort
> the messages fast, because all sorts will happen on the server.
>
> I'm looking at hacking (probably) procmail, and Courier-IMAP to maintain
>   fixed-length record files containing the message headers needed to
> support standard search and sort operations.  Not indexes, as that would
> require putting sorting code into the MDA (procmail).  At a wild guess,
>   this should provide at least an order of magnitude improvement in
> search and sort times.
>
> The ugliness is figuring out if procmail is the best delivery agent to
> add this hack to (I'm leaning there because of it's excellent support
> for delivery recipes & filters) and if there's any way this could be
> done in such a fashion that it could go into permanent project code.
>
> If anyone else here is interested in this hack, has thoughts, or
> whatever- I'm all ears.
>
> Cheers
>     Ross