[imp] Hack concept - IMAP speedup

[Ross Becker]

Just throwing this out, I'm really not certain as to which organizations 
I should be talking to about this, but Horde/IMP seems as good as any.

Currently, free IMAP servers which are capable of webmail backend is bad 
for any organization which can allow large folders. The primary reason 
is that maildir (and maildir++) has no indexing at all. This means that 
to sort a folder, you have to open every message, read the headers, suck 
out the one(s) you need, and transform them into sortable format.  Once 
you've read every message, then you can sort.  This leads to woeful 
performance.  On a Pentium 4 with 800mhz memory bus, I clocked 
Courier-IMAP taking ~14 seconds to sort a 12.5k message folder by date.

Clearly, for a webmail system, you want the server to be able to sort 
the messages fast, because all sorts will happen on the server.

I'm looking at hacking (probably) procmail, and Courier-IMAP to maintain 
  fixed-length record files containing the message headers needed to 
support standard search and sort operations.  Not indexes, as that would 
require putting sorting code into the MDA (procmail).  At a wild guess, 
  this should provide at least an order of magnitude improvement in 
search and sort times.

The ugliness is figuring out if procmail is the best delivery agent to 
add this hack to (I'm leaning there because of it's excellent support 
for delivery recipes & filters) and if there's any way this could be 
done in such a fashion that it could go into permanent project code.

If anyone else here is interested in this hack, has thoughts, or 
whatever- I'm all ears.

Cheers
    Ross