[horde] Problem of slow with server-side sorting

[Michael M Slusarz]

Quoting Rick Romero <rick at havokmon.com>:

>> That might help us narrow down if the problem is caused by Horde or  
>> if it is a problem with any IMAP client.
>
> I think it's Courier-IMAP.  I had general slowness issues with large  
> mailboxes using Courier until I upgraded to Dovecot.  Dovecot has an  
> index for each folder, and the first time I did a search from Horde  
> it came back almost immediately.  I was astonished at how fast a  
> search was.  I was expecting it to take a while.

Agreed.  For a webmail client, one should *really* be using an IMAP  
server that caches on the server side.  And realize that anytime a  
message is added to a mailbox, the whole mailbox must be resorted  
unless using arrival sort - it is impossible to tell from the client  
side where a message belongs in the sorted list.

A caching IMAP server only needs to parse this information from the  
mailbox once (on the initial scan of the message).  To sort, it can  
then simply use the cached values of the relevant sorting field and  
quickly do the necessary sorting.  For a non-caching IMAP server,  
*every* time a new message is added (and most likely on the initial  
access to the mailbox in a given session) the *entire* mailbox, and  
all messages within that mailbox, must be reparsed.  This + a mailbox  
with 1000's of messages + many users with 1000's of messages in a  
mailbox is going to saturate the disk I/O on all but the most  
expensive disk setups.

Client side sorting, while feasible for an application like  
Thunderbird, is entirely infeasible for webmail.  IMP uses some  
caching tricks to try to minimize the number of SORT calls needed, but  
it can't eliminate all, especially for a mailbox such as INBOX that is  
theoretically receiving new messages at a fairly rapid rate.

As previously mentioned, the solutions are:
1. Use a caching IMAP server (Dovecot and Cyrus are 2 examples.   
Archiveopteryx, another freely available alternative, also supports  
caching AFAICT although it likely isn't as stable as the previously  
mentioned products)
2. Lock users to arrival sort (or set the sort_limit preference to a  
low number of messages)
3. Buy/create a vastly expensive disk array that allows concurrent,  
blazing fast access

michael

-- 
___________________________________
Michael Slusarz [slusarz at horde.org]