[imp] login slowness with large inboxes after upgrade from Horde 3.0.10+IMP 4.0.4 to Horde 3.3.8 + IMP 4.3.7

[Michael M Slusarz]

Quoting Aria Bamdad <aria at bsc.gwu.edu>:

> I have the same problem with my setup.  I use an IBM IMAP server and after
> running several traces, we found that IMP tends to request specific detailed
> information about every mail item in the inbox at logon.  When you have
> 1000's of messages and if your IMAP server is not local or is not super
> fast, this results in the slow login problem reported here.  My solution to
> this was to set the $conf[server][sort_limit] variable (IMP configuration,
> Server tab) to a small number and basically turn off sorting if there are
> too many messages in a  mailbox.  Doing this, resulted in eliminating the
> massive request at login.

The bottleneck here is your IMAP server then, not IMP.  RFC 3501 is  
unfortunately limited in that it does not allow server side sorting  
for anything other than arrival time.  So to perform sorting on ANY  
other field requires the IMAP client to download all of the necessary  
headers from EVERY message in the mailbox to do the sorting on the  
client-side.  For a disconnected client, this is a potentially brutal  
operation (it is especially brutal for threaded sort, since it relies  
on several messages headers at once).  IMP mediates this by being able  
to cache header data, but you have to enable caching.

This is not a problem with IMP because EVERY client has to do this.   
You won't see this in desktop clients, however, since they can do  
things like download message headers in the background and cache these  
headers.  It is difficult, if not impossible, to do this for a webmail  
client, however.

Exacerbating the problem is that IMP uses the c-client library to  
interact with the IMAP server.  Unfortunately, the c-client library is  
known to be HORRIBLY inefficient when it comes to querying the IMAP  
server.  To view a single header (say, for example, the "from" header)  
c-client downloads the entire message envelope of every message in the  
mailbox, rather than just the header requested.  There is *nothing* we  
can do about this in IMP 4 since we don't control the IMAP commands  
being sent to the server.

So now the good news.  First, many modern IMAP servers now support  
server-side sorting and threading (RFC 5256 - only formally adopted in  
June 2008).  This eliminates issues related to sorting delays.  Better  
yet, many of these IMAP servers cache the header information  
internally so they don't have to parse all mail messages in the  
mailbox every time a sort request happens.  This drastically speeds up  
mailbox accesses.

Additionally, c-client is no longer an issue in IMP 5 since we have  
replaced the c-client library with a native Horde IMAP library that is  
magnitudes faster.

> What would be nice is if IMP only requested specifics about messages that
> appear in the current window rather than ask for every message in the
> mailbox.  The problem is that if a user selects any sort field other than by
> arrival order, then at login, IMP asks for detailed information about every
> mail item in the mailbox, slowing down login to a crawl.

This is (unfortunately) correct behavior.  You can't sort a single  
"message view".  To sort a mailbox requires processing *every* message  
in the mailbox - or else how would you determine what messages a view  
entails?  We can work around this a bit - namely, once we get a sorted  
list for a mailbox, we can continue to use this sort until the mailbox  
structure changes.  But that only provides partial relief.  The only  
way to truly remedy this situation is to use an IMAP server that  
supports RFC 5256.

michael

-- 
___________________________________
Michael Slusarz [slusarz at horde.org]