[imp] login slowness with large inboxes after upgrade from Horde 3.0.10+IMP 4.0.4 to Horde 3.3.8 + IMP 4.3.7

[Aria Bamdad]

> -----Original Message-----
> From: imp-bounces at lists.horde.org [mailto:imp-bounces at lists.horde.org]
> On Behalf Of Michael M Slusarz
> Sent: Wednesday, November 03, 2010 2:12 PM
> To: imp at lists.horde.org
> Subject: Re: [imp] login slowness with large inboxes after upgrade from
> Horde 3.0.10+IMP 4.0.4 to Horde 3.3.8 + IMP 4.3.7
> 
> Quoting Aria Bamdad <aria at bsc.gwu.edu>:
> 
> > I have the same problem with my setup.  I use an IBM IMAP server and
> after
> > running several traces, we found that IMP tends to request specific
> detailed
> > information about every mail item in the inbox at logon.  When you
> have
> > 1000's of messages and if your IMAP server is not local or is not
> super
> > fast, this results in the slow login problem reported here.  My
> solution to
> > this was to set the $conf[server][sort_limit] variable (IMP
> configuration,
> > Server tab) to a small number and basically turn off sorting if there
> are
> > too many messages in a  mailbox.  Doing this, resulted in eliminating
> the
> > massive request at login.
> 
> The bottleneck here is your IMAP server then, not IMP.  RFC 3501 is
> unfortunately limited in that it does not allow server side sorting
> for anything other than arrival time.  So to perform sorting on ANY
> other field requires the IMAP client to download all of the necessary
> headers from EVERY message in the mailbox to do the sorting on the
> client-side.  For a disconnected client, this is a potentially brutal
> operation (it is especially brutal for threaded sort, since it relies
> on several messages headers at once).  IMP mediates this by being able
> to cache header data, but you have to enable caching.
> 
> This is not a problem with IMP because EVERY client has to do this.
> You won't see this in desktop clients, however, since they can do
> things like download message headers in the background and cache these
> headers.  It is difficult, if not impossible, to do this for a webmail
> client, however.
> 
> Exacerbating the problem is that IMP uses the c-client library to
> interact with the IMAP server.  Unfortunately, the c-client library is
> known to be HORRIBLY inefficient when it comes to querying the IMAP
> server.  To view a single header (say, for example, the "from" header)
> c-client downloads the entire message envelope of every message in the
> mailbox, rather than just the header requested.  There is *nothing* we
> can do about this in IMP 4 since we don't control the IMAP commands
> being sent to the server.
> 
> So now the good news.  First, many modern IMAP servers now support
> server-side sorting and threading (RFC 5256 - only formally adopted in
> June 2008).  This eliminates issues related to sorting delays.  Better
> yet, many of these IMAP servers cache the header information
> internally so they don't have to parse all mail messages in the
> mailbox every time a sort request happens.  This drastically speeds up
> mailbox accesses.
> 
> Additionally, c-client is no longer an issue in IMP 5 since we have
> replaced the c-client library with a native Horde IMAP library that is
> magnitudes faster.
> 
> > What would be nice is if IMP only requested specifics about messages
> that
> > appear in the current window rather than ask for every message in the
> > mailbox.  The problem is that if a user selects any sort field other
> than by
> > arrival order, then at login, IMP asks for detailed information about
> every
> > mail item in the mailbox, slowing down login to a crawl.
> 
> This is (unfortunately) correct behavior.  You can't sort a single
> "message view".  To sort a mailbox requires processing *every* message
> in the mailbox - or else how would you determine what messages a view
> entails?  We can work around this a bit - namely, once we get a sorted
> list for a mailbox, we can continue to use this sort until the mailbox
> structure changes.  But that only provides partial relief.  The only
> way to truly remedy this situation is to use an IMAP server that
> supports RFC 5256.
> 
> michael
> 
> --
> ___________________________________
> Michael Slusarz [slusarz at horde.org]
> 

Michael,

Thanks for the clarifications above.   I completely agree that the problem
is not within IMP.  I am currently working with IBM to see if there are ways
we could address these IMAP server issues.

Regarding turning caching on, I have done so and have set long periods of
time for cache expiration delays but even so, a user with several thousand
messages has a long login delay after a day of inactivity (with IMP).  I am
not sure how IMP decides when the cache is no longer valid and has to be
recreated.  In my case, the only thing that helped was to set the limit on
sorting to mailboxes of 1000 or fewer messages.  I wish there was a way to
just turn off sorting at login and still allow for sorting after login.
Basically,  a parameter that allowed to force the sort order to 'by arrival'
if mailbox contains greater than X messages.  This way, the client still has
the option of sorting but does not have a performance problem loggin in.
With the current configuration options, it's either sorting is enabled or
disabled.

Thanks,
Aria