[imp] login slowness with large inboxes after upgrade from Horde 3.0.10+IMP 4.0.4 to Horde 3.3.8 + IMP 4.3.7

[Alan Kong]

Are you using Dovecot as imap server? Might be the index file(s) get 
corrupted.

Alan

On 11/4/2010 4:34 PM, Jose Manuel Blanco wrote:
> Thank you all for your replies
>
> Regarding the IMAP server side suggestions, the thing is that my old 
> webmail (Horde 3.0.10+IMP 4.0.4) is connecting to the same IMAP 
> server, and it has no problems dealing with large inboxes -1 or 2 
> seconds max. for displaying inboxes with more than 40000 mails never 
> been cached-, so my question remains: has the imp code changed in a 
> way that would explain that behaviour? For example, when I click in 
> the Date tab to reorder my messages, my old webmail only reorders the 
> ones that are visible, while the new one goes to the first (or last) 
> page instead. This suggests that the old one is only dealing with the 
> ones that are visible, but I can't be sure. The settings are the same 
> in both webmails as far as I can see.
>
> Thanks again for your help.
>
> Jose Manuel
>
>
> On 11/03/2010 07:12 PM, Michael M Slusarz wrote:
>> Quoting Aria Bamdad <aria at bsc.gwu.edu>:
>>
>>> I have the same problem with my setup.  I use an IBM IMAP server and 
>>> after
>>> running several traces, we found that IMP tends to request specific 
>>> detailed
>>> information about every mail item in the inbox at logon.  When you have
>>> 1000's of messages and if your IMAP server is not local or is not super
>>> fast, this results in the slow login problem reported here.  My 
>>> solution to
>>> this was to set the $conf[server][sort_limit] variable (IMP 
>>> configuration,
>>> Server tab) to a small number and basically turn off sorting if 
>>> there are
>>> too many messages in a  mailbox.  Doing this, resulted in 
>>> eliminating the
>>> massive request at login.
>>
>> The bottleneck here is your IMAP server then, not IMP.  RFC 3501 is 
>> unfortunately limited in that it does not allow server side sorting 
>> for anything other than arrival time.  So to perform sorting on ANY 
>> other field requires the IMAP client to download all of the necessary 
>> headers from EVERY message in the mailbox to do the sorting on the 
>> client-side.  For a disconnected client, this is a potentially brutal 
>> operation (it is especially brutal for threaded sort, since it relies 
>> on several messages headers at once).  IMP mediates this by being 
>> able to cache header data, but you have to enable caching.
>>
>> This is not a problem with IMP because EVERY client has to do this.  
>> You won't see this in desktop clients, however, since they can do 
>> things like download message headers in the background and cache 
>> these headers.  It is difficult, if not impossible, to do this for a 
>> webmail client, however.
>>
>> Exacerbating the problem is that IMP uses the c-client library to 
>> interact with the IMAP server.  Unfortunately, the c-client library 
>> is known to be HORRIBLY inefficient when it comes to querying the 
>> IMAP server.  To view a single header (say, for example, the "from" 
>> header) c-client downloads the entire message envelope of every 
>> message in the mailbox, rather than just the header requested.  There 
>> is *nothing* we can do about this in IMP 4 since we don't control the 
>> IMAP commands being sent to the server.
>>
>> So now the good news.  First, many modern IMAP servers now support 
>> server-side sorting and threading (RFC 5256 - only formally adopted 
>> in June 2008).  This eliminates issues related to sorting delays.  
>> Better yet, many of these IMAP servers cache the header information 
>> internally so they don't have to parse all mail messages in the 
>> mailbox every time a sort request happens.  This drastically speeds 
>> up mailbox accesses.
>>
>> Additionally, c-client is no longer an issue in IMP 5 since we have 
>> replaced the c-client library with a native Horde IMAP library that 
>> is magnitudes faster.
>>
>>> What would be nice is if IMP only requested specifics about messages 
>>> that
>>> appear in the current window rather than ask for every message in the
>>> mailbox.  The problem is that if a user selects any sort field other 
>>> than by
>>> arrival order, then at login, IMP asks for detailed information 
>>> about every
>>> mail item in the mailbox, slowing down login to a crawl.
>>
>> This is (unfortunately) correct behavior.  You can't sort a single 
>> "message view".  To sort a mailbox requires processing *every* 
>> message in the mailbox - or else how would you determine what 
>> messages a view entails?  We can work around this a bit - namely, 
>> once we get a sorted list for a mailbox, we can continue to use this 
>> sort until the mailbox structure changes.  But that only provides 
>> partial relief.  The only way to truly remedy this situation is to 
>> use an IMAP server that supports RFC 5256.
>>
>> michael
>>
>
>