[imp] login slowness with large inboxes after upgrade from Horde 3.0.10+IMP 4.0.4 to Horde 3.3.8 + IMP 4.3.7

[Jose Manuel Blanco]

Hi Alan

Yes, it is Dovecot IMAP server and, indeed, we've had those kind of 
issues in the past, which we solve by forcing the IMAP server to 
recreate the indexes. The problem is that we are accessing from both 
webmails (I have both old and new operational) and only the new one 
shows this kind of behaviour, so I'm discarding any influence from the 
IMAP server.

Thanks for answering
Jose Manuel

On 11/04/2010 10:08 AM, Alan Kong wrote:
> Are you using Dovecot as imap server? Might be the index file(s) get 
> corrupted.
>
> Alan
>
> On 11/4/2010 4:34 PM, Jose Manuel Blanco wrote:
>> Thank you all for your replies
>>
>> Regarding the IMAP server side suggestions, the thing is that my old 
>> webmail (Horde 3.0.10+IMP 4.0.4) is connecting to the same IMAP 
>> server, and it has no problems dealing with large inboxes -1 or 2 
>> seconds max. for displaying inboxes with more than 40000 mails never 
>> been cached-, so my question remains: has the imp code changed in a 
>> way that would explain that behaviour? For example, when I click in 
>> the Date tab to reorder my messages, my old webmail only reorders the 
>> ones that are visible, while the new one goes to the first (or last) 
>> page instead. This suggests that the old one is only dealing with the 
>> ones that are visible, but I can't be sure. The settings are the same 
>> in both webmails as far as I can see.
>>
>> Thanks again for your help.
>>
>> Jose Manuel
>>
>>
>> On 11/03/2010 07:12 PM, Michael M Slusarz wrote:
>>> Quoting Aria Bamdad <aria at bsc.gwu.edu>:
>>>
>>>> I have the same problem with my setup.  I use an IBM IMAP server 
>>>> and after
>>>> running several traces, we found that IMP tends to request specific 
>>>> detailed
>>>> information about every mail item in the inbox at logon.  When you 
>>>> have
>>>> 1000's of messages and if your IMAP server is not local or is not 
>>>> super
>>>> fast, this results in the slow login problem reported here.  My 
>>>> solution to
>>>> this was to set the $conf[server][sort_limit] variable (IMP 
>>>> configuration,
>>>> Server tab) to a small number and basically turn off sorting if 
>>>> there are
>>>> too many messages in a  mailbox.  Doing this, resulted in 
>>>> eliminating the
>>>> massive request at login.
>>>
>>> The bottleneck here is your IMAP server then, not IMP.  RFC 3501 is 
>>> unfortunately limited in that it does not allow server side sorting 
>>> for anything other than arrival time.  So to perform sorting on ANY 
>>> other field requires the IMAP client to download all of the 
>>> necessary headers from EVERY message in the mailbox to do the 
>>> sorting on the client-side.  For a disconnected client, this is a 
>>> potentially brutal operation (it is especially brutal for threaded 
>>> sort, since it relies on several messages headers at once).  IMP 
>>> mediates this by being able to cache header data, but you have to 
>>> enable caching.
>>>
>>> This is not a problem with IMP because EVERY client has to do this.  
>>> You won't see this in desktop clients, however, since they can do 
>>> things like download message headers in the background and cache 
>>> these headers.  It is difficult, if not impossible, to do this for a 
>>> webmail client, however.
>>>
>>> Exacerbating the problem is that IMP uses the c-client library to 
>>> interact with the IMAP server.  Unfortunately, the c-client library 
>>> is known to be HORRIBLY inefficient when it comes to querying the 
>>> IMAP server.  To view a single header (say, for example, the "from" 
>>> header) c-client downloads the entire message envelope of every 
>>> message in the mailbox, rather than just the header requested.  
>>> There is *nothing* we can do about this in IMP 4 since we don't 
>>> control the IMAP commands being sent to the server.
>>>
>>> So now the good news.  First, many modern IMAP servers now support 
>>> server-side sorting and threading (RFC 5256 - only formally adopted 
>>> in June 2008).  This eliminates issues related to sorting delays.  
>>> Better yet, many of these IMAP servers cache the header information 
>>> internally so they don't have to parse all mail messages in the 
>>> mailbox every time a sort request happens.  This drastically speeds 
>>> up mailbox accesses.
>>>
>>> Additionally, c-client is no longer an issue in IMP 5 since we have 
>>> replaced the c-client library with a native Horde IMAP library that 
>>> is magnitudes faster.
>>>
>>>> What would be nice is if IMP only requested specifics about 
>>>> messages that
>>>> appear in the current window rather than ask for every message in the
>>>> mailbox.  The problem is that if a user selects any sort field 
>>>> other than by
>>>> arrival order, then at login, IMP asks for detailed information 
>>>> about every
>>>> mail item in the mailbox, slowing down login to a crawl.
>>>
>>> This is (unfortunately) correct behavior.  You can't sort a single 
>>> "message view".  To sort a mailbox requires processing *every* 
>>> message in the mailbox - or else how would you determine what 
>>> messages a view entails?  We can work around this a bit - namely, 
>>> once we get a sorted list for a mailbox, we can continue to use this 
>>> sort until the mailbox structure changes.  But that only provides 
>>> partial relief.  The only way to truly remedy this situation is to 
>>> use an IMAP server that supports RFC 5256.
>>>
>>> michael
>>>
>>
>>


-- 
Jose Manuel Blanco Ramos
European Space Astronomy Centre (ESAC)
European Space Agency (ESA)

E-mail: jmblanco at sciops.esa.int
Tel: +34 91 8131 185, Fax: +34 91 8131 322
P.0. Box 78, 28691 VILLANUEVA DE LA CAÑADA
SPAIN



================================================================================================
This message and any attachments are intended for the use of the addressee or addressees only. The unauthorised disclosure, 

use, dissemination or copying (either in whole or in part) of its content is not permitted. If you received this message in 

error, please notify the sender and delete it from your system. Emails can be altered and their integrity cannot be guaranteed by   

the sender. 

Please consider the environment before printing this email.
=================================================================================================