[imp] Improving IMP message load times

[Michael M Slusarz]

Quoting Gunnar Wrobel <wrobel at pardus.de>:

> Michael M Slusarz <slusarz at horde.org> schrieb:
>
>> Quoting Mat Cantin <mat at cantinbrothers.ca>:
>>
>>> Greetings,
>>>
>
>
>>> I currently have Horde4 groupmail installed and operational on
>>> Lighttpd with the Dovecot IMAP backend for email and authentication.
>>
>>> I've been working on trying to improve IMP's performance when
>>> loading new messages that have not yet been viewed and cached. I've
>>> been able to get it down to about 3 seconds, which I understand is
>>> pretty typical because that's how long it takes to establish a new
>>> IMAP session and download the message.
>>
>> 3 seconds?  According to Firebug, it took 453ms total to view your
>> message, from the initial browser request to receiving the full
>> response from the server (granted, my mail server is located about 25
>> feet away in my basement, so network latency is essentially zero, but
>> still...)
>>
>> You are using the dynamic view right?  Traditional view will take much
>>
>> longer because it needs to rebuild the entire page on every view (and
>> it is even worse if you have the sidebar open).  Dynamic view only
>> needs to prepare the message view data when viewing a message.
>>
>>> Although 3 seconds is good, when my users want to look through
>>> several new messages quickly, it's really not that great, especially
>>
>>> when comparing the performance with Google Mail and Outlook Web
>>> Access.
>>
>> Have you tried the recommendations in imp/docs/PERFORMANCE?
>> Specifically using imapproxy?
>>
>>> Getting to the point, I've noticed that once a message has been
>>> viewed and cached, going back to that message later is
>>> instantaneous. Does anyone know if it's possible to configure IMP to
>>
>>> pre-cache all the messages in the current view? Ideally, it could
>>> quickly load and show the user's message list and then proceed to
>>> download the messages in the background. I realize that this will
>>> obviously take more server resources, but that's a trade-off we can
>>> make.
>>
>> There should be no need to pre-cache.  500ms to view a message is more
>>
>> than fast enough.
>>
>
> I'm not so certain. 500ms is still noticeable if you try to navigate  
> quickly through the message list using the keyboard.

I don't notice much of a lag.  There is the general issue that we  
can't throw a bunch of parallel requests at the PHP server since we  
can only realistically handle one request at a time.  But this is a  
PHP limitation (w/ sessions) and there does not exist a reasonable  
solution otherwise.  So you can't mash the down key 10 times and  
expect to instantly view all 10 messages.  You necessarily have to  
wait for the previous 9 messages to load before the 10th is displayed.

But in practice, this is irrelevant.  For one of my clients we did  
clicklogging tests and discovered that in practice, nobody uses the  
keyboard.  It was debatable whether to even continue supporting  
keyboard shortcuts.  Using the mouse, it is practically impossible to  
select messages fast enough to cause a significant backlog in the  
message viewing queue.

> And I assume that your machine is well tuned. So half a second is  
> probably near the lower boundary for a typical one server system. I  
> would guess that many initial installs have worse performance.

What is wrong with this?  It is unreasonable to expect a base install  
to be perfectly tuned.  The kind of things that will speed up  
performance (caching; imap proxy; IMAP server choice; mail storage  
format) are things that we can't control from an installation point of  
view - they necessarily have to setup/configured by the admin.

We can provide a roadmap on what to do, and make it easier for the  
admin to setup, but we can't activate these things by default.  I sure  
would not want to install a piece of software that auto-scans for a  
local memcache installation and used it by default, for example.

> Is prefetching a bad idea? If so: for what reasons? Does google or  
> roundcube use such an approach?

Roundcube load times are absolutely horrific.  It pains me to  
watch/look at their IMAP performance.

Gmail probably does preload (looks like they do).  But Gmail is a bad  
example since their backend leverages all sorts of server farms.  From  
what I know, they have different farms that serve different parts of  
their pages.  So they can do all this crazy stuff.  Horde/IMP is not  
designed to do this (by default - we do allow some offloading of  
things like static content to other servers).

There are also major issues with bandwidth, both network and  
IMAP-side.  Not to mention CPU and memory usage on the PHP server.

Quite frankly, Gmail is a bad benchmark/ideal to achieve.  They have a  
completely different design goal - they admit that much of what they  
do behind-the-scenes are total hacks that are only work in their local  
environment (e.g. they purposely broke IMAP to fit their local goals);  
we have to provide a turnkey solution that works on a wide-variety of  
platforms.  Plus they have almost limitless resources and a large  
support group to deal with the inevitable issues that come from  
pushing around so much data.

Would it be impossible?  No.  But unless someone throws a huge amount  
of money to fund this kind of feature, our time is better spent  
elsewhere.

And on my system at least, IMP is much faster than native thunderbird.  
  So as far as a freely available solution, I will put IMP up with any  
product in terms of speed.

michael

___________________________________
Michael Slusarz [slusarz at horde.org]