[imp] IMP performance and message display

[Michael M Slusarz]

Quoting John Murtari <jmurtari at thebook.com>:

> Folks,
>         My two-bits on this conversation.  We'd also been using
> Horde for quite a few years and primarily IMP to support web
> based email for our customers.  Have never been happy with
> it's performance on HTML messages, even in the newest version.

Any specific problems you could enumerate?  Do understand that we are  
much more proactive and thorough with our HTML filtering, so the  
trade-off you may be seeing is security vs. performance.

>         One of the other support guys recommended something
> he has been using at home called RoundCube.  We quickly had
> it up and running as a test install on our servers and folks
> love it. Does the 'ajax' magic and really renders the screen
> very close to what you would see with Outlook, Thunderbird.

IMP has had this for years.  It is called dimp.

>         I don't mean to rain on the Horde parade!  But hopefully
> the Horde development folks can look at the way these guys
> handle message display and get some ideas.

Responding to the other main post in the thread: any questions on  
performance need to be described in a bit more detail.  There are  
multiple components in an IMP installation that have to be considered:  
network bandwidth, IMAP server, disk I/O (server, IMAP server),  
database performance, HTTP server performance, PHP performance, etc.   
It makes a big difference where the bottleneck lies.

For example, Apache + PHP is, IMHO, no longer a viable solution for  
large installations.  Way too bloated.  Using something like lighttpd  
+ FastCGI is much faster and easily scalable, especially if the load  
is concentrated in the Apache/PHP process.  If, on the other hand,  
load on the database is the limiting factor (it is unclear from the  
message whether it was or not), that presents a different set of  
questions.  Are you using split read/write DB's?  IMP itself is not a  
heavy database user in and of itself (IMP's major data source is the  
IMAP server), but other choices may increase that load.  For example,  
more recent version of IMP use caching, which wasn't present on older  
versions.  However, misconfiguring the cache could potentially result  
in increased load/decreased performance.

For example, Horde caching should probably not be used on databases.   
The guaranteed persistence of DB storage (and its concurrent increased  
performance demand) is precisely *not* what is needed/desired for  
caching.  You will most likely saturate the DB and there is absolutely  
no reason this should happen.  Instead, caching should be tailored on  
the architecture of the server farm.  Using Apache + PHP with  
persistent connections from the frontend?  It will most likely be  
fastest to simply use local file caching (although you do run the risk  
of causing duplicate cache data across multiple backend servers if the  
frontend load balancer does no always point a user to the same backend  
server).  Better yet, memcached is the perfect solution for Horde-type  
caching.  At this point I can say that if you are a large installation  
and are not using memcache (or a similar solution) for caching, you  
are probably doing caching wrong (ask Facebook - without memcache, it  
would not be able to run period.)

A more likely scenario is bottleneck on your IMAP server.  This can be  
caused by various things such as the way your store messages on the  
storage backend to the speed of your disk drives.  Using a server that  
caches data can help speeds.  but the one place that admittedly is a  
potential slow spot in IMP 4 is the use of c-client (the PHP imap  
module) to access the IMAP server.  For various reasons that would  
take me many pages to explain, c-client is horrendously inefficient in  
IMAP communication and lacks certain features that used to cause us to  
have to send a bunch of additional IMAP commands to do what we want.

However, this problem has been solved.  The IMAP client has been  
rewritten to use a brand new PHP-based client in IMP 5.  I don't have  
true benchmark numbers, but I can tell you that in unofficial results  
on a large mailbox on the same server with a heavily loaded IMAP  
server, initial mailbox access took 10 seconds in the IMP 4 client and  
2 seconds in the IMP 5 client.  So I am claiming 500% performance  
improvement :)

michael

-- 
___________________________________
Michael Slusarz [slusarz at horde.org]