[imp] IMP performance and message display

[Janne Peltonen]

On Fri, Mar 20, 2009 at 07:20:43AM -0600, Michael M Slusarz wrote:
> For example, Apache + PHP is, IMHO, no longer a viable solution for  
> large installations.  Way too bloated.  Using something like lighttpd + 
> FastCGI is much faster and easily scalable, especially if the load is 
> concentrated in the Apache/PHP process.

That might be a good idea to try.

We tried using a split apache setup, with very lightweight apache
proxies in threaded mode and without php proxying the connections to
full-blown process-based apaches with php. This was fast, but ran into
weirdness with IE clients getting thrown out almost immediately. And we
tried to apply every IE specific trick we could find, to no avail.

> If, on the other hand, load on 
> the database is the limiting factor (it is unclear from the message 
> whether it was or not), that presents a different set of questions.  Are 
> you using split read/write DB's?

I'm not. But mm, the queries used in say, shares, use searches based on
bitwise fields etc... So if a read is going to take a couple minutes to
complete because no indices can be used because the queries are coded
against all cautions in the database server documentation, it's not much
use splitting writes anywhere else.  The read takes a long time even
with no writes. (This is why everything that needs shares is mostly
disabled here.)

I don't know, maybe I should give read/write splitting a try, after all.
Even if we were to move away from IMP/Horde in the long run, everything
that makes it run better in the meantime is only positive. Any known
risks with this approach?

> IMP itself is not a heavy database user 
> in and of itself (IMP's major data source is the IMAP server), but other 
> choices may increase that load.  For example, more recent version of IMP 
> use caching, which wasn't present on older versions.  However, 
> misconfiguring the cache could potentially result in increased 
> load/decreased performance.
>
> For example, Horde caching should probably not be used on databases.   
> The guaranteed persistence of DB storage (and its concurrent increased  
> performance demand) is precisely *not* what is needed/desired for  
> caching.  You will most likely saturate the DB and there is absolutely  
> no reason this should happen.  Instead, caching should be tailored on  
> the architecture of the server farm.  Using Apache + PHP with persistent 
> connections from the frontend?  It will most likely be fastest to simply 
> use local file caching (although you do run the risk of causing duplicate 
> cache data across multiple backend servers if the frontend load balancer 
> does no always point a user to the same backend server).

We use FS based caching. Load balancing is primitive, it is DNS-based,
using PowerDNS to direct users to different servers according to a hash
on their IP address.

> Better yet, 
> memcached is the perfect solution for Horde-type caching.  At this point 
> I can say that if you are a large installation and are not using memcache 
> (or a similar solution) for caching, you are probably doing caching wrong 
> (ask Facebook - without memcache, it would not be able to run period.)

OK, another point to try. Thanks. Would this be faster than the file
based caching we're currently using? Sounds improbable, but.

> A more likely scenario is bottleneck on your IMAP server.  This can be  
> caused by various things such as the way your store messages on the  
> storage backend to the speed of your disk drives.  Using a server that  
> caches data can help speeds.  but the one place that admittedly is a  
> potential slow spot in IMP 4 is the use of c-client (the PHP imap  
> module) to access the IMAP server.  For various reasons that would take 
> me many pages to explain, c-client is horrendously inefficient in IMAP 
> communication and lacks certain features that used to cause us to have to 
> send a bunch of additional IMAP commands to do what we want.

There might be something for this... our IMAP servers did experience a
noticable increase in load when we upgraded to a modern IMP.

> However, this problem has been solved.  The IMAP client has been  
> rewritten to use a brand new PHP-based client in IMP 5.  I don't have  
> true benchmark numbers, but I can tell you that in unofficial results on 
> a large mailbox on the same server with a heavily loaded IMAP server, 
> initial mailbox access took 10 seconds in the IMP 4 client and 2 seconds 
> in the IMP 5 client.  So I am claiming 500% performance improvement :)

Sounds very good to me.


--Janne
-- 
Janne Peltonen <janne.peltonen at helsinki.fi> PGP Key ID: 0x9CFAC88B
Please consider membership of the Hospitality Club (http://www.hospitalityclub.org)