[imp] Imp Performance Problem

[Registration]

> On 2015-04-03, at 16:20 , Michael M Slusarz <slusarz at horde.org> wrote:
> 
> Quoting Registration <registration at portabile.net>:
> 
>>> On 2015-03-30, at 23:21 , Registration <registration at portabile.net> wrote:
>>> 
>>> 
>>>> On 2014-10-29, at 17:45 , Registration <registration at portabile.net> wrote:
>>>> 
>>>> 
>>>>> On 2015-03-29, at 14:10 , Arjen de Korte <arjen+horde at de-korte.org> wrote:
>>>>> 
>>>>> Citeren Registration <registration at portabile.net>:
>>>>> 
>>>>>> Dear Imp users and developers,
>>>>>> 
>>>>>> my problem is as follows:
>>>>>> - Login to Horde takes 3 seconds.
>>>>>> - Loading mails takes 7 seconds for a larger IMAP folder, and 1-2 seconds for one with only a few messages in it.
>>>>>> - Top shows 97% CPU from php5-fpm, 3% from imap.
>>>>> 
>>>>> That means that performance problems are probably in php. Do you use an php opcode cache / accelerator? If so, which one?
>>>> 
>>>> XCache 2.0.0
>>>> 
>>>>> 
>>>>>> - When messages are cached, everything gets super fast.
>>>>>> - The loading times are probably proportional to the amount of message headers Imp loads for viewing.
>>>>>> 
>>>>>> My questions are:
>>>>>> - Is this normal speed?
>>>>> 
>>>>> It depends. How large is a large IMAP folder? How many folders do you have?
>>>> 
>>>> I think that Imp caches something a few dozen messages around the currently displayed ones.
>>>> So the IMAP folder size is not important, just that there are more than a few:
>>>> For example, an inbox with 10 messages is loaded in a second, a view of a folder with 900 messages
>>>> takes 6-8 seconds, and also 6-8 seconds for a folder with 20000 messages.
>>>> 
>>>>> 
>>>>>> - If not, how would I systematically search for the problem?
>>>>>> - I tried xhprof, but I can't get the part where Imp refreshes messages captured.
>>>>>> 
>>>>>> My config:
>>>>>> - Debian Wheezy, Apache, php5-fpm, Dovecot, imapproxy
>>>>> 
>>>>> Without version numbers, there is no way for us to tell what you're actually using.
>>>> 
>>>> Kernel 3.2.0-4-amd64, Apache 2.2.22, PHP 5.4.39-0+deb7u1, Zend 2.4.0, XCache 2.0.0, Dovecot 2.1.7
>>>> 
>>>>> 
>>>>>> - Auth using Imp using MySQL/PDO connection via sockets
>>>>>> - Dovecot using Maildir, PAM disabled, mysql auth only
>>>>>> - Horde caching enabled
>>>>> 
>>>>> Which cache backend?
>>>>> 
>>>>>> - everything on one host
>>>>> 
>>>>> If everything is on the same host, imapproxy is redundant an may even hurt performance if you have a modern IMAP server like Dovecot.
>>>> 
>>>> Ok, disconnected and purged it, no visible difference.
>>>> 
>>>>> 
>>>>>> - standard rented server
>>>>> 
>>>>> I wonder what the specifications of a 'standard rented server' are. :-)
>>>> 
>>>> AMD Opteron 6272, 2 GB RAM, 100 GB SSD - not high end, but not stone-age either, I think
>>>> 
>>>>> 
>>>>>> - everything up-to-date
>>>>> 
>>>>> What may be up-to-date on Debian Wheezy may still be seriously out-of-date.
>>>> 
>>>> I know, but no testing builds for me anymore...
>>> 
>>> I did a performance test using http://www.php-benchmark-script.com/bench.php, which resulted in ~7 seconds for my server, same time as the test on the benchmarks webpage.
>>> 
>>> The only thing I could find which could be related to this is the Horde-specific message:
>>> FastCGI: server "/usr/lib/cgi-bin/php5-fcgi" stderr: PHP message: Cannot write Autoloader cache to backend.
>>> I tried setting proper path permissions, but no success, the error doesn't go away.
>>> 
>> 
>> I managed to get the folder view loading time down to four seconds for a larger IMAP folder  (>1k files) like "sent" or "archive".
>> Could anyone tell me how long it takes to display a folder on their server? I don't have access to a second horde installation.
> 
> My 12,000 message Trash folder takes around ~2 seconds to load the initial view.
> 
> Understand that this kind of operation (message list) is almost certainly a bottleneck at the IMAP level, rather than the PHP level.  Your performance is going to be pretty bad on large mailboxes unless using a caching IMAP server or caching on the Horde side (or both).
> 
> michael
> 
> ___________________________________
> Michael Slusarz [slusarz at horde.org]
> 
> -- 
> imp mailing list
> Frequently Asked Questions: http://wiki.horde.org/FAQ
> To unsubscribe, mail: imp-unsubscribe at lists.horde.org
> 
> 

Thanks, that's some info I needed! So at least, it isn't orders of magnitude slower than other installations...

I have caching, it displays messages immediately, but it only caches per session.
I have these options related to caching:

imp/config/backends.local.php:
'cache' => 'hashtable',

config/conf.php:
$conf['session']['cache_limiter'] = 'nocache';
$conf['cache']['default_lifetime'] = 86400;
$conf['cache']['params']['dir'] = '/tmp/horde';
$conf['cache']['params']['sub'] = 0;
$conf['cache']['driver'] = 'File';
$conf['cache']['use_memorycache'] = 'Xcache';
$conf['imap']['cache_folders'] = true;
$conf['hashtable']['params']['hostspec'] = array('127.0.0.1');
$conf['hashtable']['params']['port'] = array('11211');
$conf['hashtable']['params']['weight'] = array();
$conf['hashtable']['params']['persistent'] = true;
$conf['hashtable']['params']['compression'] = false;
$conf['hashtable']['params']['large_items'] = true;
$conf['hashtable']['driver'] = 'Memcache';