[horde] Too many httpd processes and no response

[Arjen de Korte]

Citeren ANANT S ATHAVALE <asa at isac.gov.in>:

> ----- Message from Arjen de Korte <arjen+horde at de-korte.org> ---------
>    Date: Mon, 13 Jan 2014 15:52:03 +0100
>    From: Arjen de Korte <arjen+horde at de-korte.org>
> Subject: Re: [horde] Too many httpd processes and no response
>      To: horde at lists.horde.org
>
>> Citeren ANANT S ATHAVALE <asa at isac.gov.in>:
>>
>>> Dear List,
>>>
>>> I have released HGWE 5.1.3 to my users from last 3 days.
>>>
>>> We are facing unusual problem.
>>>
>>> When there is not much load, there is no problem at all.
>>>
>>> But, during peak hours of usage on a working day, suddenly too many httpd
>>> processes start and user gets no response.  Only solution is to restart
>>> httpd.  By this, there won't any issue for atleast one hour and problem
>>> repeats.
>>
>> My first guess would be that you allow your server to spawn too many
>> processes to serve the incoming requests. If this is more than the
>> amount of memory allows, the result will be swapping and a dramatic
>> decrease in performance. Users will try to reconnect then, worsening the
>> problem. Which webserver are you using? If Apache, which MPM are you
>> using?
>
> Using Apache.  prefork MPM.  Default, as provided by RHEL.

What is the average process size of Apache? You may need to change  
'httpd' in the below to whatever RHEL is using:

ps -ylC httpd | awk '{x += $8;y += 1} END {print "Apache Memory Usage  
(MB): "x/1024; print "Average Proccess Size (MB): "x/((y-1)*1024)}'

Make sure that the server is actually under load, to make sure the  
process sizes reported are representative. If the average process size  
multiplied by the value of MaxClients in your apache configuration is  
more than about 75-80% of the memory in the system, you're just  
waiting for a disaster to happen sooner or later.

How many concurrent users do you expect to use Horde?

>>> I am unable to diagonise this right now.
>>>
>>> 1. I have installed HGWE on RHEL 6.4
>>> 2. Using APC Cache as provided with RHEL
>>
>> Is there enough memory available for APC? Look at the apc.php status
>> page to see if there is sufficient memory available, especially if
>> you're using it as a caching backend.
>
> This, I have started monitoring now.  Being a holiday today, it has only
> used 56.2 MB.  I had restarted httpd yesterday evening.  Right now, I
> have set it to 256 MB.  System has 32 GB RAM.

Monitor that value under use and make sure it is never for more that  
about 75% full.

>>> 3. Using PHP 5.3.3-27 as provided with RHEL
>>> 4. Using SquirrelMail IMAP proxy latest stable version, compiled on the
>>> same server.
>>> 5. Using Cyrus-IMAP server on a remote server and following string logged
>>> in Horde-IMAP debug log
>>>
>>> S: * OK [CAPABILITY IMAP4rev1 LITERAL+ ID ENABLE  XIMAPPROXY]
>>> newisacmail.isac.dos.gov.in Cyrus IMAP v2.4.17-Invoca-RPM-2.4.17-6.el6
>>> server ready
>>
>> Have you enabled IMAP caching in imp/config/backends.local.php?
>
> following is the conf.php for Horde.
>
> $conf['session']['cache_limiter'] = 'nocache';
> $conf['share']['cache'] = false;
> $conf['cache']['default_lifetime'] = 86400;
> $conf['cache']['driver'] = 'Apc';
> $conf['cache']['use_memorycache'] = 'Apc';

This is redundant. You're already using a memory caching backend, so  
'use_memorycache' has no effect.

> $conf['cachecssparams']['driver'] = 'filesystem';
> $conf['cachecssparams']['lifetime'] = 86400;
> $conf['cachecss'] = true;
> $conf['cachejsparams']['driver'] = 'filesystem';
> $conf['cachejsparams']['compress'] = 'php';
> $conf['cachejsparams']['lifetime'] = 604800;
> $conf['cachejs'] = true;
> $conf['cachethemesparams']['check'] = 'appversion';
> $conf['cachethemesparams']['lifetime'] = 604800;
> $conf['cachethemes'] = true;
>
> and backendslocal.php for IMP it is:
>
> 'cache' => 'cache',

This value is not recommended. From the comments in 'imp/config/backends.php':

  *     - 'cache': Caching is enabled using the Horde cache (configured in
  *           horde/config/conf.xml). It is recommended to use either 'nosql'
  *           or 'sql' instead, as these backends have much better performance.

> Do I need to change this to NoSQL?

If you already have a NoSQL server available, yes. If you don't, a SQL  
server will work just fine. Given the amount of data to store, you'll  
not want 'cache' here, since potentially there will be way more data  
than the memory caching backend you have configured can store.

>>> I am unable to debug why httpd suddenly starts too many processes.  I have
>>>
>>> already enabled DEBUG log in Horde conf.php
>>
>> Definitly don't do this. Your investigating a performance problem here.
>> Logging at level DEBUG will generate tons of information and on a busy
>> server this will be enough to bring it to its knees.
>
> Today, being holiday, I will continue with DEBUG and disable in the
> evening.

Again, DON'T DO THIS! NEVER, EVER enable DEBUG on a production system!  
You'll get WAY too much logging to be useful on a production system.  
Typically, DEBUG is used on a system with a single (test)user to  
diagnose what is going on if a function of Horde doesn't work as  
desired.