[horde] runaway httpd processes?

[James MacLean]

Dominaux, Craig C wrote:
> I don't think the issue for us anyway is with the SQL queries because we
> truncate the horde_datatree and horde_datatree_attributes every hour to
> keep these tables low. And the reason we can get away with this is we
> don't allow any sharing or set permissions on our address books,
> calendars, notes, or tasks.
>
> Our horde_datatree_attributes don't usually get any higher than 150000
> rows or the horde_datatree with 35000 rows. 
>
> Since I put in this script login time have been very fast but didn't
> help with runaway httpd processes.
>
>
>
> -----Original Message-----
> From: horde-bounces at lists.horde.org
> [mailto:horde-bounces at lists.horde.org] On Behalf Of James MacLean
> Sent: Monday, January 29, 2007 2:43 PM
> To: Horde List
> Subject: Re: [horde] runaway httpd processes?
>
> Michael M Slusarz wrote:
>   
>> Quoting Andrew Morgan <morgan at orst.edu>:
>>
>>     
>>> On Mon, 29 Jan 2007, Dominaux, Craig C wrote:
>>>
>>>       
>>>> We are having the same issue here but haven't been able to isolate
>>>>         
> the
>   
>>>> problem.
>>>>
>>>> In the interim we setup a cron job to run every 10 minutes to search
>>>>         
>
>   
>>>> for
>>>> any HTTPD processes that have been running over 10 minutes and kill
>>>>         
> the
>   
>>>> process.
>>>> Most of our httpd processes kill themselves off before 2 minutes.
>>>>         
>> There have been several changes made to Horde/IMP lately that has 
>> tackled some possible causes of these processess.  This includes 
>> better memory usage when looping through large data structures, 
>> improvements to SQL queries (most notably in DataTree), and limiting 
>> users from performing certain activities - like viewing text messages 
>> that are extremely large - that would otherwise cripple a system 
>> serving multiple simultaneous users.
>>
>> However, you will need to use HEAD to see these changes/fixes.
>>
>> michael
>>
>> ___________________________________
>> Michael Slusarz [slusarz at horde.org]
>>     
> Second Michael's remarks and patches with what we have been experiencing
>
> locally.
>
> Our stracing always showed php going south when MySQL returned large 
> result sets. These queries seem to work fine if you run them on a 
> separate php script, but somewhere in passing them around inside of 
> Horde, no matter if it is php 4.3,5.1, or 5.2, the CPU goes up and other
>
> users start to suffer. First sign is strace showing a slew of DB data 
> returning, usually information mostly not needed :(.
>
> We had just finished making a real strange left join to deal with real 
> slow login times for new users when Chuck pointed to :
>
> http://lists.horde.org/archives/cvs/Week-of-Mon-20070122/064958.html
>
> in a previous post. That was much better then the crazy solution we had 
> and so far it's working fine with around 5,000 users on. I'd certainly 
> suggest trying them. You will need to take from the links and apply them
>
> to your CVS version as (at least for us) even HEAD is not up to date 
> with 2 of the three patches.
>
> JES
>   
Our datatree starts to slow down well before 35,000 :).

Reason, according to the tracing, is the SELECT from DataTree where 
group_uid ='horde.something' in sql.php. This is fast to execute on 
MySQL, but returns the large row result set and php then churns dealing 
with them. Turn up DEBUG and watch /tmp/horde.log to see the selects 
when you login as a new user :).

If you have set your DataTree driver to null, then I don't think that 
this applies.

We also looked at Turba result sets. We put a limit on turba result sets 
as the global address book against our 150,000+ users would cause the 
same problem when returning large finds :(.