[dev] Fix kronolith-agenda script

[Gonçalo Queirós]

On 03/21/2012 11:40 AM, Gonçalo Queirós wrote:
> Citando Jan Schneider <jan at horde.org>:
>> Zitat von Gonçalo Queirós <goncalo.queiros at portugalmail.net>:
>>> Hi there dev.
>>>
>>>    We were trying to use the kronolith-agenda script to send daily 
>>> agendas to
>>>    everyone on our service, but the problem is that we currently 
>>> have more
>>>    than 350.000 shares and the script just runs out of memory, even 
>>> with 2Gb!
>>>
>>>    Looking at the script more closely, we think there's a way to 
>>> make the
>>>    script work, regardless of the number of shares on the system, 
>>> but we would
>>>    like your opinion on that before coming out with a patch.
>>>
>>>    Current script:
>>>    1 - Get every calendar share
>>>    2 - For every share, list its events, to check if it has any 
>>> event to the
>>>    current day
>>   This is not correct, there is no such step
> Sorry, was debugging on my own code ;-)
>>
>>> 3 - For the remaining list get all users that have access to the 
>>> calendars
>>>    4 - For every user, check if he desires to receive the daily 
>>> agenda (pref)
>>>    5 - For every user that wants to receive the daily agenda, get his
>>>    calendars
>>>    6 - Again for every calendar get the ones that have any event to the
>>>    current day
>>>    7 - Send the email if there's any calendar left
>>>
>>>    For our installation the current script stops on the first step, 
>>> because
>>>    it runs out of memory.
>>>    We thought on creating sub-sets for the shares, but the problem 
>>> is that we
>>>    only know the full agenda of a user after we analyze all shares 
>>> he has
>>>    access to.
>>>
>>>    What we propose:
>>>    1 - Get every users that desire to receive the daily agenda (pref)
>>   That's exactly what steps 1, 3, and 4 do.
> I know, the difference is that instead of asking for all the shares, we
> could ask directly for all the users that wan't to receive an agenda, so
> we could eventually narrow the search.
> Also, knowing the user instead of the calendars allows us to immediately
> send the user his agenda, which will free the memory when the loop for
> that user ends.
>>> 2 - execute the steps 5,6,7 of the original script
>>>
>>>    In the worst case scenario this script will still perform better 
>>> than the
>>>    current one, because it doesn't have the first 3 steps.
>>>    With this approach we can create sub-sets of users which will 
>>> allow the
>>>    script to run until the end without running out of memory (even 
>>> if this is
>>>    a long process, it will execute)
>>>
>>>    Problems:
>>>    We don't think there's currently a method to retrieve all prefs 
>>> from the
>>>    backed by its name. Maybe we need to create it, and state that 
>>> this is for
>>>    admin purposes only and shouldn't be called by the user-level 
>>> code (just
>>>    like the listAllShares method from Horde_Share_Sql).
>>>    Currently the pref_name column is not indexed, so we expect slow 
>>> queries.
>>>    The fix for that is obvious.
>>   The problem is that not all backends support listing preference 
>> details for others than the current user. On the other hand, those 
>> backends are already missing some features anyway, and we already 
>> have a listScopes() method that is only implemented in a few backends 
>> too. In the end this might be an option.
>>   Actually, since the same script already requires the preference 
>> backend to return any user's preference, this would be a safe approach.
>>
>>   Another approach would be do take a further look at why 
>> listAllShares() exceeds memory. Well, there is not much to look 
>> actually when creating 350.000+ share objects and then attempting to 
>> sort them. Alternatively we could implement an Iterator interface in 
>> the share drivers and only receive the shares one by one while 
>> looping them.
>>
>>   Jan.
>>
>>   --
>>   The Horde Project
>> http://www.horde.org/
>>
>>
>>   --
>>   Horde developers mailing list
>>   Frequently Asked Questions: http://horde.org/faq/To unsubscribe, 
>> mail: dev-unsubscribe at lists.horde.org
> If you iterate the shares one by one i think you will end up with a
> similar problem, because you can't send an agenda before you have all the
> events from all the calendars that a user has access to. So you would
> probably end up again with the 350.000+ shares on memory.
>
> Will try to produce a patch and submit for your appreciation.

Jan, attached is a first preview patch, so you can see if things are 
going in the desired direction.
Returning a class that implements the ArrayAccess allows backward 
compatibility, but unfortunately, any array_ like function (array_merge, 
array_keys, etc) don't work, so a bit more of refactor is needed.

One thing that needs to be done is move the Horde_Share_List factory to 
an injector (if I understand the injectors correctly), but for now I 
instantiated the objects inside the Horde_Share_Sql directly.

Another problem I found was that Horde_Shares can have callbacks for 
listings, so for now I create the new Horde_Share_List object and if the 
callback is active, that object is sent to the callback. This might 
brake up for users that rely on array_ like functions, because as stated 
above they wont work now.

What do you devs think?
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: share_list_patch.txt
URL: <http://lists.horde.org/archives/dev/attachments/20120322/fcabe0f4/attachment-0001.txt>