[horde] Allowed memory exhausted in Horde/Imap/Client/Ids.php:178

[Bjoern Voigt]

Jan Schneider wrote:
> Zitat von Bjoern Voigt <bjoernv at arcor.de>:
>
>> I can't login in my Horde 5.2 (current) account anymore.
>>
>> The error message is:
>>
>>     Allowed memory size of 134217728 bytes exhausted (tried to allocate
>>     7783056 bytes)
>>
>>     1. Horde_ErrorHandler::catchFatalError()
>>
>>     Details
>>
>>     [...]
>>
>>     ErrorException Object
>>     (
>>         [message:protected] => Allowed memory size of 134217728 bytes
>> exhausted (tried to allocate 7783056 bytes)
>>         [string:Exception:private] =>
>>         [code:protected] => 0
>>         [file:protected] => /usr/share/php/Horde/Imap/Client/Ids.php
>>         [line:protected] => 179
>>         [trace:Exception:private] => Array
>>             (
>>                 [0] => Array
>>                     (
>>                         [function] => catchFatalError
>>                         [class] => Horde_ErrorHandler
>>                         [type] => ::
>>                         [args] => Array
>>                             (
>>                             )
>>
>>                     )
>>
>>             )
>>
>>         [previous:Exception:private] =>
>>         [severity:protected] => 1
>>         [logged] => 1
>>     )
>>
>> I debugged this. The crash occurs here:
>>
>>                 if (!$this->duplicates) {
>>
>>                     Horde::log(sprintf('IMAP IDs %d ids=%s.',
>> count($this->_ids), implode(",", $this->_ids)), 'ERR');
>>
>>                     $this->_ids = (count($this->_ids) > 25000)
>>
>>                         ? array_unique($this->_ids)
>>
>>                         : array_keys(array_flip($this->_ids));
>>
>>                 }
>>
>> Increasing Apache/PHP memory does not solve the problem. Currently I
>> have increased PHP memory from 128 to 512 MB.
>>
>> During crash the variable $ids has the value
>> "1:486128,486130,486132:486327,486330:486331,486334:486335,486347,486352:486353,486355,486361:486366,486432:486514,486518:486522,486526:486528,486560:486569".
>>
>>
>> $this->_ids is a very long array starting with 1,2,3 and ending with
>> 45005,45006,45007.
>>
>> Both variables are NOT from the same login.
>>
>> I think the "1" in $ids causes the problem, because all IDs between 1
>> and 486128 (1,2,3...486128) make the array so big.
>>
>> Currently I can login again. But I am unsure, if this problem comes
>> again.
>>
>> Greetings,
>> Björn
>
> Any chance that you had this happening while a login task to empty the
> trash (or spam) folder was running?
After some hours of testing and debugging, I have some results.

I tested with "tcpdump" (with IMAP protocol filter), debugging code and
some alternative "array_unique" PHP code.

The results:

 1. I could solve the problem for one account by removing and
    re-creating the Trash folder.
    The UIDs of the old Trash folder started with 1, 483854, 483855,
    483856 ...
 2. It's not easy to find the "broken" IMAP folder, because the crashing
    function has no direct access to data like username or folder name.
 3. None of the functions array_unique or array_keys(array_flip(...))
    was able to handle an array with more than around 500000 elements.
    Even if I removed the functions, the array handler functions crashed
    later in code. Memory is not the only problem. The access time
    decreases too, if big arrays should be sorted.
 4. Imapproxy had no influence to this problem. Dovecot directly caused
    the problems too.
 5. A denial-of-service attack is possible, if a lot of users create
    special IMAP folders with big UID ranges. The Horde/IMP code may
    crash or PHP may use a lot of memory.

A real fix isn't easy. I think, it's possible to change the PHP code so,
that not single UID elements are stored and sorted (e.g. 1, 2,
3,...,483854, 483855, 483856), but ranges (e.g. 1:483856).

Greetings,
Björn