[imp] Cannot download large folder

[Andrew Morgan]

On Tue, 17 Jul 2007, Chuck Hagenbuch wrote:

> Quoting Andrew Morgan <morgan at orst.edu>:
>
>>>> A more useful question might be, "Why does it require so much memory
>>>> to download a folder and how can the memory footprint be reduced?"
>>>> Note - I can trigger this error on my system also with a mailbox
>>>> that is approximately 1/3 my php memory limit.
>>> 
>>> If someone wants to trace this a bit, that'd be great.
>> 
>> Can you give me some directions on how to get started tracing memory usage? 
>> I've never traced a PHP application.  :)
>
> http://us.php.net/memory-get-usage
>
> If that's not enough, xdebug is the next step (http://xdebug.org/).

That was easy!

I added some simple debug statements to print out the memory usage at 
various steps in imp/lib/Folder.php.  I printed out the memory usage after 
each message was retrieved inside the 'for' loop, at the end of the loop, 
and after the str_replace.  Here were my results:

-----------------------------------------------------------------
Jul 17 13:25:40 webmail4 HORDE[1819]: [imp] Memory allocated: 7661556 [on 
line 603 of "/var/www-test/horde/imp/lib/Folder.php"]

... Many messages go by with memory usage increasing gradually according 
to the size of the message ...

Jul 17 13:26:23 webmail4 HORDE[1819]: [imp] Memory allocated: 51898876 [on 
line 603 of "/var/www-test/horde/imp/lib/Folder.php"]
Jul 17 13:26:23 webmail4 HORDE[1819]: [imp] Memory allocated after loop: 
51899260 [on line 610 of "/var/www-test/horde/imp/lib/Folder.php"]
-----------------------------------------------------------------

At this point it apparently was stopped for exceeding the php memory limit 
(100MB on my system).


Here are the results for a smaller folder that does not exceed the memory 
limit:

-----------------------------------------------------------------
Jul 17 13:33:15 webmail4 HORDE[2427]: [imp] Memory allocated: 7634344 [on 
line 603 of "/var/www-test/horde/imp/lib/Folder.php"]

... etc etc ...

Jul 17 13:33:17 webmail4 HORDE[2427]: [imp] Memory allocated: 18452336 [on 
line 603 of "/var/www-test/horde/imp/lib/Folder.php"]
Jul 17 13:33:17 webmail4 HORDE[2427]: [imp] Memory allocated after loop: 
18452756 [on line 610 of "/var/www-test/horde/imp/lib/Folder.php"]
Jul 17 13:33:17 webmail4 HORDE[2427]: [imp] Memory allocated after 
str_replace: 18453140 [on line 613 of "/var/www-test/horde/imp/lib/Folder.php"]

-----------------------------------------------------------------


I'm almost positive that the memory usage doubles temporarily for this 
function call:

           $body = str_replace("\r\n", "\n", $body);

However, I was able to avoid this memory usage penalty by calling 
str_replace within the loop on each message, rather than the entire huge 
string at the end.  This also uses less memory overall.  I suspect that 
PHP memory routines don't "cleanup" memory very well when the size of a 
variable is reduced as it is in the str_replace() call above.

I've attached a patch which implements this change.


However, it still fails if you try to download a zipped folder.  Again, 
this is probably caused by the temporary need for twice the memory in 
order to make a working copy of the folder.  Unfortunately I don't see any 
way around that.

In general, you'll need to set the PHP memory limit to twice the size of 
your largest folder download, plus some overhead (approximately 10-15MB).

 	Andy
-------------- next part --------------
--- lib/Folder.php.orig	2007-01-02 05:54:56.000000000 -0800
+++ lib/Folder.php	2007-07-17 14:12:00.000000000 -0700
@@ -596,13 +596,12 @@
                        require a space in front of single digit days. */
                     $date = sprintf('%s %2s %s', date('D M', $h->udate), date('j', $h->udate), date('H:i:s Y', $h->udate));
                     $body .= 'From ' . $from . ' ' . $date . "\n";
-                    $body .= imap_fetchheader($_SESSION['imp']['stream'], $i, FT_PREFETCHTEXT);
-                    $body .= imap_body($_SESSION['imp']['stream'], $i, FT_PEEK) . "\n";
+                    $body .= str_replace("\r\n", "\n", imap_fetchheader($_SESSION['imp']['stream'], $i, FT_PREFETCHTEXT));
+                    $body .= str_replace("\r\n", "\n", imap_body($_SESSION['imp']['stream'], $i, FT_PEEK) . "\n");
                 }
             }
         }
 
-        $body = str_replace("\r\n", "\n", $body);
         return $body;
     }