[imp] Crazy memory usage when viewing attachments

[Kevin Konowalec]

Horde: 3.1.2
Imp: H3 (4.1.2)
Turba: H3 (2.1.1)

We were running into pretty consistent runaway httpd children when  
users would download large (> 30 MB) attachments.  Memory usage would  
balloon to at least 4 times the size of the attachment and the  
process would monopolize the CPU for literally several minutes.   
(This was after removing or sufficiently increasing various Horde,  
PHP, and system-level limits as the httpd childs would segfault  
previously.)

After tracing through way too much PHP, two lines looked suspicious:


--- horde/lib/Horde/MIME/Part.php.bak Wed Jul  5 10:30:47 2006
+++ horde/lib/Horde/MIME/Part.php     Tue Nov 21 03:26:12 2006
@@ -1152 +1152 @@
-            $message = preg_replace("/=\r?\n/", '', $this->_contents);
+//            $message = preg_replace("/=\r?\n/", '', $this- 
 >_contents);

--- horde/imp/lib/MIME/Contents.php.bak       Thu May  4 21:30:21 2006
+++ horde/imp/lib/MIME/Contents.php   Tue Nov 21 03:26:26 2006
@@ -173 +173 @@
-            $this->_bodypart[$id] = str_replace("\r\n", "\n", $this- 
 >_bodypart[$id]);
+//            $this->_bodypart[$id] = str_replace("\r\n", "\n",  
$this->_bodypart[$id]);



These calls were being made for any and all attachments regardless of  
encoding or MIME type.  Completely commenting out both lines  
immediately solved the runaway processes.  A 100 MB JPEG attachment  
took under 10 seconds of processing before download began compared to  
over 10 minutes before (but it still used just over 400 MB of memory).

Where N is the size of the attachment, I'm gonna take a stab in the  
dark and assume the 4N memory requirement is most likely attributed  
to each byte in the attachment being represented as a 32-bit value.   
As for the CR+LF regexps, I admit I haven't gone through the RFC, but  
I'm guessing this isn't explicitly necessary for the majority of non- 
text attachments especially when they're a base64 blob.  So far,  
plain text, JPEG, TIFF, and random binary blobs have downloaded  
without incident.

So can anyone suggest a reason why we shouldn't be doing this?  Or  
can we go ahead and comment these lines out in our production servers?



On Nov 23, 2006, at 1:09 PM, Michael M Slusarz wrote:

> Quoting patrick <gibblertron at gmail.com>:
>
>> I'm noticing that IMP requires an incredible amount of RAM to display
>> a message with a relatively large attachment. For example, on of our
>> customers was complaining that she couldn't open a message that was
>> was 1.4MB in size. Our memory limit was set to 50MB, and Apache was
>> running out of memory when viewing this message. Increasing it to  
>> 64MB
>> seemed to solve it, but I'm trying to comprehend what in IMP is
>> causing it to required more than 35 times the size of the message in
>> order to view it?
>
> Any code suggestions on reducing memory usage would be much  
> appreciated.  But remember the following:
> 1.) PHP is a high-level language, and was not designed for  
> programmers to easily manage memory usage.  mallocs, etc. are  
> handled by the interpreter and is transparent to a PHP script.  So  
> complaining about memory usage, without providing us pointers on  
> how we (as PHP coders) can reduce memory usage, is not tremendously  
> helpful.
> 2.) You don't think that, for example, a windows program uses a  
> similar amount of memory to view a message?  Try running  
> thunderbird and looking at memory usage when viewing a 1.5 mb message.
>
> michael
>
> ___________________________________
> Michael Slusarz [slusarz at horde.org]
>
> -- 
> IMP mailing list - Join the hunt: http://horde.org/bounties/#imp
> Frequently Asked Questions: http://horde.org/faq/
> To unsubscribe, mail: imp-unsubscribe at lists.horde.org
>