[imp] Crazy memory usage when viewing attachments

[Chuck Hagenbuch]

Quoting Kevin Konowalec <webadmin at ualberta.ca>:

> We were running into pretty consistent runaway httpd children when  
> users would download large (> 30 MB) attachments.  Memory usage  
> would balloon to at least 4 times the size of the attachment and the  
> process would monopolize the CPU for literally several minutes.   
> (This was after removing or sufficiently increasing various Horde,  
> PHP, and system-level limits as the httpd childs would segfault  
> previously.)
>
> After tracing through way too much PHP, two lines looked suspicious:
>
>
> --- horde/lib/Horde/MIME/Part.php.bak Wed Jul  5 10:30:47 2006
> +++ horde/lib/Horde/MIME/Part.php     Tue Nov 21 03:26:12 2006
> @@ -1152 +1152 @@
> -            $message = preg_replace("/=\r?\n/", '', $this->_contents);
> +//            $message = preg_replace("/=\r?\n/", '', $this->_contents);
>
> --- horde/imp/lib/MIME/Contents.php.bak       Thu May  4 21:30:21 2006
> +++ horde/imp/lib/MIME/Contents.php   Tue Nov 21 03:26:26 2006
> @@ -173 +173 @@
> -            $this->_bodypart[$id] = str_replace("\r\n", "\n",  
> $this->_bodypart[$id]);
> +//            $this->_bodypart[$id] = str_replace("\r\n", "\n",  
> $this->_bodypart[$id]);
>
>
>
> These calls were being made for any and all attachments regardless  
> of encoding or MIME type.  Completely commenting out both lines  
> immediately solved the runaway processes.  A 100 MB JPEG attachment  
> took under 10 seconds of processing before download began compared  
> to over 10 minutes before (but it still used just over 400 MB of  
> memory).
>
> Where N is the size of the attachment, I'm gonna take a stab in the  
> dark and assume the 4N memory requirement is most likely attributed  
> to each byte in the attachment being represented as a 32-bit value.   
> As for the CR+LF regexps, I admit I haven't gone through the RFC,  
> but I'm guessing this isn't explicitly necessary for the majority of  
> non-text attachments especially when they're a base64 blob.  So far,  
> plain text, JPEG, TIFF, and random binary blobs have downloaded  
> without incident.
>
> So can anyone suggest a reason why we shouldn't be doing this?  Or  
> can we go ahead and comment these lines out in our production servers?

I'm guessing this is necessary for some types, but I agree it  
shouldn't be necessary for all of them. Can you please add this to  
bugs.horde.org?

Thanks!

-chuck

-- 
"we are plastered to the windshield of the bus that is time." - Chris