[Tickets #4753] Attachment rewrites causing HUGE memory bloat

[bugs@bugs.horde.org]

DO NOT REPLY TO THIS MESSAGE. THIS EMAIL ADDRESS IS NOT MONITORED.

Ticket URL: http://bugs.horde.org/ticket/?id=4753
-----------------------------------------------------------------------
 Ticket             | 4753
 Created By         | webadmin at ualberta.ca
 Summary            | Attachment rewrites causing HUGE memory bloat
 Queue              | IMP
 Version            | 4.1.2
 Type               | Bug
 State              | Unconfirmed
 Priority           | 3. High
 Owners             | 
-----------------------------------------------------------------------


webadmin at ualberta.ca (2006-12-07 13:12) wrote:


We were running into pretty consistent runaway httpd children when users
would download large (> 30 MB) attachments.  Memory usage would balloon to
at least 4 times the size of the attachment and the process would
monopolize the CPU for literally several minutes.  (This was after
removing or sufficiently increasing various Horde, PHP, and system-level
limits as the httpd childs would segfault previously.)  We also found that
even moderately sized attachments (>14MB) would monopolize the CPU for
several minutes and if more than one medium to large sized attachments
were downloaded at the same time from the same server the browser would
time out before the download to the client even began.

After tracing through way too much PHP, two lines looked suspicious:


--- horde/lib/Horde/MIME/Part.php.bak Wed Jul  5 10:30:47 2006
+++ horde/lib/Horde/MIME/Part.php     Tue Nov 21 03:26:12 2006
@@ -1152 +1152 @@
-            $message = preg_replace("/=\r?\n/", '', $this->_contents);
+//            $message = preg_replace("/=\r?\n/", '', $this->_contents);

--- horde/imp/lib/MIME/Contents.php.bak       Thu May  4 21:30:21 2006
+++ horde/imp/lib/MIME/Contents.php   Tue Nov 21 03:26:26 2006
@@ -173 +173 @@
-            $this->_bodypart[$id] = str_replace("\r\n", "\n",
$this->_bodypart[$id]);
+//            $this->_bodypart[$id] = str_replace("\r\n", "\n",
$this->_bodypart[$id]);



These calls were being made for any and all attachments regardless of
encoding or MIME type.  Completely commenting out both lines immediately
solved the runaway processes.  A 100 MB JPEG attachment took under 10
seconds of processing before download began compared to over 10 minutes
before (but it still used just over 400 MB of memory).

Where N is the size of the attachment, I'm gonna take a stab in the dark
and assume the 4N memory requirement is most likely attributed to each
byte in the attachment being represented as a 32-bit value.  As for the
CR+LF regexps, I admit I haven't gone through the RFC, but I'm guessing
this isn't explicitly necessary for the majority of non-text attachments
especially when they're a base64 blob.  So far, plain text, JPEG, TIFF,
and random binary blobs have downloaded without incident.