[imp] Excessive memory usage

[Andrew Morgan]

On Wed, 24 Sep 2008, Andrew Morgan wrote:

> I'm using the latest RC releases of Horde and IMP.
>
> One of my users complained that he couldn't view a large message.  The 
> message has 11 Appledouble JPEG image attachments that are about 2.5MB each. 
> When he clicked on the message to view it, all he got was a blank screen 
> after a short while.
>
> The message size is 34MB, and I found that I cannot display the message until 
> I increase the PHP memory limit to 250MB.
>
> The message parts look like this (from Alpine):
>
>    [ Part 2.1, Application/APPLEFILE (Name: "IMG_1210.JPG") 98 KB. ]
>    [ Not Shown. Use the "V" command to view or save this part. ]
>
>    [ Part 2.2, Image/JPEG (Name: "IMG_1210.JPG") 2.4 MB. ]
>    [ Not Shown. Use the "V" command to view or save this part. ]
>
> Also, it doesn't generate a thumbnail for any of the images and clicking on 
> the "view thumbnail" link in its place open a new window but it is blank.
>
> If I send myself a new message with the same exact images, but not in 
> Appledouble wrappers, then it can display the message and the thumbnails 
> correctly even with the PHP memory limit set to 100MB (our production 
> setting).
>
> Looking through the Appledouble viewer code in IMP, I'm guessing that there 
> are simply too many copies of the same data being made, bloating the memory 
> size.

Following up on my own post...

Whether using Appledouble junk or not, each attachment ends up stored in 
memory twice.

This function in imp/lib/MIME/Contents.php is the culprit:

     function getBodyPart($id)
     {
         if (!isset($this->_bodypart[$id])) {
             $imp_imap = &IMP_IMAP::singleton();
             $imp_imap->changeMbox($this->_mailbox, IMP_IMAP_AUTO);
             $this->_bodypart[$id] = imap_fetchbody($imp_imap->stream(), $this->_index, $id, FT_UID | FT_PEEK);
         }

         return isset($this->_bodypart[$id]) ? $this->_bodypart[$id] : '';
     }


As far as I can tell, the cached bodypart is never used.  I added an 
"else" clause in that function to log anytime a cached version was used. 
I looked at a wide variety of messages and message types, but I never saw 
a cached version used.

It seems like the cache would be a good idea, but it isn't being used by 
the higher level functions.  Perhaps the mime parts should only be stored 
if the cache system is enabled?

This isn't a big deal on small messages, but it's a big deal on big 
messages.  The amount of memory required doubles.

 	Andy