[imp] Excessive memory usage

[Michael M Slusarz]

Quoting Andrew Morgan <morgan at orst.edu>:

> I'm using the latest RC releases of Horde and IMP.
>
> One of my users complained that he couldn't view a large message.   
> The message has 11 Appledouble JPEG image attachments that are about  
> 2.5MB each.  When he clicked on the message to view it, all he got  
> was a blank screen after a short while.
>
> The message size is 34MB, and I found that I cannot display the  
> message until I increase the PHP memory limit to 250MB.
>
> The message parts look like this (from Alpine):
>
>     [ Part 2.1, Application/APPLEFILE (Name: "IMG_1210.JPG") 98 KB. ]
>     [ Not Shown. Use the "V" command to view or save this part. ]
>
>     [ Part 2.2, Image/JPEG (Name: "IMG_1210.JPG") 2.4 MB. ]
>     [ Not Shown. Use the "V" command to view or save this part. ]
>
> Also, it doesn't generate a thumbnail for any of the images and  
> clicking on the "view thumbnail" link in its place open a new window  
> but it is blank.
>
> If I send myself a new message with the same exact images, but not  
> in Appledouble wrappers, then it can display the message and the  
> thumbnails correctly even with the PHP memory limit set to 100MB  
> (our production setting).
>
> Looking through the Appledouble viewer code in IMP, I'm guessing  
> that there are simply too many copies of the same data being made,  
> bloating the memory size.
>
> Is this a known issue/bug?

It's a known issue in that our current MIME handling regime isn't the  
most efficient in handling what I like to call "embedded" MIME data.   
"Embedded" data is defined as any MIME part that contains data that  
can't be viewed until it itself is translated/decoded somehow (this is  
in addition to the original MIME decoding).  Additionally, the  
c-client library is known to be memory hungry when handling message  
part data so this adds to the problem.

As for the c-client issue, this problem is presently being worked on  
(a fully PHP stream-based IMAP client has nearly reached beta stage  
and will be used in the forthcoming IMP 5).  The MIME handling is also  
slated to be improved.  But the major issue in your case is that  
nobody (and I mean nobody) should be sending attachments using  
appledouble.  There is nothing gained by using appledouble that can't  
be handled by a MIME multipart/mixed message.  I realize that you  
can't control what mail programs people are using, but at some point  
you *do* have to say "I'm sorry, we just can't handle that message."

And before you think this is an IMP specific issue, I have several  
test e-mail messages that are 40 MB or so that when I view with  
Thunderbird (on my Quad Core Q6600 w/ 3 GB of RAM), it absolutely  
brings my machine to its knees.  This is an issue with the mail  
transport architecture in general, not just Horde/IMP.

michael

-- 
___________________________________
Michael Slusarz [slusarz at horde.org]