[Tickets #3359] Re: Make use of streams for speed and memory efficiency

[bugs at horde.org]

DO NOT REPLY TO THIS MESSAGE. THIS EMAIL ADDRESS IS NOT MONITORED.

Ticket URL: http://bugs.horde.org/ticket/3359
------------------------------------------------------------------------------
  Ticket             | 3359
  Updated By         | Michael Slusarz <slusarz at horde.org>
  Summary            | Make use of streams for speed and memory efficiency
  Queue              | Horde Framework Packages
  Version            | HEAD
  Type               | Enhancement
  State              | Resolved
  Priority           | 2. Medium
  Milestone          | Horde 4.0
  Patch              |
  Owners             | Michael Slusarz
------------------------------------------------------------------------------


Michael Slusarz <slusarz at horde.org> (2009-07-01 03:02) wrote:

> Nice! Do you happen to have any numbers about how this works out,  
> either speed or memory_limit needs or ... ?

Not really.  In general terms - using streams in the Imap library  
(when grabbing body parts/text) eliminates the need to read (some of)  
this data into memory.  As it stands, data under 2 MB is read into  
memory.  Larger files will be spooled to disk above that point.  So  
for smaller data, there will probably be no gain (and my guess is that  
performance might slightly be reduced due to the stream overhead code)  
but for larger files we cap memory usage at 2 MB.

The bigger gains come in Horde_Mime_Part.  For large body parts, we  
can pass the stream object we grab from Imap_Client directly to  
Horde_Part.  If possible, we can reuse this stream without any further  
action (if the data is already in 7bit, 8bit or binary format).  If  
not, we have to copy this stream to a new stream while running through  
a decompression filter.  Again, we are using temp streams so the  
maximum memory usage during this activity should be approx. 4 MB (2 MB  
for each of the streams), and we can easily free up the 2 MB for the  
one stream after we are finished.

Certain memory bottlenecks unfortunately still require strings and  
there is not much we can do about this (DB/VFS still require strings,  
so this doesn't help memory usage for large attachments; Mail doesn't  
allow us to pass a stream so we have to create the entire text of the  
message before we can send).

Right now, my focus is more on making sure I didn't break anything too badly.