[imp] Cannot download large folder

[Otto Stolz]

Hello,

Andrew Morgan had written:
> I'm almost positive that the memory usage doubles temporarily for this 
> function call:
>           $body = str_replace("\r\n", "\n", $body);

More than that, much more. A customer of mine could not download
his 81 MB folder in 512 MB memory. (This has started this thread.)

> However, I was able to avoid this memory usage penalty by calling 
> str_replace within the loop on each message, rather than the entire 
> huge string at the end.  This also uses less memory overall.

I have written:
> I have modified my Imp H3 (4.1.3) test installation (under PHP 5.2.0
> and openSUSE 10.2 (i586)) in the way outlined by Andrew. Then I have
> logged both the MBox size and the Memory Peak Size, as the Mbox was
> being assembled.
> 
> With one particular folder (size: 37.6 MB), I found
>   M ≤ 2.1 × b + 8.6 MB
> where:
>   M : Memory peak size
>   b : Size of Mbox assembled so far
> 
> Interestingly, there are plateaus in the curve, where generateMbox
> will not claim more memory as the Mbox is growing, but then the
> Memory usage will jump up to the original straight line, cf. attached
> diagram.
> 
> With two smaller mboxes, I have found:
>   M ≤ 3,0 × b + 8.3 MB  ( b ≤ 15.4 MB)
>   M ≤ 3.0 × b + 8.6 MB  ( b ≤ 14.3 MB)

Now, for a final test, my customer has successfully downloaded his 81 MB
folder, and I have included the pertinent data in the attached diagram.
It is very similar to the first one mentioned above: Again the plateaus,
and the edges lie almost on a straight line:
    M ≤ 2.0 × b + 9.9 MB  ( b ≤ 78.2 MB)

So I reckon, with Andrew’s patch (thanks again!), we can state the
following rule of thumb:
   For downloading real large folders, the memory limit should be
   at least 2 × folder size + 10 MB.

Best wishes,
   Otto Stolz
-------------- next part --------------
A non-text attachment was scrubbed...
Name: generateMbox.png
Type: image/png
Size: 8006 bytes
Desc: not available
Url : http://lists.horde.org/archives/imp/attachments/20070723/04fdc1d4/attachment.png