[Tickets #5913] Re: Lower memory usage while downloading files

[bugs at bugs.horde.org]

DO NOT REPLY TO THIS MESSAGE. THIS EMAIL ADDRESS IS NOT MONITORED.

Ticket URL: http://bugs.horde.org/ticket/?id=5913
-----------------------------------------------------------------------
 Ticket             | 5913
 Updated By         | olger.diekstra at cardno.com.au
 Summary            | Lower memory usage while downloading files
 Queue              | Gollem
 Version            | HEAD
 Type               | Enhancement
 State              | Feedback
 Priority           | 2. Medium
 Owners             | Chuck Hagenbuch
+New Attachment     | view.php
-----------------------------------------------------------------------


olger.diekstra at cardno.com.au (2007-12-03 16:23) wrote:

Hi Chuck,

I'm more than happy to help make it work for every filesystem. Took me a
while to figure out how to get files out of the cvs system though, but I've
got the files you modified. (btw, running Horde 3.1.5 and Gollem H3 1.0.3
for testing).
Tested your modifications, but no joy. Tried downloading a 380MB file,
with various memory settings, all the way up to 800MB memory_limit. Keeps
eating the memory.

The thing is though, in the gollem/view.php file there is a line:
         $data = $GLOBALS['gollem_vfs']->read($filedir, $filename);
which reads everything that's returned from the "read ( , )" function into
a variable (in memory, which is a local resource!).
In the function "read( , )" you use another variable to read the file into
a variable, so basically your doubling the memory need to be able to
download a file.

Memory is a local resource. It doesn't care if it came from SQL, FTP, SMB.
It's stored in local memory in the machine serving the browser. As opposed
to copying the entire file into memory, why not copy it to disk (temp
directory comes to mind)? (I've used a hardcoded '/tmp/' for now, but its
easy enough to get the system variable).
Using my previously ReadFileChunked function, that would be easy and
compatible with any backend. It would need some additional checking I
suppose, but this illustrates the basic idea (gollem/view.php), see
attachment.

For some reason it doesn't copy the file from the VFS to the local
filesystem, but you may know why.

I think it gets the basic idea across.

Cheers! Olger.


> I know that you are trying to be helpful and I appreciate that. But 
> at some point you need to start to understand the codebase that you 
> are working with.
>
> What your changes do is make sure that Gollem can't work with any 
> backend except a local filesystem. I hope it's obvious that that's 
> not an acceptable change.
>
> My commits were to the CVS version of Horde and Gollem (Horde is at 
> 3.2-RC1, Gollem will be released as Gollem 1.1 once Horde 3.2 is 
> released - see http://wiki.horde.org/ReleaseManagement). If you are 
> interested in helping us in a way that can be incorporated into the 
> code and that will help all Gollem users, you need to test that 
> version as well. You can get snapshots from http://snaps.horde.org/.
>
> You should be able to just drop in the new file.php and view.php 
> files as well, but since I'm not sure which version you mean by 
> "latest and greatest", I can't guarantee that. Horde 3.2 is backwards 
> compatible with Horde 3.0, though.