[imp] Excessive memory usage

[Michael M Slusarz]

Quoting Matus UHLAR - fantomas <uhlar at fantomas.sk>:

>> >On 24.09.08 22:08, Michael M Slusarz wrote:
>> >>As for the c-client issue, this problem is presently being worked on
>> >>(a fully PHP stream-based IMAP client has nearly reached beta stage
>> >>and will be used in the forthcoming IMP 5).
>
>> Quoting Matus UHLAR - fantomas <uhlar at fantomas.sk>:
>> >Should we expect imp to use php socket connection insead of php imap
>> >functions, or did i misunderstand that?
>
> On 25.09.08 02:26, Michael M Slusarz wrote:
>> Yes.  See here: http://cvs.horde.org/incubator/Imap_Client
>
> Do you also expect it to be reasonably fast and not too CPU-intensive than
> binary c-client?

Current tests indicate the PHP socket code is faster than c-client.   
This is due in part to the fact that c-client is extremely inefficient  
when it comes to certain operations.  For example, to do an overview  
fetch of a list of X # of messages takes a single call in the socket  
code and takes X calls for c-client, and each c-client call downloads  
info on that message and the next 20 messages (don't ask why).

But more important, having the code in PHP gives us complete control  
in terms of features, bug control, error reporting, etc.  Even if it  
were more CPU-intensive, these tradeoffs would be worth it.

>> >Could some memory-effective libraries, binary or PHP-bases, help there?
>> >(e.g. effective MIME decoding/encoding)
>
>> MIME processing is always going to be somewhat memory intensive.
>> Streams *may* be a solution to limit memory when it comes to the body
>> part of the message. however, at some point, that entire message is
>> going to be in memory at some point (with the possible exception of
>> streaming message part data when downloading a message).
>
> I understand that it will be memory-hungry. I just think it's often TOO
> hungry. needing 128MB of memory to process 20MB mail is too much imho.

I agree in principle, but everyone needs to realize (as I have  
mentioned countless times before) that PHP is not designed to be able  
to easily manage memory.  This is not C where you can analyze malloc()  
calls accordingly.

> I prefer binary packages (if they are properly coded) - they should be
> faster than PHP (no matter how effective is the zend engine).

But there is all sorts of issues with binary packages (see, e.g., c-client).

> hmmm when I had the memory problems, I noticed than php failed in functions
> that were doing base64 encoding... something that really could be done in
> binary (and there is PECL mime library iirc)

Base64 encoding *is* done in binary - the PHP function  
base64_encode().  If you think that function is too memory intensive,  
then bug the PHP people and have them optimize it.

michael

-- 
___________________________________
Michael Slusarz [slusarz at horde.org]