[imp] Persistent IMAP connections

[Egan]

On Mon, 21 May 2001 22:24:39 -0400, Chuck Hagenbuch <chuck@horde.org>
wrote:

>> After searching the archives I see there has been some discussion of
>> possibly using PHP4 sessions to create persistent IMAP connections.
>
>Where did you see this? I don't recall any conversations along these lines.


Some of it goes back almost two years, to the PHP3 days.  This gets
long, even after distilling out all the extraneous material.  But you
asked for it.  So here we go.  :-)


>Subject:  Re: [imp] IMAPD server overload
>From:     "Ronald F. Guilmette" <rfg@monkeys.com>
>Date:     1999-10-22 8:42:44
>
>(I have to confess, I'm getting out of my depth here.  I don't know nada
>about how IMP manages sessions.  I'm just assuming that some sort of session
>key gets created on the server and then jammed down to the browser, and
>then the server gets that back everytime the user does something.  Yes?)
>
>Anyway, if that is indeed what happens, then it is just a matter of ...
>using that session key as a request qualifier with every IMAP request
>and then having all of the IMP code send all of its IMAP requests to a
>sort of intermediate persistant agent/proxy


You responded to Ronald:


>Subject:  Re: [imp] IMAPD server overload
>From:     Chuck Hagenbuch <chagenbu@wso.williams.edu>
>Date:     1999-10-23 4:39:59
>
>> Of course, the agent/proxy thingy I'm speaking of would itself persistantly
>> hold open the connection(s) to the IMAP server(s), thus eliminating all of
>> that startup/shutdown overhead.
>
>Yeah, that'd be nice. Someone want to write it?
>
>-chuck


Then Ronald said:


>Subject:  Re: [imp] IMAPD server overload
>From:     "Ronald F. Guilmette" <rfg@monkeys.com>
>Date:     1999-10-23 18:31:19
>
>Ummm... well... since you asked...
>
>As it happens, I myself have written a bunch of ``server'' type programs
>like this, i.e. just the kind that might solve the fundamental performance
>problem that was mentioned, i.e. too much forking.
>
>I've written serveral server type thingies that do not fork and yet handle
>numerous/multiple clients at a time.  (If your OS has proper threads
>support, doing this is a piece of cake.  But even implementing such a
>server without any threads support... as I have done, several times now...
>is not really very difficult, once you get the hang of it.)


And another contributor said in response to that:


>Subject:  [imp] IMAP connection cache (was RE: [imp] IMAPD server overload)
>From:     "Mayers, 112106" <p.mayers@ic.ac.uk>
>Date:     1999-10-24 16:56:11
>
>Erm... We don't really want that kind of server (excessive forking is *not*
>the problem - it's the connection setup overhead, at one IMAP connection per
>PHP script run). What we want is some way to proxy IMAP requests. That proxy
>would keep the connection open.
>
>Unfortunately, (and a lot of people seem to have missed this) this would
>require c-client to be modified as well. You'd have to give the proxy some
>way of knowing *which* one of the open IMAP connections you're referring to.
>To do that requires changing the IMAP 'over the wire' protocol slightly (not
>a *lot*, but slightly) and to do *that* requires modding c-client. c-client
>would also require a sharedmem area containing open cached connections,
>*and* you'd require some changes to PHP and IMP as well.


At first, I also thought of modifying c-client, but now I realize that
adding a PHP session request qualifier to each request will eliminate
the need to modify c-client.

Later, in February of this year:


>Subject:  Re: [imp] imp causes high load ?
>From:     Rich Lafferty <rich@horde.org>
>Date:     2001-02-06 17:25:25
>[Download message RAW]
>
>> Since we are beating this one.
>> Why doesnt IMP keep a persistent connection?
>
>Mostly because no-one's written it to do so yet. :-) It's not trivial,
>though -- the simplest problematic question is "When do you close the
>connection". 


And you said:


>Subject:  Re: [imp] imp causes high load ?
>From:     Chuck Hagenbuch <chuck@horde.org>
>Date:     2001-02-06 17:28:07
>
>> I know Netscape Messanger can keep persistent connections.
>> Is it a limitation from the c-client library?
>
>Netscape Messenger is an application that stays in memory for the lifetime that 
>it is run. IMP is a php script. To keep a persistant connection, apache would 
>need to hold on to it for us. However, there's no way that I know of to have a 
>global pool of resources shared between all apache children, and if persistant 
>connections weren't shared between children, then you'd get one connection per 
>user per apache process, which would break things horribly and be even worse 
>than we have now.
>
>-chuck


And Atif replied:


>Subject:  Re: [imp] imp causes high load ?
>From:     Atif Ghaffar <aghaffar@developer.ch>
>Date:     2001-02-06 18:04:05
>
>How about one connection per session?
>The connection goes away when the session goes away.
>If the user does not logout, the connection is closed with a timeout.


And more recently, another contributor added:


>Subject:  [imp] [IMP] reducing imapd cruelty?
>From:     Vasilios Hoffman <zeus@olympus.phys.wesleyan.edu>
>Date:     2001-05-13 0:11:07
>
>	A question for you all.  Other than using cyrus imapd, or
>distributing services as the scalable webmail howto suggests, are there
>any suggestions for reducing the number of imapds spawned and/or the disk
>I/O created by IMP?
>
>Currently, imapd is killing our setup -- almost every time a student here
>loads up a new page in IMP, a new imap connection is made and their
>INBOXes (which grow to enormous sizes, even with disk quotas) are
>rescanned.  The I/O is tremendous, and cripples the machine during lunch
>or other rush hours.
>
>An earlier thread mentioned a cached proxy, and I understand that
>persistent imap connections are a fundamental problem in a stateless
>system


And that brings us back to now:


>Any work in this direction would certainly be great. I'm a bit buried at the 
>moment, but if it looks like this is feasible, I'd love to help out and 
>facilitate in any way I can.
>
>> I can already see that IMP will not scale well if it has to make a new
>> IMAP connection for every HTTP request.
>
>It does surprisingly well, actually, but persistant connections would certainly 
>make things much better if done well.


Yes, imp does well.  But UW-IMAP does not, as Vasilios said:


>almost every time a student here loads up a new page in IMP, a new imap
>connection is made and their INBOXes (which grow to enormous sizes, even
>with disk quotas) are rescanned.  The I/O is tremendous, and cripples
>the machine


I did not find Ronald's discussion of a possible IMAP proxy when I
searched the archive the first time.  I did not go back that far until
today, while re-seaching the archive to answer your question on where
I saw the discussion of using PHP4 sessions to create persistent IMAP
connections.  Before coming to the office today, I had been thinking
about how to avoid the IMAP connection setup/teardown overhead, and I
independently conceived the same idea as Ronald, which you said was a
good idea if only someone would write it.

To summarize what has been said, plus my idea of implementation:

Although we can maintain limited state information with PHP4 sessions,
PHP4 still provides no method of keeping a TCP socket open to the IMAP
server between browser requests.

So to echo Ronald's idea:

Develop a client-server IMAP proxy, based on the echo client-server
described in Stevens' UNIX Network Programming, Volume 1.  As Stevens
said, his example could be the foundation for any TCP client-server,
all you need to add is the code needed for handling your application
data.

Use a prethreaded server model, described in the same Stevens' text,
where the server would be one process with many threads.  Threads
would service requests from clients, proxy the requests to the IMAP
server(s), and hand responses back to the clients.

There is some extra overhead in this approach, because you must do a
copy of the socket data in each direction.  But I expect this will be
a negligible increase of overhead, in comparison to the huge overhead
eliminated by avoiding the IMAP server open/close on every browser
request.

With PHP4 sessions, imp should only need limited tweaking.  As Ronald
said, imp should be "using that session key as a request qualifier
with every IMAP request."  IMP can use a config variable to determine
whether the "IMAP server" is actually an IMAP proxy, and thus know
when to add a session key qualifier to each request.  The IMAP proxy
will strip off the request qualifier before passing each request to
the IMAP server.

The IMAP proxy will use the PHP session key and other qualifying
information such as client ip address to determine if it already has
an open server connection for that client.  The IMAP proxy will spoof
open and close requests; it will pass other requests unchanged (except
for stripping off the request qualifier).

To answer Rich's question "When do you close the connection," each
IMAP server connection held by the IMAP proxy will be closed either
when the client logs out from the IMAP server, via his imp logout
function, or after 7 minutes of inactivity from the client, whichever
occurs first.

I think this IMAP proxy concept is the way to go.  Now who wants to
tackle the SMOP?  ;-)

Egan