[dev] I/O hammering

Egan egan@sevenkings.net
Wed, 06 Jun 2001 00:54:20 -0400
Previous message: [dev] I/O hammering
Next message: [dev] Horde release
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]
On Tue,  5 Jun 2001 10:56:46 -0400, Chuck Hagenbuch <chuck@horde.org>
wrote:

>>     UID FETCH 5857 UID
>>     1 FETCH (UID 5857)
>>     OK UID FETCH COMPLETED
>
>This is most likely caused by the mail_uid_sequence() call in PHP's 
>imap_fetch_overview.

You are right.

It took me a while of looking at imap_fetch_overview and c-client to
understand why it's being done that way, but now I see.

The purpose of mail_uid_sequence is to set elt->sequence to T, for
each UID.  The net effect is a mapping of UIDs to MSGNOs in the
imap_fetch_overview for loop, presumably to keep the code simple for
the programmer.

For each UID, c-client's mail_uid_sequence is calling mail_elt()->
sequence, just to set a bit in the elt which says "this is a message
we are interested in for the next processing steps."

Unfortunately, setting a bit in the elt, or at least doing it via
mail_uid_sequence, results in a TCP packet round trip for every UID,
which is a significant hit on performance, and especially so if you
are talking to an IMAP server over a slow link.


>I don't know if there's a way around that with c-client.

The c-client internal.txt doc file says that elt->sequence is "only
used internally by drivers."  I suppose php_imap.c qualifies as a
driver, at least for its purpose of using elt->sequence in this way.

However, there is another bit in the elt called "searched."  As the
docs say, it is set whenever the "message matches search criteria in
most recent mail_search_full() call."  mail_search_full requires a
SEARCHPGM *pgm (really a structure, not a program) to specify your
search criteria.  The search criteria is very flexible.  Among other
possibilities, it will let you specify a SEARCHSET *uid which is a
"set of unique identifiers." (just what we need!)

I have already written some c-client code which uses one of these
search "programs," because I needed to delete messages more than 40
days old.  Setting up the search criteria is not hard.  C-client sets
the "searched" bit for every messages which matches, and you just set
up a for loop to process the ones tagged as "searched."

It should not be hard to adapt this technique and use it to replace
the call to mail_uid_sequence() in imap_fetch_overview.  If it works
the way I expect, sending the search criteria in a single TCP round
trip, it will eliminate a lot of the TCP overhead I see in the trace.

I will work on it here and report any progress.

Egan
















>> Also, it seems that the "FETCH 1,2:21 (ENVELOPE," could omit unneeded
>> information.  The "BODY.PEEK[HEADER.FIELDS" included in the request is
>> returning PATH, MESSAGE-ID, NEWSGROUPS, FOLLOWUP-TO, and REFERENCES.
>> I don't see where any of those are used for building the message list.
>> It appears that dropping the BODY.PEEK from the request to omit those
>> fields from the return data would reduce the data volume of that one
>> request maybe 30-40%.
>
>We can't do that level of tweaking without modifying c-client.
>
>-chuck
>
>--
>Charles Hagenbuch, <chuck@horde.org>
>Black and white and grey, all the shades of truth.



>From jrkuipers@lauwerscollege.nl Date: Wed,  6 Jun 2001 08:25:03 +0200
Return-Path: <jrkuipers@lauwerscollege.nl>
Mailing-List: contact dev-help@lists.horde.org; run by ezmlm
Delivered-To: mailing list dev@lists.horde.org
Received: (qmail 33855 invoked from network); 6 Jun 2001 06:25:45 -0000
Received: from c3207.upc-c.chello.nl (HELO linux.lauwerscollege.nl) (212.187.3.207)
  by horde.org with SMTP; 6 Jun 2001 06:25:45 -0000
Received: (from www@localhost)
	by linux.lauwerscollege.nl (8.9.3/8.8.7) id IAA26663
	for dev@lists.horde.org; Wed, 6 Jun 2001 08:25:03 +0200
Received: from 212.187.3.207 ( [212.187.3.207])
	as user jrkuipers@localhost by horde.lauwerscollege.nl with HTTP;
	Wed,  6 Jun 2001 08:25:03 +0200
Message-ID: <991808703.3b1dccbf22051@horde.lauwerscollege.nl>
Date: Wed,  6 Jun 2001 08:25:03 +0200
From: Jan Kuipers <jrkuipers@lauwerscollege.nl>
To: dev@lists.horde.org
MIME-Version: 1.0
Content-Type: text/plain
Content-Transfer-Encoding: 8bit
User-Agent: Internet Messaging Program (IMP) 2.3.7-cvs
Subject: Bug in Gollum

Hello,

I get the following, when trying to access Gollem:

Warning:  Constant no_action already defined in /usr/local/apache/htdocs/hordenew/horde/gollem/lib/Gollem.php on line 13
Warning:  Constant update_prefs already defined in /usr/local/apache/htdocs/hordenew/horde/gollem/lib/Gollem.php on line 21
Warning:  Cannot add header information - headers already sent by (output started at /usr/local/apache/htdocs/hordenew/horde/gollem/lib/Gollem.php:13) in /usr/local/apache/htdocs/hordenew/horde/lib/Lang.php on line 62
Warning:  Cannot add header information - headers already sent by (output started at /usr/local/apache/htdocs/hordenew/horde/gollem/lib/Gollem.php:13) in /usr/local/apache/htdocs/hordenew/horde/gollem/index.php on line 27


-- 
Jan Kuipers,
Systeembeheer Lauwers College


>From janmailing@gmx.de Date: Wed,  6 Jun 2001 10:33:04 +0200
Return-Path: <janmailing@gmx.de>
Mailing-List: contact dev-help@lists.horde.org; run by ezmlm
Delivered-To: mailing list dev@lists.horde.org
Received: (qmail 37315 invoked from network); 6 Jun 2001 08:37:04 -0000
Received: from mailout05.sul.t-online.com (HELO mailout05.sul.t-online.de) (194.25.134.82)
  by horde.org with SMTP; 6 Jun 2001 08:37:04 -0000
Received: from fwd05.sul.t-online.de 
	by mailout05.sul.t-online.de with smtp 
	id 157Yo8-0004gh-06; Wed, 06 Jun 2001 10:36:52 +0200
Received: from linux.wg.de (320034214675-0001@[217.80.111.120]) by fmrl05.sul.t-online.com
	with esmtp id 157Ynx-1keF5EC; Wed, 6 Jun 2001 10:36:41 +0200
Received: from localhost (localhost [127.0.0.1])
	by linux.wg.de (8.11.0/8.11.0/SuSE Linux 8.11.0-0.4) with ESMTP id f568X4R12093
	for <dev@lists.horde.org>; Wed, 6 Jun 2001 10:33:04 +0200
Received: from 192.168.60.1 ( [192.168.60.1])
	as user jan@mail.wg.de by linux.wg.de with HTTP;
	Wed,  6 Jun 2001 10:33:04 +0200
Message-ID: <991816384.3b1deac01aa62@linux.wg.de>
Date: Wed,  6 Jun 2001 10:33:04 +0200
From: Jan Schneider <janmailing@gmx.de>
To: dev@lists.horde.org
References: <991320132.3b16584423533@linux.wg.de> <991800227.3b1daba3ca3ff@marina.horde.org>
In-Reply-To: <991800227.3b1daba3ca3ff@marina.horde.org>
MIME-Version: 1.0
Content-Type: text/plain
Content-Transfer-Encoding: 8bit
User-Agent: Internet Messaging Program (IMP) 2.3.7-cvs
X-Sender: 320034214675-0001@t-dialin.net
Subject: Re: [dev] Horde portal

Hi,

I already recognized your commits but didn't find the time yet to have a closer 
look.

Jan.

Zitat von Chuck Hagenbuch <chuck@horde.org>:

> Quoting Jan Schneider <janmailing@gmx.de>:
> 
> > I'm starting to implement a "portal" for horde, that is after login to
> horde 
> > you get a summary of information from you horde apps. E.g. new emails from
> 
> > imp, todays appointments from kronolith and open tasks from nag. Just 
> > imagine "Outlook Today".
> > 
> > I thought I would manage to implement this without indepth knowhow of the
> 
> > horde framework with a little bit of cut and paste and some general php
> > skills. But I was wrong.
> 
> Take a look at what I've committed the last day or so to list Nag tasks, etc.
> 
> It should be done this way - through the registry, abstracted, etc - Horde 
> should _never_ make an explicit reference to an individual application.
> 
> I'll take a look at a messages summary, also, but I'm curious what you think
> of 
> what's there, and if you can follow it to create your own.
> 
> -chuck
> 
> --
> Charles Hagenbuch, <chuck@horde.org>
> Black and white and grey, all the shades of truth.
> 
> -- 
> Horde Developers mailing list: http://horde.org/
> Frequently Asked Questions: http://horde.org/faq/
> To unsubscribe, mail: dev-unsubscribe@lists.horde.org
> 
> 


::::::::::::::::::::::::::::::::::::::::
AMMMa AG - discover your knowledge
:::::::::::::::::::::::::::
Detmolder Str. 25-33 :: D-33604 Bielefeld
fon +49.521.96878-0 :: fax  +49.521.96878-20
http://www.ammma.de
::::::::::::::::::::::::::::::::::::::::::::::


>From chuck@horde.org Date: Thu,  7 Jun 2001 09:53:44 -0400
Return-Path: <chuck@horde.org>
Mailing-List: contact dev-help@lists.horde.org; run by ezmlm
Delivered-To: mailing list dev@lists.horde.org
Received: (qmail 75921 invoked from network); 7 Jun 2001 13:55:26 -0000
Received: from 208-59-250-206.c3-0.smr-ubr1.sbo-smr.ma.cable.rcn.com (HELO marina.horde.org) (208.59.250.206)
  by horde.org with SMTP; 7 Jun 2001 13:55:26 -0000
Received: by marina.horde.org (Postfix, from userid 33)
	id 2604839EE; Thu,  7 Jun 2001 09:53:45 -0400 (EDT)
Received: from 206.243.191.252 ( [206.243.191.252])
	as user chuck@localhost by marina.horde.org with HTTP;
	Thu,  7 Jun 2001 09:53:44 -0400
Message-ID: <991922024.3b1f8768c99e1@marina.horde.org>
Date: Thu,  7 Jun 2001 09:53:44 -0400
From: Chuck Hagenbuch <chuck@horde.org>
To: dev@lists.horde.org
MIME-Version: 1.0
Content-Type: text/plain
Content-Transfer-Encoding: 8bit
User-Agent: Internet Messaging Program (IMP) 2.3.7-cvs
Subject: Horde release

Is there anything (all of us being rather busy aside) holding up a Horde 1.3.4 
release at this point? We need to push things out a bit quicker, I think, 
especially in alpha stages...

-chuck

--
Charles Hagenbuch, <chuck@horde.org>
Black and white and grey, all the shades of truth.
Previous message: [dev] I/O hammering
Next message: [dev] Horde release
Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]