[dev] I/O hammering (was: imp folder prefix)

[Egan]

On Tue, 29 May 2001 20:25:24 +0100, "Anil Madhavapeddy"
<anil@recoil.org> wrote:

>Outlook caches the message bodies to local disk

OK, I went into Outlook and used the local file cleanup "reset" option
to purge all my local cache, forcing it to download headers again, and
thus fairly compare it to IMP.


>UW might have something similar).  Or just tcpdump and parse the
>output through a script.

I don't know what UW has, so I used tcpdump -x, piped to a perl hex
decoder called tcpdecode.  With 4 messages in my INBOX, I traced
Outlook vs. IMP.  Comparing the two traces, I see that Outlook does a
UID fetch:

    UID FETCH 1:* (
        BODY.PEEK [HEADER.FIELDS (References ... Newsgroups)]
        ENVELOPE
        RFC822.SIZE
        UID
        FLAGS
        INTERNALDATE
    )

Then IMAPD returns the requested info for all 4 messages, in a single
response:

    1 FETCH (UID 5243 ... blah blah
    2 FETCH (UID 5244 ... blah blah
    3 FETCH (UID 5245 ... blah blah
    4 FETCH (UID 5246 ... blah blah

And then, IMAPD says:

    OK UID FETCH COMPLETED

Then, Outlook does:

    UID FETCH 1:5246 (UID FLAGS)

And IMAPD responds with:

    1 FETCH (UID 5243 FLAGS (\Seen))
    2 FETCH (UID 5244 FLAGS ())
    3 FETCH (UID 5245 FLAGS (\Seen))
    4 FETCH (UID 5246 FLAGS (\Seen))
    OK UID FETCH completed

And that is the end.  Outlook has built its message list, and it's
done.  Very brief and to the point.  And it does not hammer the disk.

:-)

The IMP trace, OTOH, well ... it starts with:

    UID SORT

then

    UID FETCH 5243 UID

and then a sequence fetch:

    FETCH 1,2:4 (
        ENVELOPE        
        BODY.PEEK [HEADER.FIELDS (Path Message_ID Newsgroups ...)]
        INTERNALDATE
        RFC822.SIZE
        FLAGS
    )

Then IMAPD returns the requested info for all 4 messages, in a single
response

    1 FETCH (ENVELOPE ... blah blah
    2 FETCH (ENVELOPE ... blah blah
    3 FETCH (ENVELOPE ... blah blah
    4 FETCH (ENVELOPE ... blah blah

And then, IMAPD says:

    OK FETCH COMPLETED

Then IMP does:

    FETCH 1 BODY.PEEK[HEADER]

*** And IMAP returns the entire header!!! ***  And says:

    OK FETCH COMPLETED

Then IMP does:

    FETCH 1 (BODYSTRUCTURE FLAGS)

IMAPD returns them and says:

    OK FETCH COMPLETED

And then IMP does:

    UID FETCH 5244 UID

And then IMAPD says:

    OK UID FETCH COMPLETED

AND THEN! AND THEN! AND THEN!

    IMP repeats this agonizing cycle for each remaining message.  Just
to build the message list.  Arrrgghhhh!!!!

IMP does not talk well to UW-IMAP.  Maybe it does better with Courier,
but work is needed to get it in shape for UW-IMAP.  I'll do what I can
to help, but I'm just the village idiot. :-\

Egan