[Tickets #8922] Re: Sort messages by arrival date not working correctly

[bugs at horde.org]

DO NOT REPLY TO THIS MESSAGE. THIS EMAIL ADDRESS IS NOT MONITORED.

Ticket URL: http://bugs.horde.org/ticket/8922
------------------------------------------------------------------------------
  Ticket             | 8922
  Updated By         | Michael Slusarz <slusarz at horde.org>
  Summary            | Sort messages by arrival date not working correctly
  Queue              | IMP
  Version            | 4.3.6
  Type               | Bug
  State              | Not A Bug
  Priority           | 2. Medium
  Milestone          |
  Patch              |
  Owners             |
------------------------------------------------------------------------------


Michael Slusarz <slusarz at horde.org> (2010-03-19 12:46) wrote:

> But why would users be interested in IMAP servers internal info like  
> message sequence number/UID?

Because it is the fastest sort.  All IMAP servers necessarily need to  
track their message sequence numbers, so sorting by this value is  
"free" (as opposed to date sort - this requires each message's header  
to be parsed to get the date value).

> Looking at the IMAP SORT RFC (http://www.faqs.org/rfcs/rfc5256.html)  
> I found the ARRIVAL sort type:
>
>       ARRIVAL
>          Internal date and time of the message.  This differs from the
>          ON criteria in SEARCH, which uses just the internal date.

Yes.  ARRIVAL SORT = message sequence number.  This statement is  
nothing more than providing a keyword to use for SORT to do the same  
thing as a regular arrival sort.

> And this is exactly what gets called if I disable the IMP _arrival  
> cache. So _arrival cache pretends to implement this IMAP search type  
> but does something different - sorts on UID instead of IMAP internal  
> date. As a result our users see messages that are almost randomly  
> sorted. The worst thing is that arrival sort type is the default.

_arrival cache does not to "pretend to implement this IMAP search  
type".  The arrival cache implements a cache of the message sequence  
numbers.

Arrival sort is on by default because it is the *only* sort that can  
reasonably be used on large mailboxes on ALL IMAP servers.

Unfortunately, you are assuming a lot of things about IMAP servers.   
First, the SORT extension wasn't standardized until June *2008*.  So  
there can be no assumption that the IMAP server supports SORT (not to  
mention IMP 4 was first released 5-6 years ago, before SORT existed on  
most IMAP servers).

Second, many IMAP servers don't support SORT.  It's not required of an  
IMAP 4rev1 server.  Thus, to sort by anything other than internal  
arrival time requires the IMAP server to parse *EVERY* header of  
*EVERY* message in the mailbox (a MUA can use FETCH to only return the  
text of the desired header, but this still requires the IMAP server to  
internally parse every header to collate this information).

Third, even for IMAP servers that support SORT, there is no guarantee  
that SORT is inexpensive.  Many IMAP servers may not cache this  
information, so every SORT call to the server requires parsing of the  
entire mailbox.

The issue with sorting a mailbox, at least with the original IMAP  
4rev1 spec, is that IMAP was originally designed to act in a connected  
environment.  Meaning that you logged on once in the morning and your  
e-mail client handled issues like sorting ONCE (since further changes  
could be caught via untagged responses since the IMAP client was  
always connected to the IMAP server).  IMAP was NOT designed for  
disconnected clients, such as webmail, that were constantly  
connecting/disconnecting from the server.  Tools have slowly been  
developed to help this situation (imapproxy, RFC extensions, webmail  
server caching) but usage of these tools can not be guaranteed in any  
single installation.

Thus, the only reasonable default sort is ARRIVAL.