[dev] [Corrected] Horde_Imap_Client and fetching vanished messages.

[Michael J Rubinsky]

Quoting Michael M Slusarz <slusarz at horde.org>:

> Quoting Michael J Rubinsky <mrubinsk at horde.org>:
>
>> Some background: This code is all for the purpose of syncing email  
>> over ActiveSync. I'm using modseq and changedsince to retrieve the  
>> uids of any recently changed email. In this context, 'changed'  
>> would mean a new, never before seen email, or an email that has had  
>> the seen flag added or removed.
>
> Pardon my ignorance: what does the ActiveSync client send to the  
> server to indicate the current status of its synchronized cache?  Is  
> it a user-definable cache ID?  Or is it a timestamp of the last  
> sync?  Or something else?

First off, you probably know this, but the ActiveSync client knows  
*nothing* about IMAP (or POP3 for that meatter). In fact, it doesn't  
care at all where the messages come from, as long as it receives them  
in the format defined by the ActiveSync protocol. As far as state  
information goes, the only thing the ActiveSync client sends to or  
receives from the ActiveSync server is it's current syncKey. The  
syncKey is basically a random hash string with an integer tacked onto  
the end of it. The server generates this key during the first sync of  
each collection (email|contacts|calendar etc...). The client sends  
this key along with each SYNC and PING request to notify the server  
what it is assuming the last known state was. When the state changes,  
the server increments the syncKey after sending changes to the client.  
This is the only bit of identifying information the client ever gets  
or sends. Server side, this syncKey is linked to the server state at  
the time that the syncKey was generated. So, e.g., with contacts or  
calendar data, this state is basically a timestamp. We use this  
timestamp to query the History system to get server changes to send.

For mail, the state consists of the modseq/nextuid/uidvalidity data,  
along with a list of UIDs and their seen state that are on the device.

> If user-definable, you would obviously be using a combination of  
> UIDVALIDITY + MODSEQ if QRESYNC is available.  If QRESYNC is not  
> available, it will be a combination of UIDVALIDTY + UIDNEXT.  But  
> note the latter cannot ever reliably catch flag changes (you need to  
> do a FETCH FLAGS on every sync to reliably catch flag changes  
> without QRESYNC).
>
>> If I don't use 'changedsince', I would have to retrieve the  
>> complete set of UIDs again, along with flags and compare each  
>> message with my stored device state.
>
> This should only need to be done once, on the first sync, regardless  
> of whether QRESYNC is available or not.

I'm confused. If QRESYNC/CONDSTORE is NOT available, how else would I  
be able to catch flag changes, other than querying the imap client for  
what flags are on each message and comparing them with the device's  
state? To be clear, I'm talking about querying the imap client, not  
the server itself. I assume the client will be smart about when it  
actually talks to the server, and what it asks for.

>> I also have to compare the list of UIDs on the server with the UIDs  
>> known to be on the client to determine which messages have been  
>> expunged/deleted as well - which is what I was hoping 'vansihed'  
>> would have helped with (I'm currently just array_diff()ing the  
>> server uid list with the devices uid list to determine this).
>
> This is exactly what VANISHED does, but it requires QRESYNC.

Ok. Thanks for the clarification. At least I know I was attempting to  
use it correctly. Now that you fixed the bug, it should work as  
expected.

> In the absence of QRESYNC, it is still trivial to determine the list  
> of *new* messages since the last sync - since the cache ID will be  
> using the last known UID, it is simply a matter of FETCH'ing all  
> UIDs greater than this value.  But this does not catch flag changes  
> and it does not catch messages that have since been deleted.  So,  
> practically, you do need to do this array_diff() (both ways - one to  
> catch new messages, one to catch deleted messages) to sync.

This is why I thought I needed to query the imap client for the list  
of UIDs and flag state when QRESYNC is not available - so I can  
compare the flags against what the device thinks each message's flag  
state is.

>> Caching in the imap client would obviously help with imap server  
>> load, but given how frequently this must occur, I'd like to avoid  
>> having to iterate over each message to check the status of the  
>> flags to determine what has changed. Using changedsince gets me  
>> only the messages that have changed, greatly reducing the number of  
>> messages I'd have to iterate.
>
> As mentioned above, without QRESYNC this is impossible.  (Actually,  
> CHANGEDSINCE was defined with CONDSTORE)

See comment above. If QRESYNC is not available you say it still  
shouldn't be necessary to get a list of UIDs and flags. Not sure how  
else I would be able to catch flag changes? Am I misunderstanding  
something again?

>
>> We discussed briefly in IRC the possibility of just requiring a  
>> IMAP server that supports the needed extensions for this to work.  
>> Even if we decide to not require this in the future, it would still  
>> be nice to provide the enhanced performance for those servers that  
>> *do* support it.
>
> From what you have said so far, you are requiring this advanced  
> functionality.  Since ActiveSync email syncing can be thought of as  
> an advanced feature, I have no issues with making this a  
> requirement, although it will decrease the number of users this  
> feature is available to (IIRC, QRESYNC is really only available with  
> Dovecot and Cyrus).

I'd like to make use of the functionality if it's present. Now that  
VANISHED works correctly, this gets me a lot closer, thank you.  Once  
I have it working WITH QRESYNC, I'll add support for those servers  
that do NOT support it, with the understanding that this will likely  
greatly increase server load when used with ActiveSync.

Thanks for taking the time to explain this :)

-- 
mike

The Horde Project (www.horde.org)
mrubinsk at horde.org