[dev] [Corrected] Horde_Imap_Client and fetching vanished messages.

[Michael M Slusarz]

Quoting Michael J Rubinsky <mrubinsk at horde.org>:

> Some background: This code is all for the purpose of syncing email  
> over ActiveSync. I'm using modseq and changedsince to retrieve the  
> uids of any recently changed email. In this context, 'changed' would  
> mean a new, never before seen email, or an email that has had the  
> seen flag added or removed.

Pardon my ignorance: what does the ActiveSync client send to the  
server to indicate the current status of its synchronized cache?  Is  
it a user-definable cache ID?  Or is it a timestamp of the last sync?   
Or something else?

If user-definable, you would obviously be using a combination of  
UIDVALIDITY + MODSEQ if QRESYNC is available.  If QRESYNC is not  
available, it will be a combination of UIDVALIDTY + UIDNEXT.  But note  
the latter cannot ever reliably catch flag changes (you need to do a  
FETCH FLAGS on every sync to reliably catch flag changes without  
QRESYNC).

> If I don't use 'changedsince', I would have to retrieve the complete  
> set of UIDs again, along with flags and compare each message with my  
> stored device state.

This should only need to be done once, on the first sync, regardless  
of whether QRESYNC is available or not.

> I also have to compare the list of UIDs on the server with the UIDs  
> known to be on the client to determine which messages have been  
> expunged/deleted as well - which is what I was hoping 'vansihed'  
> would have helped with (I'm currently just array_diff()ing the  
> server uid list with the devices uid list to determine this).

This is exactly what VANISHED does, but it requires QRESYNC.

In the absence of QRESYNC, it is still trivial to determine the list  
of *new* messages since the last sync - since the cache ID will be  
using the last known UID, it is simply a matter of FETCH'ing all UIDs  
greater than this value.  But this does not catch flag changes and it  
does not catch messages that have since been deleted.  So,  
practically, you do need to do this array_diff() (both ways - one to  
catch new messages, one to catch deleted messages) to sync.

> Caching in the imap client would obviously help with imap server  
> load, but given how frequently this must occur, I'd like to avoid  
> having to iterate over each message to check the status of the flags  
> to determine what has changed. Using changedsince gets me only the  
> messages that have changed, greatly reducing the number of messages  
> I'd have to iterate.

As mentioned above, without QRESYNC this is impossible.  (Actually,  
CHANGEDSINCE was defined with CONDSTORE)

> We discussed briefly in IRC the possibility of just requiring a IMAP  
> server that supports the needed extensions for this to work. Even if  
> we decide to not require this in the future, it would still be nice  
> to provide the enhanced performance for those servers that *do*  
> support it.

 From what you have said so far, you are requiring this advanced  
functionality.  Since ActiveSync email syncing can be thought of as an  
advanced feature, I have no issues with making this a requirement,  
although it will decrease the number of users this feature is  
available to (IIRC, QRESYNC is really only available with Dovecot and  
Cyrus).

michael

___________________________________
Michael Slusarz [slusarz at horde.org]