[dev] STATUS_SYNC* discussion (Was: [Tickets #11612] Re: Broken imap fetch query)

[Michael M Slusarz]

Quoting Michael J Rubinsky <mrubinsk at horde.org>:

> I do some switching based on the availability of QRESYNC and use the  
> STATUS_HIGHESTMODSEQ value when it's available. At least at the time  
> I wrote the code, QRESYNC was only being reported by  
> Horde_Imap_Client when caching was turned on, so in a way I'm  
> relying on the cache being available, but not actually using the data.

This doesn't sound right.  Meaning a call to:

$imap->queryCapability('QRESYNC');

Should return true/false based on the availability of the extension on  
the server, not the current caching state.

> For that, I only take UIDNEXT into consideration for performance reasons.

Word of warning: UIDNEXT is *not* available automatically on all  
servers, so it is not necessarily a performance advantage (UIDNEXT  
does not have to be returned automatically when SELECTing/EXAMINEing a  
mailbox.  IIRC, Courier does not do this).  I've added a  
STATUS_UIDNEXT_FORCE option that will automatically determine this  
value if it doesn't exist, but this may involve an additional server  
call (and it can't reliably generate a UIDNEXT value for an empty  
mailbox).

UIDNEXT is worthless for flag changes and message deletions so I am  
assuming you are only using it to determine if new messages have  
possibly been added to the mailbox.

> Only after some change has been detected does ActiveSync request the  
> change specifics - at which point I want to know all the changes  
> available, including VANISHED and flag changes.

The advantage of using QRESYNC in SELECT/EXAMINE, instead of this  
approach, is that it allows this synchronization without the overhead  
of 2 additional server calls.

> I will need to look at this more closely, since it sounds like I  
> might be able to write some logic to first synchronize/compare  
> *ActiveSync* code's last known MODSEQ with the *IMAP* client's  
> STATUS_SYNCMODSEQ on the first access of each request and then for  
> every other check during the PING loop simply check STATUS_SYNCMODSEQ.

This is how we do things in IMP.  You take advantage if these values  
happen to match, and fallback to a normal mailbox sync otherwise.

  To further complicate things, I need to track
> different versions of the cached MODSEQ/UIDNEXT values. ActiveSync  
> uses a sync_key to specify the last known state of the collection  
> being synched and we need to keep at least two of these keys cached  
> since the device can request the old sync_key if it never received  
> the data in full or there was some other communication issue - not  
> uncommon on a mobile device.

So... are you using an imap client object that *is* caching?  My  
initial idea was to do something like implementing a stub cache driver  
that does nothing more than return a MODSEQ value (i.e. the activesync  
modseq value).  This cache driver is then used to open the mailbox,  
and any flag/vanished changes can be retrieved via the STATUS_SYNC*  
methods.  The disadvantage is you can't cache anything using this  
method, so it only works if using a non-caching client object.

The better solution (and maybe what we previously talked about) is to  
somehow abstract the syncing code so that the calling code doesn't  
care whether QRESYNC/CONDSTORE is in use.  I could see a "getToken"  
method that returns a status identifier of a mailbox (MODSEQ or  
UIDNEXT).  This token can than be passed to a sync function, which  
would return the list of deleted messages/flag changes - how this  
information is obtained would be transparent to the calling code (i.e.  
QRESYNC could leverage STATUS_SYNC*; non CONDSTORE enabled clients  
would have to use the original inefficient mailbox syncing code).

>> But see Ticket #11590 - If not using Imap_Client's full caching,  
>> there is still an opportunity to take advantage of this.  Namely:  
>> creating a custom cache driver that has the known MODSEQ value that  
>> can be returned from it.  The mailbox will be sync'd on open, and  
>> the flag values can be cached in the custom cache driver so that  
>> SYNCFLAGUIDS/VANISHED doesn't need to hit the server again (the  
>> custom cache driver could be an in-memory cache).
>
> Hmm, I could use/refactor ActiveSync's current caching object  
> (Horde_ActiveSync_Folder_Imap) as a custom cache driver and let  
> activesync decide which sync_key's data to populate it with before  
> injecting it into the imap client. I know this will let me capture  
> changes after the mailbox syncs, but how would I capture the changes  
> during that first sync, by a normal fetch query using the MODSEQ  
> that I have?

Not following you here.  Maybe I clarified what I meant by a stub  
cache object above.

> I'm probably misunderstanding something but for example:
> ActiveSync has a last known MODSEQ of 100 and hasn't connected in  
> e.g., a week. During that time the mailbox's MODSEQ has changed to  
> 200. During that first connection, STATUS_SYNCMODSEQ will be set to  
> 200 so any of the STATUS_SYNC* values will be compared to 200, not  
> to 100, right? How do I get that first change set without having to  
> write switching code to determine if we are using a cache etc...?

No.  STATUS_SYNCMODSEQ will be set to the MODSEQ of the mailbox as  
existed before the mailbox was ever opened in this session.  In other  
words, STATUS_SYNCMODSEQ is the cached MODSEQ value.  (If it returns  
200, it does nothing more than replicate the STATUS_HIGHESTMODSEQ  
option).

So STATUS_SYNCFLAGUIDS and STATUS_VANISHED contain all changes that  
were sync'd to whatever existed in the cache at the beginning of the  
page request (if messages are flagged/deleted subsequent to the  
mailbox opening, these changes will also be stored).

Conceptually, you could actually use STATUS_SYNCFLAGUIDS and  
STATUS_VANISHED if your MODSEQ happens to be *greater* than the cached  
MODSEQ - you will just get some changes returned that you already know  
about.  Depending on how many duplicate changes there are, it is still  
likely more efficient than having to make the 2 round-trips to the  
IMAP server to get the information otherwise.

michael

___________________________________
Michael Slusarz [slusarz at horde.org]