[dev] STATUS_SYNC* discussion (Was: [Tickets #11612] Re: Broken imap fetch query)

[Michael M Slusarz]

Quoting Michael J Rubinsky <mrubinsk at horde.org>:

> Quoting Michael M Slusarz <slusarz at horde.org>:
>
>> Quoting Michael J Rubinsky <mrubinsk at horde.org>:
>>
>>> Quoting Michael M Slusarz <slusarz at horde.org>:
>
>>> From what you are telling me though, I should be using  
>>> STATUS_UIDNEXT_FORCE for this since the data might not be  
>>> available without the extra server call, correct?
>>
>> Yes - this is a MUST, not a SHOULD.  Or else on certain servers,  
>> you will never see UIDNEXT change - it will always be 0.
>
> Done, thanks for the advice.
>
>> From what it sounds like, you are keeping duplicate cache data  
>> around.  We only need one cache on the server - let the IMP Imap  
>> object do that for you.  Then you just need to carry around the  
>> state of your mailbox as it is known on the activesync device.   
>> This can already be easily done, without IMAP extension querying,  
>> via getCacheId().
>
> When QRESYNC is in use, that's pretty much what I'm doing. I persist  
> the state of the mailbox using MODSEQ, UIDVALIDITY, and UIDNEXT.  
> It's when QRESYNC isn't available that I'm also storing the UIDs and  
> flags.

Why are you storing/comparing flags?  I don't think it's asking too  
much to require someone who wants to use activesync to be using a  
CONDSTORE server, or else they are not getting flag changes.  That's  
the way IMP used to work, and is a reasonable trade-off.

I look at this as analogous to "mailbox" support in POP3.  It would  
definitely be possible to create a system on the Horde/IMP server that  
would allow mailboxes to be abstracted on top of the POP3 model (i.e.  
you keep a local message-id map using SQL storage to indicate which  
messages live in which mailboxes).  But that's really a waste of time  
and re-inventing the wheel since this problem has already been solved  
by IMAP.  Same with flag change tracking.  This issue has been solved  
by CONDSTORE.  Re-inventing a wheel as a way to work around this isn't  
useful when you can just tell an admin to upgrade their IMAP server.

And you should be storing UIDs regardless of whether QRESYNC is  
available.  It can drastically reduce the amount of work necessary to  
resync as seen by the recent VANISHED discussion on the dovecot list.  
At a minimum, keep track of the lowest/highest known UIDs so that you  
can provide a range string to the IMAP commands rather than doing an  
all message command ('1:*').

>> What is missing right now is the ability to take this cache ID  
>> token and give it to Horde_Imap_Client_Base, and have it do all the  
>> necessary things to 1) check the UIDVALIDITY, 2) return the list of  
>> flag changes, and 3) return the list of vanished UIDs.  This is  
>> what I am going to write.  It would do things like automatically  
>> use STATUS_SYNC* if the cache token matched the IMP Imap object  
>> cached state when opening the mailbox for the first time in that  
>> page load.  It would also be able to handle CONDSTORE-only servers,  
>> and be able to do syncing on basic IMAP4rev1 servers.  And this is  
>> all transparent to the calling code.
>>
>> Although I mentioned that getCacheId() already exists, it will not  
>> be used for the new code - it has the design flaw of trying to  
>> allow data access to the string itself (the API documentation for  
>> the return value explains, for example, how to grab the  
>> uidvalidity).  Instead, I am going to implement a getSyncId()  
>> method that returns a string that has no defined format (it's  
>> probably going to be nothing more than a base64 encoded version of  
>> the getCacheId() result).  And then I will implement a  
>> syncMailbox() method that takes the token as its only argument.   
>> This method will throw an exception if the UIDVALIDITY changed,  
>> will return false/null if the mailbox hasn't changed, and will  
>> return four entries if the mailbox has changed:
>>
>>  - flags: Horde_Imap_Client_Ids object of the list of messages  
>> whose flags changed.
>>  - newmsgs: (Boolean) Whether new messages were delivered to the mailbox
>>  - syncid: (String) The new syncid token.
>>  - vanished: Horde_Imap_Client_Ids object of the list of messages  
>> that were expunged.
>>
>> Would this be sufficient for what you need to do in your code?
>
> This sounds great, but I'm still a little fuzzy on how this would  
> work for ActiveSync without ActiveSync controlling the caching.  For  
> example, when not using QRESYNC how would flag changes be determined  
> between any arbitrary state on the ActiveSync client and the current  
> server state?

You can determine flag changes between MODSEQs with CONDSTORE also.

Guess I am confused why you keep saying QRESYNC instead of CONDSTORE.   
For your purposes, everything you can do with QRESYNC you can do with  
CONDSTORE.

The only feature CONDSTORE doesn't provide is an ability to determine  
a list of UIDs that have been deleted since a given modseq.  But so  
what?  You can easily replicate this at the Horde level by determining  
the difference between the list of known UIDs and the current list of  
UIDs in the mailbox (this can now be done for you automatically if you  
call vanished() with the 'ids' parameter).

And you are right: without CONDSTORE you would have to re-sync on  
every sync since there is no way to catch flag changes.  As mentioned  
above, if I was writing the ActiveSync code, I would ignore flag  
changes for non-CONDSTORE servers.

> I need to track more than one different state in case the client  
> reissues the last request. Would the caching be able to still  
> calculate the deltas for an older syncId?

Yes, if using CONDSTORE.  We are just doing IMAP server calls mostly  
(with the one optimization that STATUS_SYNC* cached values are used if  
the initial IMP Imap object mailbox sync happened to be the same  
MODSEQ as the activesync cached data.)

> Currently, the mailbox state is saved to the database, keyed by  
> ActiveSync's syncKey. When not using QRESYNC, each entry is a  
> complete snapshot of the ActiveSync folder state - a list of UIDs on  
> the device along with any flags. There is normally two different  
> versions of this data - the state as it should be if all changes  
> that were just sent were received successfully, and the state of the  
> device *before* the changes were sent.

Are you storing these as "deltas" to each other?  In other words, the  
former should only include the changes to the latter.

> If what you are talking about would be able to handle this  
> transparently, then that would be a complete solution for me,  
> otherwise I will still probably need to track UIDs and flags  
> independently if not using QRESYNC.

Browsed the IMAP code in ActiveSync, and I don't claim to be an  
expert, but these are my thoughts after the above discussion (these  
are my opinions; ymmv):

1.) You need to replace your "QRESYNC" check with a "CONDSTORE" check instead.
2.) Store UIDs (or at least min/max UIDs) for CONDSTORE servers to  
optimize IMAP server commands.
3.) Storing flag state for CONDSTORE servers is not necessary (this  
seems to be what you are doing in  
Horde_ActiveSync_Imap_Adapter#getMessageChanges()).  If a UID is  
indicated as changed, just go ahead and re-sync that particular entry,  
even if it doesn't cause any noticeable change at the activesync UI  
level.  Any gain you might get from the fact that the flag change  
isn't one that you care about, or the flag change was subsequently  
reversed, is offset by the overhead in caching this data.
4.) Ignore all flag changes in non-CONDSTORE servers.

What this means is that for any mailbox, you have the following state entries:

"Before sync" state: IMAP mailbox cached state (UIDNEXT, UIDVALIDITY,  
HIGHESTMODSEQ), list of UIDs on the server
"After sync" state: new IMAP mailbox cached state (UIDNEXT,  
UIDVALIDITY, HIGHESTMODSEQ), list of UIDs sync'd

All FETCH data should be cached by the IMP Imap Client cache object.   
You should not be caching any FETCH details in activesync code.

michael

___________________________________
Michael Slusarz [slusarz at horde.org]