[dev] STATUS_SYNC* discussion (Was: [Tickets #11612] Re: Broken imap fetch query)

[Michael M Slusarz]

Quoting Michael J Rubinsky <mrubinsk at horde.org>:

> Quoting Michael M Slusarz <slusarz at horde.org>:
>
>> Quoting Michael J Rubinsky <mrubinsk at horde.org>:
>>
>> UIDNEXT is worthless for flag changes and message deletions so I am  
>> assuming you are only using it to determine if new messages have  
>> possibly been added to the mailbox.
>
> Correct. Some background: With ActiveSync, the client "PINGS" the  
> server for changes in long-lived loop. At this stage, we don't care  
> about what the changes are, just that there *is* a change we want to  
> transmit to the device. The current request is then terminated and a  
> new SYNC request is issued, where we send the exact changes to the  
> device. When I say for "performance reasons" I mean that at the PING  
> stage, I am only checking for new messages. That is the only change  
> that I want to trigger a SYNC. This saves countless new connections  
> (and save battery life and bandwidth) by not causing a new SYNC to  
> be issued when there is only a flag change or deletion. In other  
> words, the only thing that will trigger a new SYNC request is a new  
> email arriving in the mailbox - but once the SYNC request is  
> triggered, it asks for new, vanished, and flags.
>
> From what you are telling me though, I should be using  
> STATUS_UIDNEXT_FORCE for this since the data might not be available  
> without the extra server call, correct?

Yes - this is a MUST, not a SHOULD.  Or else on certain servers, you  
will never see UIDNEXT change - it will always be 0.

Unfortunately, for servers that don't support CONDSTORE and don't send  
UIDNEXT, there will be unnecessary syncs when a mailbox goes from  
having at least 1 message to being empty.  In this case, UIDNEXT  
returns 0 since it is impossible to determine the next UID in the  
mailbox.

>>>> But see Ticket #11590 - If not using Imap_Client's full caching,  
>>>> there is still an opportunity to take advantage of this.  Namely:  
>>>> creating a custom cache driver that has the known MODSEQ value  
>>>> that can be returned from it.  The mailbox will be sync'd on  
>>>> open, and the flag values can be cached in the custom cache  
>>>> driver so that SYNCFLAGUIDS/VANISHED doesn't need to hit the  
>>>> server again (the custom cache driver could be an in-memory cache).
>>>
>>> Hmm, I could use/refactor ActiveSync's current caching object  
>>> (Horde_ActiveSync_Folder_Imap) as a custom cache driver and let  
>>> activesync decide which sync_key's data to populate it with before  
>>> injecting it into the imap client. I know this will let me capture  
>>> changes after the mailbox syncs, but how would I capture the  
>>> changes during that first sync, by a normal fetch query using the  
>>> MODSEQ that I have?
>>
>> Not following you here.  Maybe I clarified what I meant by a stub  
>> cache object above.
>
> Basically, I'm talking about implementing a custom cache driver for  
> the imap client that would be populated with the data that  
> activesync currently persists to database storage (the modseq,  
> uidnext etc...). But again, I'm not sure how to go about doing this  
> cleanly with current code while still being able to use the imap  
> object from IMP.

 From what it sounds like, you are keeping duplicate cache data  
around.  We only need one cache on the server - let the IMP Imap  
object do that for you.  Then you just need to carry around the state  
of your mailbox as it is known on the activesync device.  This can  
already be easily done, without IMAP extension querying, via  
getCacheId().

What is missing right now is the ability to take this cache ID token  
and give it to Horde_Imap_Client_Base, and have it do all the  
necessary things to 1) check the UIDVALIDITY, 2) return the list of  
flag changes, and 3) return the list of vanished UIDs.  This is what I  
am going to write.  It would do things like automatically use  
STATUS_SYNC* if the cache token matched the IMP Imap object cached  
state when opening the mailbox for the first time in that page load.   
It would also be able to handle CONDSTORE-only servers, and be able to  
do syncing on basic IMAP4rev1 servers.  And this is all transparent to  
the calling code.

Although I mentioned that getCacheId() already exists, it will not be  
used for the new code - it has the design flaw of trying to allow data  
access to the string itself (the API documentation for the return  
value explains, for example, how to grab the uidvalidity).  Instead, I  
am going to implement a getSyncId() method that returns a string that  
has no defined format (it's probably going to be nothing more than a  
base64 encoded version of the getCacheId() result).  And then I will  
implement a syncMailbox() method that takes the token as its only  
argument.  This method will throw an exception if the UIDVALIDITY  
changed, will return false/null if the mailbox hasn't changed, and  
will return four entries if the mailbox has changed:

   - flags: Horde_Imap_Client_Ids object of the list of messages whose  
flags changed.
   - newmsgs: (Boolean) Whether new messages were delivered to the mailbox
   - syncid: (String) The new syncid token.
   - vanished: Horde_Imap_Client_Ids object of the list of messages  
that were expunged.

Would this be sufficient for what you need to do in your code?

michael

___________________________________
Michael Slusarz [slusarz at horde.org]