[dev] STATUS_SYNC* discussion (Was: [Tickets #11612] Re: Broken imap fetch query)

[Michael J Rubinsky]

Quoting Michael M Slusarz <slusarz at horde.org>:

<snip>

> Why are you storing/comparing flags?  I don't think it's asking too  
> much to require someone who wants to use activesync to be using a  
> CONDSTORE server, or else they are not getting flag changes.  That's  
> the way IMP used to work, and is a reasonable trade-off.

Honestly, because I want to avoid the "It works with ZPush" complaints  
and it isn't that much more work to implement. Few things will  
irritate users more than seeing an inbox full of unread messages on  
their device after they have already read them on the desktop. I  
already need to get the flag state from the server anyway (I have to  
filter out the /deleted flag since AS doesn't support it). Do you have  
any information regarding the prevalence of CONDSTORE support in  
modern IMAP servers? If it's mostly ubiquitous then I'd have no  
problem removing it.

> I look at this as analogous to "mailbox" support in POP3.  It would  
> definitely be possible to create a system on the Horde/IMP server  
> that would allow mailboxes to be abstracted on top of the POP3 model  
> (i.e. you keep a local message-id map using SQL storage to indicate  
> which messages live in which mailboxes).  But that's really a waste  
> of time and re-inventing the wheel since this problem has already  
> been solved by IMAP.  Same with flag change tracking.  This issue  
> has been solved by CONDSTORE.  Re-inventing a wheel as a way to work  
> around this isn't useful when you can just tell an admin to upgrade  
> their IMAP server.
>
> And you should be storing UIDs regardless of whether QRESYNC is  
> available.  It can drastically reduce the amount of work necessary  
> to resync as seen by the recent VANISHED discussion on the dovecot  
> list. At a minimum, keep track of the lowest/highest known UIDs so  
> that you can provide a range string to the IMAP commands rather than  
> doing an all message command ('1:*').

Yeah, I've already come to the conclusion that I need to stop relying  
on QRSYNC for determining EXPUNGED messages. I'll be adding this  
shortly.

<snip>

>> This sounds great, but I'm still a little fuzzy on how this would  
>> work for ActiveSync without ActiveSync controlling the caching.   
>> For example, when not using QRESYNC how would flag changes be  
>> determined between any arbitrary state on the ActiveSync client and  
>> the current server state?
>
> You can determine flag changes between MODSEQs with CONDSTORE also.
>
> Guess I am confused why you keep saying QRESYNC instead of  
> CONDSTORE.  For your purposes, everything you can do with QRESYNC  
> you can do with CONDSTORE.

Mostly because I'm confused and didn't understand the difference.

> The only feature CONDSTORE doesn't provide is an ability to  
> determine a list of UIDs that have been deleted since a given  
> modseq.  But so what?  You can easily replicate this at the Horde  
> level by determining the difference between the list of known UIDs  
> and the current list of UIDs in the mailbox (this can now be done  
> for you automatically if you call vanished() with the 'ids'  
> parameter).

Yeah, this is what I was doing for non-qresync servers. Now I see this  
is even easier now that you have added the ids parameter. I'll be  
using this for all servers now.

> And you are right: without CONDSTORE you would have to re-sync on  
> every sync since there is no way to catch flag changes.  As  
> mentioned above, if I was writing the ActiveSync code, I would  
> ignore flag changes for non-CONDSTORE servers.
>
>> I need to track more than one different state in case the client  
>> reissues the last request. Would the caching be able to still  
>> calculate the deltas for an older syncId?
>
> Yes, if using CONDSTORE.  We are just doing IMAP server calls mostly  
> (with the one optimization that STATUS_SYNC* cached values are used  
> if the initial IMP Imap object mailbox sync happened to be the same  
> MODSEQ as the activesync cached data.)
>
>> Currently, the mailbox state is saved to the database, keyed by  
>> ActiveSync's syncKey. When not using QRESYNC, each entry is a  
>> complete snapshot of the ActiveSync folder state - a list of UIDs  
>> on the device along with any flags. There is normally two different  
>> versions of this data - the state as it should be if all changes  
>> that were just sent were received successfully, and the state of  
>> the device *before* the changes were sent.
>
> Are you storing these as "deltas" to each other?  In other words,  
> the former should only include the changes to the latter.

No, because each state has to stand on it's own. Once a new synckey is  
obtained from the device, all previous synckeys, except the last, are  
removed. It's simply not worth the overhead to reparse and maintain  
the data structures each time to ensure we have at least one good full  
set of data and to track/ensure which synckey is the authoritative  
data at the root of the tree so to speak.

>> If what you are talking about would be able to handle this  
>> transparently, then that would be a complete solution for me,  
>> otherwise I will still probably need to track UIDs and flags  
>> independently if not using QRESYNC.
>
> Browsed the IMAP code in ActiveSync, and I don't claim to be an  
> expert, but these are my thoughts after the above discussion (these  
> are my opinions; ymmv):
>
> 1.) You need to replace your "QRESYNC" check with a "CONDSTORE"  
> check instead.

Yeah, makes sense now. Thanks for the clarification.

> 2.) Store UIDs (or at least min/max UIDs) for CONDSTORE servers to  
> optimize IMAP server commands.

Agree here now too. Thanks.

> 3.) Storing flag state for CONDSTORE servers is not necessary (this  
> seems to be what you are doing in  
> Horde_ActiveSync_Imap_Adapter#getMessageChanges()).  If a UID is  
> indicated as changed, just go ahead and re-sync that particular  
> entry, even if it doesn't cause any noticeable change at the  
> activesync UI level.  Any gain you might get from the fact that the  
> flag change isn't one that you care about, or the flag change was  
> subsequently reversed, is offset by the overhead in caching this data.

I'm not storing the flag changes here. I am setting the current flags  
for the messages that have been reported as changed in the folder  
object that is used to transport the changes back to the activesync  
code from the imap adapter. This is basically the information that is  
to be transmitted to the device. When CONDSTORE is not available, the  
data is not persisted server side.

> 4.) Ignore all flag changes in non-CONDSTORE servers.

I would consider this if CONDSTORE is ubiquitous.

> What this means is that for any mailbox, you have the following  
> state entries:
>
> "Before sync" state: IMAP mailbox cached state (UIDNEXT,  
> UIDVALIDITY, HIGHESTMODSEQ), list of UIDs on the server
> "After sync" state: new IMAP mailbox cached state (UIDNEXT,  
> UIDVALIDITY, HIGHESTMODSEQ), list of UIDs sync'd
>
> All FETCH data should be cached by the IMP Imap Client cache object.  
>  You should not be caching any FETCH details in activesync code.

I don't *think* I'm caching this anywhere now...

-- 
mike

The Horde Project (www.horde.org)
mrubinsk at horde.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 6062 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.horde.org/archives/dev/attachments/20121106/4e272f20/attachment.bin>