[dev] Question about problem with VANISHED CHANGEDSINCE requests.

[Michael M Slusarz]

Quoting Michael J Rubinsky <mrubinsk at horde.org>:

> We've discussed this issue before, and I realize it is most likely a  
> problem with the dovecot server. I want to verify this before I  
> start a discussion on the dovecot@ list though.
>
> The problem is VANISHED queries sometimes seem to return ALL UIDs  
> that ever existed in the mailbox. It follows no pattern that I can  
> detect - though I had a thought that it might happen if the the  
> email that was associated with the particular MODSEQ value being  
> used in the CHANGEDSINCE query is no longer present in the mailbox.  
> Anyway, once it happens it *always* happens with the given MODSEQ  
> value.
>
> I used to work around it in code by checking if a UID returned was  
> below the expected minimum, but this no longer works due to recent  
> changes in the imap code - it now hits against the server memory  
> limit before I get a chance to check the UIDs.

Can you expand on this?  You shouldn't be running into any memory  
limit with the new code.

> Anyway, I just want to verify that my error makes, and it is indeed  
> a dovecot problem given the following IMAP log:

> C: 8 UID FETCH 1:* UID (VANISHED CHANGEDSINCE 587742)
> S: * VANISHED (EARLIER)  
> 1:119944,119947:123482,123484:128043,128045:130388,130390:130456,130458:132127,132129:135832,135834:136201,136203:136301,136303:136316,136318:138502,138504:140099,140101:141908,141910:142607,142609:142744,142746:142865,142867:143160,143164:143173,143176:143179,143181:143187,143189:143190,143192:143195,143197:143270,143272:143273,143275:143280,143282:143302,143305:143307,143309:143318,143320:143333,143335:143337,143339:143341,143343:143375,143377:143400,143402:143551,143553:143637,143640:143706,143708:144069,144071:144178,144180:144307,144309:144316,144318:144329,144331:144931,144933:144983,144985:145000,145002:145069,145071:145072,145074:145132,145136:145178,145180:145191,145193

Yes, this looks incorrect.  From RFC 5162 [3.2]:

    The VANISHED UID FETCH modifier instructs the server to report those
    messages from the UID set parameter that have been expunged and whose
    associated mod-sequence is larger than the specified mod-sequence.
    That is, the client requests to be informed of messages from the
    specified set that were expunged since the specified mod-sequence.

I'm assuming there's only a handful (if any) deletes since 587742.  So  
yes, all of the UIDs from the set should not be returned.  (If a  
person wanted this, they could issue "UID FETCH 1:* UID (VANISHED  
CHANGEDSINCE 1)").

There's definitely something wonky going on in the code.  There's a  
certain tipping point of modseqs where the values become incorrect.   
For a mailbox with HIGHESTMODSEQ of 54004, I see this:

a uid fetch 1:* UID (VANISHED CHANGEDSINCE 53881)
* VANISHED (EARLIER) 1:37308,37310:40788,40791:41032,41034:41083
a OK Fetch completed.
b uid fetch 1:* UID (VANISHED CHANGEDSINCE 53882)
* VANISHED (EARLIER) 41029:41030,41047:41083
b OK Fetch completed.

You can report over at the dovecot list or let me know if you want me  
to do the same.

michael

___________________________________
Michael Slusarz [slusarz at horde.org]