[imp] Only some messages download and display over ActiveSync on phones

[Michael J Rubinsky]

Quoting Sebastian Arcus <s.arcus at open-t.co.uk>:

> On 10/09/2021 00:16, Michael J Rubinsky wrote:
>>
>> Quoting Sebastian Arcus <s.arcus at open-t.co.uk>:
>>
>>> I am facing a really strange issue with Horde/Imp on mobile  
>>> devices. A quick background:
>>>
>>> The setup is:
>>>
>>> Horde 5.2.21
>>> Lighttpd 1.4.59
>>> PHP 7.3.10
>>> Dovecot 2.3.7.2
>>> Exim 4.92.2
>>>
>>> This install has been running for years, and as far as I can think  
>>> nothing has been updated or reconfigured recently - at least not  
>>> in the last few weeks.
>>>
>>> In the last few days, one of the users has started to experience  
>>> some emails not downloading on his mobile devices - for no  
>>> explainable reason.
>>>
>>> 1. The emails are showing up fine over IMAP in Thuderbird.
>>> 2. The mobile devices with problems are two iPhones, one iPad, and  
>>> I just tested it in Bluemail on an Android phone.
>>> 3. It isn't always the same messages - if I remove and re-create  
>>> the accounts, I get a different random selection of messages  
>>> downloaded, after which the sync stops. For example I might have  
>>> most messages from today, then yesterday and the day before  
>>> missing, then some messages from the day before. There is nothing  
>>> logical about it.
>>>
>>> I've checked everywhere I can think of:
>>>
>>> 1. I've forced resyncing of mailboxes in Dovecot (although I don't  
>>> think the problem is there, as all messages show properly in  
>>> Thunderbird
>>> 2. I've changed email passwords
>>> 3. I've deleted and re-added email accounts on the iPhone
>>> 4. I've hard-reset the iPhones, to try to clear the email cache
>>> 5. I've configured the email accounts on an Android phone - still  
>>> same issues
>>> 6. I've checked Horde ActiveSync error logs - aside from errors  
>>> about duplicate sync keys, there doesn't seem to be anything else  
>>> there.
>>> 7. I've tried downloading emails over internal wifi and over 4G
>>> 8. I've changed sync span on mobiles between 1 day, 3 days, 1  
>>> week, 1 month - the same random behaviour persists
>>>
>>> This is happening just to this user, who has two email accounts  
>>> configured on two iPhones and one iPad. All other 10 users are  
>>> fine. Everyone works off the same server, receiving email on  
>>> iPhones.
>>>
>>> I've spent 6 hours on this and frankly I can't make any sense of  
>>> it. Any suggestions would be much appreciated. Thank you.
>>
>> I'd really need to see a sync log and imap debug log to make an  
>> educated guess as to what might be happening. Some things that  
>> might be related, off the top of my head:
>>
>>  * Is this user's prefs set to move deleted messages to Trash or  
>> mark them as deleted?
>>  * Some memory issue or timeout being hit due a large number/size  
>> of messages?
>>  * Unlikely, since you say it happens to random messages, but might  
>> be some issue with the strategy we use to determine message changes  
>> (for Dovecot this would probably be MODSEQ based).
>>
>> If you can get me a copy of the sync log for this user I might be  
>> able to help narrow down the issue. Feel free to mail me directly  
>> mrubinsk at horde.org.
>
> Just checking if you got my email with the logs. The phone is still  
> syncing the emails completely erratically. The only error I keep on  
> seeing in the logs is:
>
> 2021-09-16T13:36:21+01:00 ERR: HORDE SQL QUERY FAILED:  
> SQLSTATE[23505]: Unique violation: 7 ERROR:  duplicate key value  
> violates unique constraint "horde_activesync_state_pkey"
>
> Or any other suggestions I could try? Thank you in advance
> -- 
> imp mailing list
> Frequently Asked Questions: http://wiki.horde.org/FAQ
> To unsubscribe, mail: imp-unsubscribe at lists.horde.org


The separate accounts are not using the same synckey, though I can see  
why you might have thought so. The first number in the log entry is  
the process id of the request. Since the same log is being used for  
multiple accounts on the same device, you need to make sure you are  
looking at the log entries from the correct request. The entries are  
interspersed since they requests and responses from different accounts  
can happen asynchronously.

That being said, the issue has something to do with data being sent in  
one of the sync responses.  That's why you are seeing a log of the  
duplicate key value errors. The device isn't accepting (or perhaps  
didn't receive) the response, so it's requesting the same set of  
changes again (by using the same synckey).  It's really hard to say  
what, if anything, the device doesn't like about that response, but if  
I had to take a wild guess I would say it has something to do with the  
emoji characters in the subject line.

Those characters seem to work fine for me on an android device. I  
don't have an iOS device handy right now for testing, but I can get  
access to one. When I can, I'll see if I can find anything more out.   
In the meantime, do other iOS devices on your server receive those  
types of subjects without issue?





-- 
mike
The Horde Project
http://www.horde.org
https://www.facebook.com/hordeproject
https://www.twitter.com/hordeproject
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-keys
Size: 9272 bytes
Desc: PGP Public Key
URL: <https://lists.horde.org/archives/imp/attachments/20210919/e52e8ac7/attachment.key>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 851 bytes
Desc: PGP Digital Signature
URL: <https://lists.horde.org/archives/imp/attachments/20210919/e52e8ac7/attachment.sig>