[imp] Only some messages download and display over ActiveSync on phones

[Sebastian Arcus]

On 19/09/2021 17:06, Michael J Rubinsky wrote:
> 
> Quoting Sebastian Arcus <s.arcus at open-t.co.uk>:
> 
>> On 10/09/2021 00:16, Michael J Rubinsky wrote:
>>>
>>> Quoting Sebastian Arcus <s.arcus at open-t.co.uk>:
>>>
>>>> I am facing a really strange issue with Horde/Imp on mobile devices. 
>>>> A quick background:
>>>>
>>>> The setup is:
>>>>
>>>> Horde 5.2.21
>>>> Lighttpd 1.4.59
>>>> PHP 7.3.10
>>>> Dovecot 2.3.7.2
>>>> Exim 4.92.2
>>>>
>>>> This install has been running for years, and as far as I can think 
>>>> nothing has been updated or reconfigured recently - at least not in 
>>>> the last few weeks.
>>>>
>>>> In the last few days, one of the users has started to experience 
>>>> some emails not downloading on his mobile devices - for no 
>>>> explainable reason.
>>>>
>>>> 1. The emails are showing up fine over IMAP in Thuderbird.
>>>> 2. The mobile devices with problems are two iPhones, one iPad, and I 
>>>> just tested it in Bluemail on an Android phone.
>>>> 3. It isn't always the same messages - if I remove and re-create the 
>>>> accounts, I get a different random selection of messages downloaded, 
>>>> after which the sync stops. For example I might have most messages 
>>>> from today, then yesterday and the day before missing, then some 
>>>> messages from the day before. There is nothing logical about it.
>>>>
>>>> I've checked everywhere I can think of:
>>>>
>>>> 1. I've forced resyncing of mailboxes in Dovecot (although I don't 
>>>> think the problem is there, as all messages show properly in 
>>>> Thunderbird
>>>> 2. I've changed email passwords
>>>> 3. I've deleted and re-added email accounts on the iPhone
>>>> 4. I've hard-reset the iPhones, to try to clear the email cache
>>>> 5. I've configured the email accounts on an Android phone - still 
>>>> same issues
>>>> 6. I've checked Horde ActiveSync error logs - aside from errors 
>>>> about duplicate sync keys, there doesn't seem to be anything else 
>>>> there.
>>>> 7. I've tried downloading emails over internal wifi and over 4G
>>>> 8. I've changed sync span on mobiles between 1 day, 3 days, 1 week, 
>>>> 1 month - the same random behaviour persists
>>>>
>>>> This is happening just to this user, who has two email accounts 
>>>> configured on two iPhones and one iPad. All other 10 users are fine. 
>>>> Everyone works off the same server, receiving email on iPhones.
>>>>
>>>> I've spent 6 hours on this and frankly I can't make any sense of it. 
>>>> Any suggestions would be much appreciated. Thank you.
>>>
>>> I'd really need to see a sync log and imap debug log to make an 
>>> educated guess as to what might be happening. Some things that might 
>>> be related, off the top of my head:
>>>
>>>  * Is this user's prefs set to move deleted messages to Trash or mark 
>>> them as deleted?
>>>  * Some memory issue or timeout being hit due a large number/size of 
>>> messages?
>>>  * Unlikely, since you say it happens to random messages, but might 
>>> be some issue with the strategy we use to determine message changes 
>>> (for Dovecot this would probably be MODSEQ based).
>>>
>>> If you can get me a copy of the sync log for this user I might be 
>>> able to help narrow down the issue. Feel free to mail me directly 
>>> mrubinsk at horde.org.
>>
>> Just checking if you got my email with the logs. The phone is still 
>> syncing the emails completely erratically. The only error I keep on 
>> seeing in the logs is:
>>
>> 2021-09-16T13:36:21+01:00 ERR: HORDE SQL QUERY FAILED: 
>> SQLSTATE[23505]: Unique violation: 7 ERROR:  duplicate key value 
>> violates unique constraint "horde_activesync_state_pkey"
>>
>> Or any other suggestions I could try? Thank you in advance
>> -- 
>> imp mailing list
>> Frequently Asked Questions: http://wiki.horde.org/FAQ
>> To unsubscribe, mail: imp-unsubscribe at lists.horde.org
> 
> 
> The separate accounts are not using the same synckey, though I can see 
> why you might have thought so. The first number in the log entry is the 
> process id of the request. Since the same log is being used for multiple 
> accounts on the same device, you need to make sure you are looking at 
> the log entries from the correct request. The entries are interspersed 
> since they requests and responses from different accounts can happen 
> asynchronously.
> 
> That being said, the issue has something to do with data being sent in 
> one of the sync responses.  That's why you are seeing a log of the 
> duplicate key value errors. The device isn't accepting (or perhaps 
> didn't receive) the response, so it's requesting the same set of changes 
> again (by using the same synckey).  It's really hard to say what, if 
> anything, the device doesn't like about that response, but if I had to 
> take a wild guess I would say it has something to do with the emoji 
> characters in the subject line.
> 
> Those characters seem to work fine for me on an android device. I don't 
> have an iOS device handy right now for testing, but I can get access to 
> one. When I can, I'll see if I can find anything more out.  In the 
> meantime, do other iOS devices on your server receive those types of 
> subjects without issue?

Just an update - it's been 3 days since I connected only one device to 
the email account - and the problem persists. So it seems having several 
devices connected to the same account is not the cause. I will try and 
take a closer look at the emoji characters possibility - maybe something 
starts to make sense there. Thank you for the suggestion!