[imp] Only some messages download and display over ActiveSync on phones

[Sebastian Arcus]

On 19/09/2021 17:06, Michael J Rubinsky wrote:
> 
> Quoting Sebastian Arcus <s.arcus at open-t.co.uk>:
> 
>> On 10/09/2021 00:16, Michael J Rubinsky wrote:
>>>
>>> Quoting Sebastian Arcus <s.arcus at open-t.co.uk>:
>>>
>>>> I am facing a really strange issue with Horde/Imp on mobile devices. 
>>>> A quick background:
>>>>
>>>> The setup is:
>>>>
>>>> Horde 5.2.21
>>>> Lighttpd 1.4.59
>>>> PHP 7.3.10
>>>> Dovecot 2.3.7.2
>>>> Exim 4.92.2
>>>>
>>>> This install has been running for years, and as far as I can think 
>>>> nothing has been updated or reconfigured recently - at least not in 
>>>> the last few weeks.
>>>>
>>>> In the last few days, one of the users has started to experience 
>>>> some emails not downloading on his mobile devices - for no 
>>>> explainable reason.
>>>>
>>>> 1. The emails are showing up fine over IMAP in Thuderbird.
>>>> 2. The mobile devices with problems are two iPhones, one iPad, and I 
>>>> just tested it in Bluemail on an Android phone.
>>>> 3. It isn't always the same messages - if I remove and re-create the 
>>>> accounts, I get a different random selection of messages downloaded, 
>>>> after which the sync stops. For example I might have most messages 
>>>> from today, then yesterday and the day before missing, then some 
>>>> messages from the day before. There is nothing logical about it.
>>>>
>>>> I've checked everywhere I can think of:
>>>>
>>>> 1. I've forced resyncing of mailboxes in Dovecot (although I don't 
>>>> think the problem is there, as all messages show properly in 
>>>> Thunderbird
>>>> 2. I've changed email passwords
>>>> 3. I've deleted and re-added email accounts on the iPhone
>>>> 4. I've hard-reset the iPhones, to try to clear the email cache
>>>> 5. I've configured the email accounts on an Android phone - still 
>>>> same issues
>>>> 6. I've checked Horde ActiveSync error logs - aside from errors 
>>>> about duplicate sync keys, there doesn't seem to be anything else 
>>>> there.
>>>> 7. I've tried downloading emails over internal wifi and over 4G
>>>> 8. I've changed sync span on mobiles between 1 day, 3 days, 1 week, 
>>>> 1 month - the same random behaviour persists
>>>>
>>>> This is happening just to this user, who has two email accounts 
>>>> configured on two iPhones and one iPad. All other 10 users are fine. 
>>>> Everyone works off the same server, receiving email on iPhones.
>>>>
>>>> I've spent 6 hours on this and frankly I can't make any sense of it. 
>>>> Any suggestions would be much appreciated. Thank you.
>>>
>>> I'd really need to see a sync log and imap debug log to make an 
>>> educated guess as to what might be happening. Some things that might 
>>> be related, off the top of my head:
>>>
>>>  * Is this user's prefs set to move deleted messages to Trash or mark 
>>> them as deleted?
>>>  * Some memory issue or timeout being hit due a large number/size of 
>>> messages?
>>>  * Unlikely, since you say it happens to random messages, but might 
>>> be some issue with the strategy we use to determine message changes 
>>> (for Dovecot this would probably be MODSEQ based).
>>>
>>> If you can get me a copy of the sync log for this user I might be 
>>> able to help narrow down the issue. Feel free to mail me directly 
>>> mrubinsk at horde.org.
>>
>> Just checking if you got my email with the logs. The phone is still 
>> syncing the emails completely erratically. The only error I keep on 
>> seeing in the logs is:
>>
>> 2021-09-16T13:36:21+01:00 ERR: HORDE SQL QUERY FAILED: 
>> SQLSTATE[23505]: Unique violation: 7 ERROR:  duplicate key value 
>> violates unique constraint "horde_activesync_state_pkey"
>>
>> Or any other suggestions I could try? Thank you in advance
>> -- 
>> imp mailing list
>> Frequently Asked Questions: http://wiki.horde.org/FAQ
>> To unsubscribe, mail: imp-unsubscribe at lists.horde.org
> 
> 
> The separate accounts are not using the same synckey, though I can see 
> why you might have thought so. The first number in the log entry is the 
> process id of the request. Since the same log is being used for multiple 
> accounts on the same device, you need to make sure you are looking at 
> the log entries from the correct request. The entries are interspersed 
> since they requests and responses from different accounts can happen 
> asynchronously.
> 
> That being said, the issue has something to do with data being sent in 
> one of the sync responses.  That's why you are seeing a log of the 
> duplicate key value errors. The device isn't accepting (or perhaps 
> didn't receive) the response, so it's requesting the same set of changes 
> again (by using the same synckey).  It's really hard to say what, if 
> anything, the device doesn't like about that response, but if I had to 
> take a wild guess I would say it has something to do with the emoji 
> characters in the subject line.
> 
> Those characters seem to work fine for me on an android device. I don't 
> have an iOS device handy right now for testing, but I can get access to 
> one. When I can, I'll see if I can find anything more out.  In the 
> meantime, do other iOS devices on your server receive those types of 
> subjects without issue?

Thank you very much for finding the time to look at the logs - much 
appreciated. In answer to your questions:

1. The same sync key doesn't seem to be used between email accounts, but 
unless I misread the logs, the same sync key seems to be used for the 
same email account but for different devices. I was wondering if that 
can cause this behaviour

2. When I connected an Android device to the same email accounts (at the 
same time as the iPhones), I was experiencing the same sort of partial 
and inconsistent sync behaviour on Android (with BlueMail)

3. As far as I know, none of the other 7 iPhones have experienced so far 
this type of syncing issues - and the system has been in place for about 
6 years. But in all other cases, each email account is connected to only 
one device over ActiveSync and through Horde. This is the only user who 
has multiple devices connected to the same email account(s) through 
ActiveSync (through Horde).

If you let me know exactly which emoji might be at fault, I could run 
some tests using them.

If you need any more info from me or if I can do anything else to test 
things, please let me know.