[dev] [commits] Horde branch master updated. 8527f1d6995f63567686e958ca5dd758c8188367

[Michael M Slusarz]

Quoting Michael J Rubinsky <mrubinsk at horde.org>:

> Quoting Michael M Slusarz <slusarz at horde.org>:
>
>> Quoting "Michael J. Rubinsky" <mrubinsk at horde.org>:
>>
>>> commit 8527f1d6995f63567686e958ca5dd758c8188367
>>> Author: Michael J Rubinsky <mrubinsk at horde.org>
>>> Date:   Wed Dec 12 12:44:38 2012 -0500
>>>
>>>   Hopefully finally resolve charset issues breaking iOS syncing.
>>>
>>>   Must ensure the default charset is us-ascii for messages that contain
>>>   no charset information.
>>>
>>> framework/ActiveSync/lib/Horde/ActiveSync/Imap/Adapter.php |    9 +++++----
>>> 1 files changed, 5 insertions(+), 4 deletions(-)
>>
>> You may have addressed this in further commits... but IMHO you  
>> should default to UTF-8 for this.  Considering that US-ASCII is a  
>> complete subset of UTF-8, odds are that this would catch a few  
>> additional messages that contain non-7bit characters.
>
> The problem I'm trying to prevent here is a broken email that  
> contains no charset information, yet contains non ascii AND  
> non-utf-8 data from breaking the sync. For example, a spam email  
> containing some invalid UTF-8 data with no charset information for  
> the part. Before this change, because of the various default charset  
> prefs, the default *was* being set to UTF-8 (in  
> IMP_Application::_init IIRC). During the process of building the  
> activesync message we perform a Horde_String::convertCharset on the  
> body since ActiveSync messages MUST contain only valid UTF-8 data.  
> Without using the 'force' option, the charset conversion is never  
> performed - so the invalid data is sent to the device and can cause  
> the sync to break completely on iOS (Android is a bit more tolerant  
> here, as it will ignore the bad data). I was originally setting the  
> 'force' option, but this mangled valid UTF-8 data when it passed  
> through Utf8_encode().
>
> I now also call Horde_String::validUtf8() on the data and re-convert  
> the charset with 'force' if invalid utf-8 data is found. It's extra  
> work, but the alternative is having a random email completely break  
> the client's ability to sync with any mailbox.
>
> This has been a particularly annoying and problematic issue that I  
> (hope) is now finally resolved.

We do something similar to this in Horde_Mime_Headers when trying to  
auto-detect the charset of headers that are not correctly MIME  
encoded.  We try the most common non-ASCII charsets (UTF-8,  
windows-1252).  Although that code, if it fails, outputs as-is, since  
it is a display issue and won't break things, like your situation.  An  
option in this latter case would be to simply strip out non-7bit  
characters instead.

michael

___________________________________
Michael Slusarz [slusarz at horde.org]