[imp] Encoding woes with the preview pane in DIMP from Horde 3

Fri Aug 26 09:55:19 UTC 2011

Zitat von Gunnar Wrobel <wrobel at horde.org>:

> Quoting Jan Schneider <jan at horde.org>:
>
>> Zitat von Gunnar Wrobel <wrobel at horde.org>:
>>
>>> Quoting Jan Schneider <jan at horde.org>:
>>>
>>>> Zitat von Gunnar Wrobel <wrobel at horde.org>:
>>>>
>>>>> Hi!
>>>>>
>>>>> When removing the various locales on the base system where Horde  
>>>>> 3 is installed I can break the message preview in DIMP. Pretty  
>>>>> much the same thing described in http://bugs.horde.org/ticket/5955
>>>>>
>>>>> That issue is however closed as not being a bug and I'm not  
>>>>> certain I understand the issue completely so I thought I ask here.
>>>>>
>>>>> Horde 3 is running on Ubuntu 10.04 and I removed the locales  
>>>>> with these commands ...
>>>>>
>>>>> apt-get remove locales
>>>>> rm -rf /usr/lib/locale/*.utf8
>>>>>
>>>>> ... cleared the Horde cache afterwards ....
>>>>>
>>>>> rm /tmp/cache_*
>>>>>
>>>>> ... and restarted the apache server:
>>>>>
>>>>> /etc/init.d/apache restart
>>>>>
>>>>> For messages that contain German Umlauts (that were displayed  
>>>>> fine before) the DIMP preview is empty now. At the same time I  
>>>>> get this stack trace in the PHP error log:
>>>>>
>>>>>
>>>>> [25-Aug-2011 17:14:02] PHP Warning:  json_encode() [<a  
>>>>> href='function.json-encode'>function.json-encode</a>]: Invalid  
>>>>> UTF-8 sequence in argument in /www/client/imp/lib/JSON.php on  
>>>>> line 86
>>>>> [25-Aug-2011 17:14:02] PHP Stack trace:
>>>>> [25-Aug-2011 17:14:02] PHP   1. {main}() /www/client/dimp/imp.php:0
>>>>> [25-Aug-2011 17:14:02] PHP   2. IMP::sendHTTPResponse()  
>>>>> /www/client/dimp/imp.php:767
>>>>> [25-Aug-2011 17:14:02] PHP   3. IMP_Serialize_JSON::encode()  
>>>>> /www/client/imp/lib/IMP.php:1883
>>>>> [25-Aug-2011 17:14:02] PHP   4. json_encode()  
>>>>> /www/client/imp/lib/JSON.php:86
>>>>>
>>>>> As far as I understand it so far the following happens:
>>>>>
>>>>> When the application gets pushed in the Registry the code tries  
>>>>> to set the language environment (NLS::setLang()). This method  
>>>>> will try to enable the correct locale but will also check that  
>>>>> the locale with the current character set set actually exists.  
>>>>> In case the browser supports "utf" the system will try for  
>>>>> character set "UTF-8".
>>>>>
>>>>> Since I removed those locales however NLS::setLang() will fall  
>>>>> back to a working character set. In my case the 'ISO-8859-1'  
>>>>> charset. After that NLS::getCharset() will always return this  
>>>>> value.
>>>>>
>>>>> Once the system tries to show the message preview it looks at  
>>>>> the message converts it to the current character set  
>>>>> ('ISO-8859-1') and pipes it into json_encode(). The latter one  
>>>>> chokes as it expects UTF-8 and the preview breaks.
>>>>>
>>>>> I don't think I have the full picture yet so I'm mainly asking  
>>>>> if my observation could be correct. If yes, is this to be  
>>>>> expected and should the user just ensure that all locales are  
>>>>> present in UTF-8? Or is this something that could easily be fixed?
>>>>
>>>> This is the expected behavior. We require UTF-8 support of the  
>>>> system for Horde 4 which basically means UTF-8 locales if using  
>>>> translations. I was sure this was even mentioned in the  
>>>> prerequisites section of INSTALL, or at least in TRANSLATIONS,  
>>>> but obviously not.
>>>
>>> In the Horde 3 INSTALL instructions there is a small paragraph  
>>> saying "If you still see this problem after upgrading PHP, your  
>>> server may have incorrect locale information.  See `Bug #5955`_  
>>> for further information." But that is specifically targeted at the  
>>> preview problem.
>>>
>>> For Horde 3 this means I will just ensure we make the necessary  
>>> instructions more prominent in the Kolab server installation text.
>>>
>>> But for Horde 4 I could imagine a few improvements as we seem to  
>>> have no problem detecting the locales present:
>>>
>>> 1) Add a test for this in test.php so that the admin immediately  
>>> sees which translations/charsets are actually available. This test  
>>> should fail if there is no UTF-8 support in any locale.
>>>
>>> 2) We should disable the language selection on the login screen  
>>> for languages not supported by the system.
>>>
>>> Does that make sense or did I misinterpret your response?
>>
>> No, this is actually a good idea. Though we probably need an FAQ  
>> entry immediately then, to explain why there aren't any  
>> translations showing up in the drop down list. ;-)
>
> Yes, would probably be much reduced. But I would hope the test.php  
> results already help the admins with the correct setup.
>
> I added http://bugs.horde.org/ticket/10457
>
> Can you enlighten me why we actually require the UTF-8 locales? For  
> the specific situation I was investigating I did not really get it  
> as I assumed that the conversion of character sets does not depend  
> on the locale being present. But then again I'm really not much into  
> encoding - at least not as much as I probably should be.

We no longer support non-UTF-8 capable browsers. And gettext  
determines the language and encoding to return from the currently set  
locale. If the locale cannot be set because it isn't installed,  
gettext cannot provide the translation in the requested encoding. This  
is a limitation of gettext; charset conversion in Horde works  
independent from that and without the locales being installed, because  
it is charset-dependent, not locale-dependent.

Jan.

-- 
Do you need professional PHP or Horde consulting?
http://horde.org/consulting/