[imp] Encoding woes with the preview pane in DIMP from Horde 3

[Gunnar Wrobel]

Quoting Jan Schneider <jan at horde.org>:

> Zitat von Gunnar Wrobel <wrobel at horde.org>:
>
>> Quoting Jan Schneider <jan at horde.org>:
>>
>>> Zitat von Gunnar Wrobel <wrobel at horde.org>:
>>>
>>>> Hi!
>>>>
>>>> When removing the various locales on the base system where Horde  
>>>> 3 is installed I can break the message preview in DIMP. Pretty  
>>>> much the same thing described in http://bugs.horde.org/ticket/5955
>>>>
>>>> That issue is however closed as not being a bug and I'm not  
>>>> certain I understand the issue completely so I thought I ask here.
>>>>
>>>> Horde 3 is running on Ubuntu 10.04 and I removed the locales with  
>>>> these commands ...
>>>>
>>>> apt-get remove locales
>>>> rm -rf /usr/lib/locale/*.utf8
>>>>
>>>> ... cleared the Horde cache afterwards ....
>>>>
>>>> rm /tmp/cache_*
>>>>
>>>> ... and restarted the apache server:
>>>>
>>>> /etc/init.d/apache restart
>>>>
>>>> For messages that contain German Umlauts (that were displayed  
>>>> fine before) the DIMP preview is empty now. At the same time I  
>>>> get this stack trace in the PHP error log:
>>>>
>>>>
>>>> [25-Aug-2011 17:14:02] PHP Warning:  json_encode() [<a  
>>>> href='function.json-encode'>function.json-encode</a>]: Invalid  
>>>> UTF-8 sequence in argument in /www/client/imp/lib/JSON.php on  
>>>> line 86
>>>> [25-Aug-2011 17:14:02] PHP Stack trace:
>>>> [25-Aug-2011 17:14:02] PHP   1. {main}() /www/client/dimp/imp.php:0
>>>> [25-Aug-2011 17:14:02] PHP   2. IMP::sendHTTPResponse()  
>>>> /www/client/dimp/imp.php:767
>>>> [25-Aug-2011 17:14:02] PHP   3. IMP_Serialize_JSON::encode()  
>>>> /www/client/imp/lib/IMP.php:1883
>>>> [25-Aug-2011 17:14:02] PHP   4. json_encode()  
>>>> /www/client/imp/lib/JSON.php:86
>>>>
>>>> As far as I understand it so far the following happens:
>>>>
>>>> When the application gets pushed in the Registry the code tries  
>>>> to set the language environment (NLS::setLang()). This method  
>>>> will try to enable the correct locale but will also check that  
>>>> the locale with the current character set set actually exists. In  
>>>> case the browser supports "utf" the system will try for character  
>>>> set "UTF-8".
>>>>
>>>> Since I removed those locales however NLS::setLang() will fall  
>>>> back to a working character set. In my case the 'ISO-8859-1'  
>>>> charset. After that NLS::getCharset() will always return this  
>>>> value.
>>>>
>>>> Once the system tries to show the message preview it looks at the  
>>>> message converts it to the current character set ('ISO-8859-1')  
>>>> and pipes it into json_encode(). The latter one chokes as it  
>>>> expects UTF-8 and the preview breaks.
>>>>
>>>> I don't think I have the full picture yet so I'm mainly asking if  
>>>> my observation could be correct. If yes, is this to be expected  
>>>> and should the user just ensure that all locales are present in  
>>>> UTF-8? Or is this something that could easily be fixed?
>>>
>>> This is the expected behavior. We require UTF-8 support of the  
>>> system for Horde 4 which basically means UTF-8 locales if using  
>>> translations. I was sure this was even mentioned in the  
>>> prerequisites section of INSTALL, or at least in TRANSLATIONS, but  
>>> obviously not.
>>
>> In the Horde 3 INSTALL instructions there is a small paragraph  
>> saying "If you still see this problem after upgrading PHP, your  
>> server may have incorrect locale information.  See `Bug #5955`_ for  
>> further information." But that is specifically targeted at the  
>> preview problem.
>>
>> For Horde 3 this means I will just ensure we make the necessary  
>> instructions more prominent in the Kolab server installation text.
>>
>> But for Horde 4 I could imagine a few improvements as we seem to  
>> have no problem detecting the locales present:
>>
>> 1) Add a test for this in test.php so that the admin immediately  
>> sees which translations/charsets are actually available. This test  
>> should fail if there is no UTF-8 support in any locale.
>>
>> 2) We should disable the language selection on the login screen for  
>> languages not supported by the system.
>>
>> Does that make sense or did I misinterpret your response?
>
> No, this is actually a good idea. Though we probably need an FAQ  
> entry immediately then, to explain why there aren't any translations  
> showing up in the drop down list. ;-)

Yes, would probably be much reduced. But I would hope the test.php  
results already help the admins with the correct setup.

I added http://bugs.horde.org/ticket/10457

Can you enlighten me why we actually require the UTF-8 locales? For  
the specific situation I was investigating I did not really get it as  
I assumed that the conversion of character sets does not depend on the  
locale being present. But then again I'm really not much into encoding  
- at least not as much as I probably should be.

Cheers,

Gunnar


>
> Jan.
>
> -- 
> Do you need professional PHP or Horde consulting?
> http://horde.org/consulting/

-- 
Core Developer
The Horde Project

e: wrobel at horde.org
t: +49 700 6245 0000
w: http://www.horde.org

pgp: 9703 43BE
tweets: http://twitter.com/pardus_de
blog: http://log.pardus.de