[imp] Encoding woes with the preview pane in DIMP from Horde 3

[Jan Schneider]

Zitat von Gunnar Wrobel <wrobel at horde.org>:

> Quoting Jan Schneider <jan at horde.org>:
>
>> Zitat von Gunnar Wrobel <wrobel at horde.org>:
>>
>>> Hi!
>>>
>>> When removing the various locales on the base system where Horde 3  
>>> is installed I can break the message preview in DIMP. Pretty much  
>>> the same thing described in http://bugs.horde.org/ticket/5955
>>>
>>> That issue is however closed as not being a bug and I'm not  
>>> certain I understand the issue completely so I thought I ask here.
>>>
>>> Horde 3 is running on Ubuntu 10.04 and I removed the locales with  
>>> these commands ...
>>>
>>> apt-get remove locales
>>> rm -rf /usr/lib/locale/*.utf8
>>>
>>> ... cleared the Horde cache afterwards ....
>>>
>>> rm /tmp/cache_*
>>>
>>> ... and restarted the apache server:
>>>
>>> /etc/init.d/apache restart
>>>
>>> For messages that contain German Umlauts (that were displayed fine  
>>> before) the DIMP preview is empty now. At the same time I get this  
>>> stack trace in the PHP error log:
>>>
>>>
>>> [25-Aug-2011 17:14:02] PHP Warning:  json_encode() [<a  
>>> href='function.json-encode'>function.json-encode</a>]: Invalid  
>>> UTF-8 sequence in argument in /www/client/imp/lib/JSON.php on line  
>>> 86
>>> [25-Aug-2011 17:14:02] PHP Stack trace:
>>> [25-Aug-2011 17:14:02] PHP   1. {main}() /www/client/dimp/imp.php:0
>>> [25-Aug-2011 17:14:02] PHP   2. IMP::sendHTTPResponse()  
>>> /www/client/dimp/imp.php:767
>>> [25-Aug-2011 17:14:02] PHP   3. IMP_Serialize_JSON::encode()  
>>> /www/client/imp/lib/IMP.php:1883
>>> [25-Aug-2011 17:14:02] PHP   4. json_encode()  
>>> /www/client/imp/lib/JSON.php:86
>>>
>>> As far as I understand it so far the following happens:
>>>
>>> When the application gets pushed in the Registry the code tries to  
>>> set the language environment (NLS::setLang()). This method will  
>>> try to enable the correct locale but will also check that the  
>>> locale with the current character set set actually exists. In case  
>>> the browser supports "utf" the system will try for character set  
>>> "UTF-8".
>>>
>>> Since I removed those locales however NLS::setLang() will fall  
>>> back to a working character set. In my case the 'ISO-8859-1'  
>>> charset. After that NLS::getCharset() will always return this value.
>>>
>>> Once the system tries to show the message preview it looks at the  
>>> message converts it to the current character set ('ISO-8859-1')  
>>> and pipes it into json_encode(). The latter one chokes as it  
>>> expects UTF-8 and the preview breaks.
>>>
>>> I don't think I have the full picture yet so I'm mainly asking if  
>>> my observation could be correct. If yes, is this to be expected  
>>> and should the user just ensure that all locales are present in  
>>> UTF-8? Or is this something that could easily be fixed?
>>
>> This is the expected behavior. We require UTF-8 support of the  
>> system for Horde 4 which basically means UTF-8 locales if using  
>> translations. I was sure this was even mentioned in the  
>> prerequisites section of INSTALL, or at least in TRANSLATIONS, but  
>> obviously not.
>
> In the Horde 3 INSTALL instructions there is a small paragraph  
> saying "If you still see this problem after upgrading PHP, your  
> server may have incorrect locale information.  See `Bug #5955`_ for  
> further information." But that is specifically targeted at the  
> preview problem.
>
> For Horde 3 this means I will just ensure we make the necessary  
> instructions more prominent in the Kolab server installation text.
>
> But for Horde 4 I could imagine a few improvements as we seem to  
> have no problem detecting the locales present:
>
>  1) Add a test for this in test.php so that the admin immediately  
> sees which translations/charsets are actually available. This test  
> should fail if there is no UTF-8 support in any locale.
>
>  2) We should disable the language selection on the login screen for  
> languages not supported by the system.
>
> Does that make sense or did I misinterpret your response?

No, this is actually a good idea. Though we probably need an FAQ entry  
immediately then, to explain why there aren't any translations showing  
up in the drop down list. ;-)

Jan.

-- 
Do you need professional PHP or Horde consulting?
http://horde.org/consulting/