[imp] Encoding woes with the preview pane in DIMP from Horde 3

[Gunnar Wrobel]

Quoting Jan Schneider <jan at horde.org>:

> Zitat von Gunnar Wrobel <wrobel at horde.org>:
>
>> Hi!
>>
>> When removing the various locales on the base system where Horde 3  
>> is installed I can break the message preview in DIMP. Pretty much  
>> the same thing described in http://bugs.horde.org/ticket/5955
>>
>> That issue is however closed as not being a bug and I'm not certain  
>> I understand the issue completely so I thought I ask here.
>>
>> Horde 3 is running on Ubuntu 10.04 and I removed the locales with  
>> these commands ...
>>
>> apt-get remove locales
>> rm -rf /usr/lib/locale/*.utf8
>>
>> ... cleared the Horde cache afterwards ....
>>
>> rm /tmp/cache_*
>>
>> ... and restarted the apache server:
>>
>> /etc/init.d/apache restart
>>
>> For messages that contain German Umlauts (that were displayed fine  
>> before) the DIMP preview is empty now. At the same time I get this  
>> stack trace in the PHP error log:
>>
>>
>> [25-Aug-2011 17:14:02] PHP Warning:  json_encode() [<a  
>> href='function.json-encode'>function.json-encode</a>]: Invalid  
>> UTF-8 sequence in argument in /www/client/imp/lib/JSON.php on line 86
>> [25-Aug-2011 17:14:02] PHP Stack trace:
>> [25-Aug-2011 17:14:02] PHP   1. {main}() /www/client/dimp/imp.php:0
>> [25-Aug-2011 17:14:02] PHP   2. IMP::sendHTTPResponse()  
>> /www/client/dimp/imp.php:767
>> [25-Aug-2011 17:14:02] PHP   3. IMP_Serialize_JSON::encode()  
>> /www/client/imp/lib/IMP.php:1883
>> [25-Aug-2011 17:14:02] PHP   4. json_encode()  
>> /www/client/imp/lib/JSON.php:86
>>
>> As far as I understand it so far the following happens:
>>
>> When the application gets pushed in the Registry the code tries to  
>> set the language environment (NLS::setLang()). This method will try  
>> to enable the correct locale but will also check that the locale  
>> with the current character set set actually exists. In case the  
>> browser supports "utf" the system will try for character set "UTF-8".
>>
>> Since I removed those locales however NLS::setLang() will fall back  
>> to a working character set. In my case the 'ISO-8859-1' charset.  
>> After that NLS::getCharset() will always return this value.
>>
>> Once the system tries to show the message preview it looks at the  
>> message converts it to the current character set ('ISO-8859-1') and  
>> pipes it into json_encode(). The latter one chokes as it expects  
>> UTF-8 and the preview breaks.
>>
>> I don't think I have the full picture yet so I'm mainly asking if  
>> my observation could be correct. If yes, is this to be expected and  
>> should the user just ensure that all locales are present in UTF-8?  
>> Or is this something that could easily be fixed?
>
> This is the expected behavior. We require UTF-8 support of the  
> system for Horde 4 which basically means UTF-8 locales if using  
> translations. I was sure this was even mentioned in the  
> prerequisites section of INSTALL, or at least in TRANSLATIONS, but  
> obviously not.

In the Horde 3 INSTALL instructions there is a small paragraph saying  
"If you still see this problem after upgrading PHP, your server may  
have incorrect locale information.  See `Bug #5955`_ for further  
information." But that is specifically targeted at the preview problem.

For Horde 3 this means I will just ensure we make the necessary  
instructions more prominent in the Kolab server installation text.

But for Horde 4 I could imagine a few improvements as we seem to have  
no problem detecting the locales present:

  1) Add a test for this in test.php so that the admin immediately  
sees which translations/charsets are actually available. This test  
should fail if there is no UTF-8 support in any locale.

  2) We should disable the language selection on the login screen for  
languages not supported by the system.

Does that make sense or did I misinterpret your response?

Cheers,

Gunnar


>
> Jan.
>
> -- 
> Do you need professional PHP or Horde consulting?
> http://horde.org/consulting/
>
> -- 
> IMP mailing list
> Frequently Asked Questions: http://horde.org/faq/
> To unsubscribe, mail: imp-unsubscribe at lists.horde.org

-- 
Core Developer
The Horde Project

e: wrobel at horde.org
t: +49 700 6245 0000
w: http://www.horde.org

pgp: 9703 43BE
tweets: http://twitter.com/pardus_de
blog: http://log.pardus.de