[imp] Encoding woes with the preview pane in DIMP from Horde 3

[Gunnar Wrobel]

Hi!

When removing the various locales on the base system where Horde 3 is  
installed I can break the message preview in DIMP. Pretty much the  
same thing described in http://bugs.horde.org/ticket/5955

That issue is however closed as not being a bug and I'm not certain I  
understand the issue completely so I thought I ask here.

Horde 3 is running on Ubuntu 10.04 and I removed the locales with  
these commands ...

apt-get remove locales
rm -rf /usr/lib/locale/*.utf8

... cleared the Horde cache afterwards ....

rm /tmp/cache_*

... and restarted the apache server:

/etc/init.d/apache restart

For messages that contain German Umlauts (that were displayed fine  
before) the DIMP preview is empty now. At the same time I get this  
stack trace in the PHP error log:


[25-Aug-2011 17:14:02] PHP Warning:  json_encode() [<a  
href='function.json-encode'>function.json-encode</a>]: Invalid UTF-8  
sequence in argument in /www/client/imp/lib/JSON.php on line 86
[25-Aug-2011 17:14:02] PHP Stack trace:
[25-Aug-2011 17:14:02] PHP   1. {main}() /www/client/dimp/imp.php:0
[25-Aug-2011 17:14:02] PHP   2. IMP::sendHTTPResponse()  
/www/client/dimp/imp.php:767
[25-Aug-2011 17:14:02] PHP   3. IMP_Serialize_JSON::encode()  
/www/client/imp/lib/IMP.php:1883
[25-Aug-2011 17:14:02] PHP   4. json_encode() /www/client/imp/lib/JSON.php:86

As far as I understand it so far the following happens:

When the application gets pushed in the Registry the code tries to set  
the language environment (NLS::setLang()). This method will try to  
enable the correct locale but will also check that the locale with the  
current character set set actually exists. In case the browser  
supports "utf" the system will try for character set "UTF-8".

Since I removed those locales however NLS::setLang() will fall back to  
a working character set. In my case the 'ISO-8859-1' charset. After  
that NLS::getCharset() will always return this value.

Once the system tries to show the message preview it looks at the  
message converts it to the current character set ('ISO-8859-1') and  
pipes it into json_encode(). The latter one chokes as it expects UTF-8  
and the preview breaks.

I don't think I have the full picture yet so I'm mainly asking if my  
observation could be correct. If yes, is this to be expected and  
should the user just ensure that all locales are present in UTF-8? Or  
is this something that could easily be fixed?

Cheers,

Gunnar

-- 
Core Developer
The Horde Project

e: wrobel at horde.org
t: +49 700 6245 0000
w: http://www.horde.org

pgp: 9703 43BE
tweets: http://twitter.com/pardus_de
blog: http://log.pardus.de