[sync] bug #157: charset hell

[Karsten Fourmont]

Hi,

I digged a bit deeper into the whole charset nightmare thing.

Horde API functions import and export data in charset NLS::getCharset(), 
which is generally UTF8, as long as the browser supports it. Conversion 
is done in the backend (like sql) driver. So that's something like 
option 2c) of my previous post.

Normally the NLS::getCharset is determined by language and browser 
capabilities. However that's not helpful for headless operation. Here 
the charset is better determined by the encoding in <?xml 
encodig="whatever">. So I added a setCharset method to NLS, allowing the 
rpc code to manually set the charset. Seems to work fine: sync4j umlauts 
are back working (at least for me...)

I also checked in a small patch that makes the initial SlowSync fork 
even when the history entries are missing. This solves some "my initial 
slowsync does not sent entries from server to client".

More to come.

Cheers,
  Karsten



Karsten Fourmont wrote:
> Hi,
> 
> when using the sync4j outlook connector, umlauts seem to be broken with 
> current cvs. The sync4j charset is correctly recognized as iso-85591 as 
> can be seen in the following log line
> 
> SyncML: authorized=1 version=1 msgid=1 source=sc-pim-outlook
> target=http://voltaire.local user=karsten charset=ISO-8859-1
> 
> it's the value of $GLOBALS['_HORDE_STRING_CHARSET'].
> 
> So I thought the charset conversion would happen automagically somewhere
> close to the database. I thought this used to work in early 2005. But 
> now it's broken.
> 
> Any ideas? Where would be the best place to do the charset conversion?
> Maybe add some charset parameter to the various lib/api.php functions?
> 
> I'm not sure where the carset conversion should occur. Some options:
> 
> 1) when invoking lib/api.php functions, data is supossoed to be passed 
> in UTF8. Conversion has to be done prior to invoking api functions (in 
> the SyncML code)
> 
> 2) when invoking api functions, data is supossoed to be passed in 
> charset $GLOBALS['_HORDE_STRING_CHARSET']. Conversion is done
> 
> 2a) by the lib/api.php functions themself
> 
> 2b) by the iCalendar module: input is $GLOBALS['_HORDE_STRING_CHARSET'], 
> output is utf8
> 
> 2c) somewhere else very close to the backend driver...
> 
> Any suggestions?
> 
> Cheers,
>   Karsten