Forum OpenACS Q&A: Asian Characters and UTF-8 Encoding - Any experiences?

Hi Frank,
I've got some experience with character set problems (not with Asian characters but I'm sure the principles apply). You say that Oracle is using US7ASCII encoding - that definitely doesn't sound good. You should change it to UTF-8. Run this query to check the character set:

select * from NLS_DATABASE_PARAMETERS
where parameter = 'NLS_CHARACTERSET';

You should see the following if it's UTF-8:
PARAMETER                      VALUE
------------------------- ------------------------------
NLS_CHARACTERSET              UTF8

If, as you say, your character set is not UTF-8, you have a bit of work ahead of you. I doubt your Asian characters were entered correctly, so you'll need to clean out your data, convert your database to UTF8 and re-enter the data.

Also put the following settings in AOLserver's configuration file (in the ns_section ns/parameters section):

ns_param  HackContentType    1
ns_param  DefaultCharset    utf-8
ns_param  HttpOpenCharset    utf-8
ns_param  OutputCharset      utf-8
ns_param  URLCharset        utf-8

Add this line to the AOLserver wrapper script to set your environment variable:
set NLS_LANG=.UTF8

Here's an excellent resource from Oracle
http://web.archive.org/web/20050213010105/http://www.oracle.com/technology/products/oracle8i/htdocs/faq_combined.htm

Hope this helps,
Brian

Thanks a lot, Brian!

That sounds excellent. To conclude your statement: There is no other nasty issue with double-byte characters (like: Oracle driver implementation, screwed AOLServer implementation, ...) except for the Oracle charset.

And yes, our default Oracle charset is still US7ASCII, following a recomendation of Philip of around 1999...

Bests,
Frank

(Not that it's relevant, but I rather doubt that Philip G. ever recommended ASCII7 for the Oracle character set, even in 1999.)

The fact is that at least through Oracle 8.1.7.x, ASCII7 is the default. Oracle also stupidly hides this in their installer, so unless you specifically look for it when installing Oracle, you will almost certainly end up with with an ASCI7 rather than UTF8 database - bad. My old outdated Oracle install notes have some more info on that UTF8 character set stuff.

Ni Nis,

thanks a lot for your comment. May I ask you why you are using different encodings for Russian and Chinese?

Bests,
Frank

Richard, in your AOLserver config file, do you perhaps have some settings like these?:

ns_section ns/parameters
   ns_param OutputCharset iso-8859-1
   ns_param HackContentType 1

ns_section ns/MimeTypes
   set mime_plain {text/plain; charset=iso-8859-1}
   set mime_html  {text/html; charset=iso-8859-1}

   # See also "http://dqd.com/~mayoff/encoding-doc.html" for advice on
   # character sets and MIME types in AOLserver.

   ns_param Default     $mime_plain
   ns_param NoExtension $mime_plain
   ns_param .txt  $mime_plain
   ns_param .text $mime_plain
   ns_param .htm  $mime_html
   ns_param .html $mime_html

That's what I use in AOLserver 4.0.10, but note that I am purposely serving only iso-8859-1 content. If you are trying to serve UTF-8, some of the settings above would probably break stuff for you.

Hi All,

Well I was searching for Unicode to Arabic conversions and came across this post.

What I am trying to do is convert some unicode data that I have into Arabic, there is a function written that does the conversion using a sequence of Case stmts and it searches for the unicode chars and then replaces it with the corresponding Arabic character.

e.g. IF strT = '067E' THEN Dest := Dest || Chr(129 USING NCHAR_CS); --129Arabic Peh

Well the function works correctly in Oracle 9i and above but refuses to compile in versions of Oracle below 9i.

Here are some more details:
Oracle Version: Oracle 8.1.7
Compilation error: PLS 00561- Characterset mismatch on value for parameter 'Right'.

I am assuming it is the Nchar_cs that creates the problem.

Would appreciate any help on this. Thx in Advance

Regards
Chetz

Hi to all
I have a problem. I connect Oracle with php. It works, but it has two problems. ( I have two problems).
It withdrows all the columns twice, and doesnt show the data which is put in the data base with cyrilic encoding. THe encoding of the data base is UTF-8. Also I tried to set the encodung in HTML - PHP code UTF-8,but it doesn't work.
I have the same problem with mySql. But I find SQL query which soles this problem. the code is "mysql_query("set names cp 1251"); ", which is inserted right after the code for connection with the databese.
I try this query with oracle, but it doesn't work.

Please help!

Evica

Hey Brad

I've never used the NCHAR and NVARCHAR stuff, so maybe that is the way to go, but I understood that you might run into trouble having your database character set as WE8ISO8859P1. It might be worth running a quick test with a new UTF8 database, just to rule that out.

I presume you tried the AOLServer config file settings described above?

Does it work if you hardcode some Chinese characters directly into an INSERT statement in a Tcl proc?

Brian