Forum OpenACS Q&A: OpenACS Internationalization

---

I would really like to see internationalization support in OpenACS, but I would present this problem in the language that I am trying to make OpenACS to work with...

I wanna bring this thread up again because it seems that no one has ever gotten OpenACS to work with the Chinese language (or any other non-Eng language?). If anyone has any suggestion / comment / experience with this issue, please post. Here is what I did recently:

I have read Building a Multilingual Web Service Using ACS in ASJ and have attempted to use OpenACS with Big5 character encoding (Tranditional Chinese). The ASJ doc mentions that there are a few levels of problems that have to be resolved before ACS can handle international character encodings.


1. the RDBMS level
2. the DBMS driver level
3. the aolserver level
4. the ACS Tcl code level

In my attempt to make my installation of OpenACS to work with my desired language, here is what I did to address point 1 and 3 of the above problems respectively:

1. building the PostgreSQL database with UNICODE or EUI_TW encoding (I tried both)

3. Using the latest aolserver3.2+ad10

I was immature enough to test if OpenACS would work with Chinese after doing the above two solutions ONLY. I tried using different nsd binaries, the nsd76 and nsd8 as well, because there were postings about differences in handling of double-byte characters in tcl76 and tcl8.

The way I test if OpenACS would work was to submit Chinese characters in the user biography and retrieve to see if it presents the same Chinese characters. All I got was junk characters.

In short, It didn't work.

The remaining 2 problems that have to be resolved are the DB driver and ACS TCL code.

I would like to know how the pgdriver v1.1 (the official OpenACS driver) handles character encodings. Is there any encoding translation? Or does it do SQL queries in UNICODE? If not, is it a planned feature in the near future?

For the very last problem, the ACS Tcl problem, I assume we can port the internationalization patch for ACS to OpenACS available here

If you have ever work with other languages with OpenACS, please post your experience. Any input would be greatly appreciated.

I have a similar problem with german umlauts, here is a summary of some tests (with umlauts instead of Chinese characters).

Aolserver version is always AOLserver/3.2+ad10.


> 1. Test PG directly. Create a table with a
> "text" field, then insert and retrieve Chinese
> characters. Make sure that works.

That works. Postgres has Multibyte enabled and the database is in
UNICODE encoding (according to encoding).


> 2. Write a simple Tcl script that retrieves rows
> ALREADY IN THE TABLE, ns_write them,
> and see if you get Chinese characters out.

Works both with nsd76 and nsd8x for me.

Writing posted values into a file: works with both.

Sending mail with ns_sendmail always works (posted values, read from
db, read from file) with both.

Reading a file from the file system with umlauts in it and storing
its text in the db: works with nsd76, fails with nsd8x.


So they only place where characters get translated wrong seems to be
on the aolserver->postgres path when using nsd8x.


> (try with nsd76 and nsd8x - 76 is much better
> for OpenACS in general because Tcl 8x has
> memory leaks associated with regexps).

Somewhere at arsdigita I read that those issues are now resolved and
everybody should use nsd8x. They will eventually stop support for
nsd76 - e.g. ns_cache only works with nsd8x.

Unfortunately there seems to be no other alternative than to use nsd76
right now if you need a non US encoding.


Is it safe to say that according to the tests above it is an issue
with the nspostgres driver?


> I know that one set of bug fixes for PG 7.0.3
> involved internationalization. I suggest
> downloading and installing that with the right
> switches, for starters.

Tried it, same troubles. (PostgreSQL 7.0.3 on i686-pc-linux-gnu,
compiled by gcc 2.95.2)

Hi MK Tam,

What version of the PostgresQL driver are you using when you got success in the bboard with Big5?

Thanks in advance for your response.

Hi MK Tam,

How about the localization (or locale, whatever it's called) setting in PostgreSQL? What did you use? Unicode ? or TW? Thanks for your response.

Ok here are more detailed tests that my friend and I went thru today, we used RPM installations for OpenACS:

1. Using psql, insert into and select Big5 (double-byte character encoding for Chinese) data from the database, with both UNICODE and EUI_TW multi-byte database settings..... *all worked*.

2. Simple tcl proc to select the existing Big5 data from the database, thru the OpenACS architecture (browser << aolserver + tcl << pgdriver << postgresql)..... *worked*

3. Simple tcl proc to insert Big5 data into the database, thru the OpenACS architecture (browser >> aolserver + tcl >> pgdriver >> postgresql) and then check data with the same proc from #2 above..... ****FAILED****

Can we safely say that it is the problem of aolserver + tcl now? And if so, is there any solution to this?

I'm a German native speaker and I succeeded in using all
(Western)-European Languages by simply putting the following Line at
the end of ReturnHeaders and ReturnHeadersNoCache:

ns_startcontent -charset "iso-8859-1"

The reason why this works is simple: Tcl uses UTF8, but I'd like
iso-8859-1. So the ns_startcontent proc tells Tcl to use iso-8859-1.
You should be able to put in any charset because the headers are
still written in whatever they are written normally, as the
charset-conversion starts below the header-output.

BTW, I use openacs 3.2.4, SuSE 7.0/7.1 and PostgreSQL 7.0.3 from
rpm.

Hi Tilmann!

encoding in psql gives me SQL_ASCII for encoding. I followed all
threads on encoding at openacs.org but nothing worked for me, so I
tried it myself.

Now I have no problems with strings that are sent via post or get
operations or retrieved from the database.

But I've also put this in my server.tcl file:

set charset iso-8859-1

ns_section "ns/parameters"

ns_param  URLCharset    $charset

ns_param  OutputCharset $charset

ns_param  HackContentType true

But only after the modifications of ReturnHeaders and
ReturnHeadersNoCache the whole thing worked and I've had no problems
so far. I've tested it heavily using german umlauts.

I would like to direct those people who are interested in Internationalization of OpenACS to this thread too:

OpenACS Internationalization HOWTO v0.1

Nevermid, it was my fault.

I had been playing with ns_setformencoding and i had place two of them in set_the_usual_form_variables.

After i've left that proc in its original state, everything's working smoothly. Sorry.