Forum OpenACS Q&A: Character Set Problem with web form input

I have a characterset problem reading german umlauts from web forms.
(Using OpenACS4 and PG7.1 with LATIN-1 Encoding)

These german umlauts are converted to strange characters as i can see 
from log file notices  ( ö becomes ö for example ). If written back 
to the browser these characters are converted back again and 
displayed correctly. But into the database ö and friends are 
written. 

This conversion happens only with web form input. Data read from text 
files gets successfully written to the database (as ö,ä ...). 

I have tried changing the [encoding system] but that did not help. 
The other Aol Server patches seemed to deal with writting back issues 
not with reading from forms. 

Interestingly ns_getform claims to use "iso8859-1 for charset iso-
8859-1" that would be correct, but then why are these strange 
characters generated ?

TIA

<pre>
tils@tp:~$ psql -l
        List of databases
Database  |  Owner  | Encoding
-----------+----------+----------
oacs      | tils    | UNICODE

...
</pre>

About the weird characters in the error log: that's ok - propably because the terminal or whatever you use to look at the log does not understand unicode. Also psql will display those weird characters in an otherwise correct setup. There are some older threads here on this topic where this is explained much better.
<p>
Depending on your version of OpenACS and Aolserver you might need some of Henry Minski's patches to make form input work correctly.
<p>
I run a recent cvs checkout of OpenACS 4 and Aolserver 3.3ad13, and that combination seems to contain all necessary fixes already.

There's a simpler method of doing this, even without the AD version of AOLServer (+ad13). You can use the current 3.4.2 AOLServer from aolserver.com or Sourceforge.

The only thing you need to do is place some undocumented params in your config.tcl:

Then it's even ok if you run "ASCII" encoding in your Postgres database, Latin1 will work also. I don't think (as by now) you have to use the (slow) Unicode stuff with Postgres.

For what it's worth, my solution was different than what everyone else on these boards has ever told me.

In fact, even this site (openacs.org) seems to be doing it wrong because this particular message is being sent to me iso-8859-1 encoded. It should be encoded utf-8 so that all the characters display properly.

As a test, if you're using IE, you can go to View | Encoding and play around a bit and see if the characters don't clear up.

Anyway, what I've found is that you have to set the encoding to UTF-8, and then the characters display properly.

What I've done in my nsd.tcl file is:

ns_param        HackContentType 1
ns_param        URLCharset      "utf-8"
ns_param        OutputCharset   "utf-8"
ns_param        HttpOpenCharset "utf-8"

I also changed several functions in ad-utilities.tcl.preload, particularly ReturnHeaders, ad_return_top_of_page, and all the other ReturnHeaders* functions to show:

Content-Type: $content_type; charset=utf-8

Then everything worked nicely. I suppose you have to have multibyte support compiled into PG, but based on some simple testing on my computer (I don't remember if I enabled multibyte support -- my databases are of type SQL_ASCII), and everything works just fine.

I hope this helps the people out there who are having troubles...