Forum OpenACS Q&A: howto run 3.2.5 with European characters on nsd8x

I tried to get nsd8x running couple month ago, but didn't really
make it back then... Does anyone have a working 3.2.x site on nsd8x?
Which steps would I have to make in order to change my current nsd76
system (pg encoding=SQL_ASCII) to nsd8x???

Database setup: the OACS database in PostgreSQL was created with

 createdb -E LATIN2 yourdb 

PGCLIENTENCODING='UNICODE' ; export PGCLIENTENCODING

Now, to get iso 8859-2 output from Aolserver, you need to add a few lines to your nsd.tcl:

 ns_section "ns/parameters"
        ns_param   OutputCharset iso-8859-2
        ns_param   UrlCharset iso-8859-2
        ns_param   HttpOpenCharset iso-8859-2

 ns_section "ns/mimetypes"
    ns_param ".html" "text/html; charset=iso-8859-2"
    ns_param ".txt" "text/plain; charset=iso-8859-2"
    ns_param ".adp" "text/plain; charset=iso-8859-2"

 encoding system iso8859-2

 ns_set put $form $name $value 

 ns_set put $form $name [encoding convertfrom iso8859-2 $value]

Well, that is it, more or less. Two issues remain: getting the right charset in outgoing emails generated by OpenACS and correct ADP parsing. I am not sure how I solved these - IIRC the trick with ADP is to use another parser (from the two available) if you do not get good results from the start. With email - you need to modify sendmail.tcl and add to headers three lines describing the charset, in my case it is:

MIME-Version: 1.0
Content-Type: text/plain; charset=iso-8859-2
Content-Transfer-Encoding: 8bit

As for my setup, I am running Aolserver 3.2ad12 and PostgreSQL 7.1.3 (configured with --enable-unicode-conversion --enable-locale --enable-multibyte=UNICODE)

David, let me know if that works for you. If it does, I will look further into my code - I remember there was some additional tweaking needed to make OACS ADP templates output correct charset as well.

1.) I have it almost working, but while Marcin is proposing to create the db with LATIN2 (in my case LATIN1), Henry is proposing UNICODE...

createdb --encoding=unicode yourdb

createdb -E LATIN2 yourdb

Which should I take? (pros - cons...)

2.) I read some threads about occurring problems with improper sorting using upper() or lower(). Is this correct? If yes, what can I do about it? I read something about Lang=C ??!!??

3.) Will there be a problem importing my existing database (sql-ascii), which is not compiled with unicode-conversion, into the newly compiled postgres with unicode-conversion + unicode or Latin1??

Thanks

1.) In order to make ns_write return the right characters I had to add the following line to the end of the proc "ReturnHeaders":

ns_startcontent -charset "iso-8859-1"

(ns_return works without this patch)

2.) In oder to use ns_startcontent you will have to install AOLServer 3.3.1ad13

3.) I compiled PG7.1.3 like this:

./configure --enable-locale --enable-recode --enable-multibyte --enable-unicode-conversion --with-tcl --with-perl --with-CXX --enable-syslog

4.) When you are using AOLserver 3.1.1ad13 you don't have to put the line "encoding system iso8859-1" at the end of the nsd.tcl file, because there is an entry in home/aolserver/modules/tcl/init.tcl already:

encoding system [ns_config ns/server/[ns_info server] SystemEncoding iso8859-1]

It might have been set to utf-8 originally... I don't really remember... so just check init.tcl

5.) I created my database with:

createdb --encoding=unicode yourdb

---------------------------------------------------------------------

What works now:

1.) Everything is being displayed correctly in a browser (.adp / .tcl / .txt / .html) !!!

What doesn't seem to work properly:

1.) When I create a database with:

createdb --encoding=LATIN1 yourdb

it looks like everything is beeing imported correctly, but when I do a "select * from users;" 0 rows are being returned???!!!

The db is imported correctly when I created the db with:

createdb --encoding=unicode yourdb

2.) When I udpate users.first_names with a German character like "ä" the result is being displayed correctly in the browser. But when I check the database from a shell via "psql mydb" the just added German character is being displayed weirdly... The shell displays the German characters that got imported from my old database correctly though!!!

3.) The same thing happens when I update a file from the file-storage via file-manager... When I edit the file from a browser all the special characters are being displayed correctly. But when I open the file from the shell using "vim" the special characters are being displayed weirdly!!!

4.) Community members' pictuers are not displayed by my browser anymore.. (They are stored in the database and not in the file-storage)

set db [ns_db gethandle]

set file_type [database_to_tcl_string_or_null $db "select portrait_file_type
from users
where user_id = $user_id"]

if [empty_string_p $file_type] {
    ad_return_error "Couldn't find portrait" "Couldn't find a portrait for User $user_id"
    return
}

ReturnHeaders $file_type


ns_pg blob_write $db $portrait_id

[05/Mar/2002:12:46:39]
    Error: can't read "portrait_id": no such variable
    can't read "portrait_id": no such variable
        while executing
    "ns_pg blob_write $db $portrait_id
    "
        (file "/web/server1/www/shared/portrait-bits.tcl" line 30)
        invoked from within
    "source $script"
        (procedure "ns_sourceproc" line 6)
        invoked from within
    "ns_sourceproc cns51 {}"
    Notice: Querying '
            select user_id, token, secure_token,
                   last_ip, last_hit from sec_sessions
            where session_id = 33244;'

I just created a db with "createdb -E latin1" and imported my old (sql-ascii) database...

As I had my /etc/profile set to PGCLIENTENCODING='unicode' I first had to do a:

PGCLIENTENCODING='latin1' ; export PGCLIENTENCODING

before I could import the database. Otherwise errormsg occurred...

When I enter the database via shell psql and don't want to see corrupted special characters I also have to set:

PGCLIENTENCODING='latin1' ; export PGCLIENTENCODING

Does this sound correct and familiar???

./configure --enable-locale --enable-multibyte

createdb --encoding=LATIN1 somedbname

PGCLIENTENCODING=UNICODE