Forum OpenACS Q&A: Support for multibyte: .tcl ok .adp not

I followed the thread:
https://openacs.org/bboard/q-an d-a-fetch-msg.tcl?msg_id=0001CE&topic_id=12&topic=OpenACS%204%2e0%20De sign and applied the patches to Openacs 3.2.4(download from cvs).

The handling of multibyte (Big5, a Tradictional Chinese encoding) works fine in form post/get and all .tcl pages until I tested against ecommerce module which use .adp templates. The templated .adp page did not display the encoding correctly.

Then I traced down to the code in ad_return_template which is in ad-style.tcl. In ad_return_template proc, it calls ns_adp_parse which is defined in adp.c in aolserver source. It looks like ns_adp_parse might be the cause of the problem. Does that mean one should avoid using ad_return_template when one needs an encording other than ascii ? any pointers would be appreciated.

Kenny Chiang

Collapse
Posted by Gaizka Villate on
You can use the templates by changing just one line in ad_return_template:

This line:

eval "uplevel { ns_write "$http_header[ns_adp_parse -file" $fully_qualified_template_filename"]" }"
for this couple of lines:
 set content_type "text/html"
 eval "uplevel { ns_write "$http_header[ns_startcontent -type $content_type][ns_adp_parse -file "$fully_qualified_template_filename"]" }"
It works for latin1 characters, i guess it will be the same for Big5.

Now, could you please ask me a question?
I asked in his thread (https://openacs.org/forums/message-view?message_id=18629 ) if anybody could post his configuration of a working system. Could you please tell me how have you compiled POstgreSQL, if your tables are created as Unicode, and what encoding system are you using within AOLserver?

I'd really apreciate it. If you read my question in the thread above, i've got problems with upper/lower chars.

I don't even know if Big5 has anything similar to upper/lower chars, but if it does, are they working for you?

Anyway, could you post your configuration schema?

Thanks a lot.

Collapse
Posted by Henry Minsky on
You cannot use the adp parser directly on files - it will assume they are in UTF8 encoding.

I have some patches which instead load the adp file into a Tcl string, using the encoding conversion of your choice (as specified by the mime type that you assign to .adp files in your server config file). It then calls ns_adp_parse with the -string option and passes that string. The patch also sets the appropriate output encoding so that AOLserver converts the file from UTF8 to the correct encoding when it returns it to the browser.

My patches are at http://www.ai.mit.edu/people/hqm/openacs

My .tcl config file is shown below. We are running Japanese OpenACS sites with OpenACS3.2.5 and shiftJIS encoding. Note that it is key to turn off the default mapping of .adp files to the adp parser, that I think the default OpenACS config file has. Instead, we use the big registered proc that passes everything under /* to ad_handle_abstract_url

The db's are created using the "createdb --encoding=unicode" flag



#
# AOLserver/OpenNSD Sample Configuration File
#
# Improved with lots of comments and things you'll need for
# running OpenACS.
#
# by Roberto Mello (rmello@fslc.usu.edu)
# $Id: nsd.txt,v 1.6 2001/02/15 06:29:43 rmello Exp $

ns_log notice "nsd.tcl: starting to read config file..."

# 
# If httpport is set to 80, you'll have to start AOLserver as root and pass the user 
# AOLserver will run as in the command line. (e.g. ./nsd -u nsadmin -g nsadmin -t ../nsd.tcl)
# (assuming you're starting AOLserver from the {aolserverdir}/bin directory.

set httpport               80 
set httpsport             443 

#
# Make sure your /etc/hostname is setup right, with your full domain. If you want AOLserver 
# to listen to www.foobar.com instead of foobar.com then hard code the appropriate domain  
# in the line below (e.g. set hostname www.foobar.com).
#
set hostname              [ns_info hostname]  
set address               [ns_info address] 

# 
# You can name your server whatever you want, but you'll need a directory with that
# name under {aolserverdir}/servers/ writable by you AOLserver user.
#
# For example: if your AOLserver is in /usr/local/aolserver and your server is "server1"
# you will need a /usr/local/aolserver/servers/server1 directory.
#
set server		   "openacs325" 
set servername             "openacs325" 

# 
# AOLserver's home and binary directories. Autoconfigurable. 
#
set homedir                 [file dirname [ns_info config]] 
set bindir                  [file dirname [ns_info nsd]] 

#
# Where are your pages going to live ?
#
set pageroot                /web/${server}/www 
set directoryfile           index.adp,index.tcl,index.html,index.htm

# 
# nsssl: Only loads if keyfile.pem and certfile.pem exist.
# If you are using SSL, make sure you have these dirs and files (refer
# to the AOLserver docs)

set sslkeyfile ${homedir}/servers/${server}/modules/nsssl/keyfile.pem

set sslcertfile ${homedir}/servers/${server}/modules/nsssl/certfile.pem 

# 
# Global server parameters 
#

ns_section "ns/parameters" 
        ns_param   home         $homedir 
        ns_param   debug        false 
        ns_param   mailhost     localhost 
        ns_param   ServerLog    ${homedir}/log/${server}-error.log 
#
        ns_param   LogRoll      on
	ns_param   HackContentType 1
	ns_param   URLCharset      shift_jis
	ns_param   OutputCharset   shift_jis
	ns_param   HttpOpenCharset shift_jis

# 
# Thread library (nsthread) parameters 
# 
ns_section "ns/threads" 
        ns_param   mutexmeter      true      ;# measure lock contention 
        #ns_param   stacksize [expr 128*1024] ;# Per-thread stack size for hungry C modules

# 
# MIME types. 
# 
#  Note: AOLserver already has an exhaustive list of MIME types, but in
#  case something is missing you can add it here. 
#

ns_section "ns/mimetypes" 
        ns_param   default         "*/*"     ;# MIME type for unknown extension 
        ns_param   noextension     "*/*"     ;# MIME type for missing extension 
        #ns_param   ".xls"         "application/vnd.ms-excel''
        ns_param .html "text/html; charset=shift_jis"
        ns_param .tcl "text/html; charset=shift_jis"
        ns_param .adp "text/html; charset=shift_jis"

# 
# Tcl Configuration 
# 
ns_section "ns/server/${server}/tcl" 
        ns_param autoclose 	"on" 
        ns_param debug 		"false" 
 

# This is where this server's private Tcl library is. All .tcl files in this directory 
# are parsed when AOLserver starts. The crucial procedures used in OpenACS are
# defined through this library
#
        ns_param library "/web/${server}/tcl" 

############################################################ 
# 
# Server-level configuration 
# 
#  There is only one server in AOLserver, but this is helpful when multiple
#  servers share the same configuration file.  This file assumes that only
#  one server is in use so it is set at the top in the "server" Tcl variable
#  Other host-specific values are set up above as Tcl variables, too.
# 
ns_section "ns/servers" 
        ns_param   $server     $servername 

# 
# Server parameters 
# 
ns_section "ns/server/${server}" 
        ns_param   directoryfile        $directoryfile 
        ns_param   pageroot             $pageroot 
        ns_param   globalstats          false     ;# Enable built-in statistics 
        ns_param   urlstats             false     ;# Enable URL statistics 
        ns_param   maxurlstats          1000      ;# Max number of URL's to do stats on
        ns_param   enabletclpages       true      ;# Parse tcl files in pageroot (dangerous)

#
# If you want to customize AOLserver's response to things like ``Internal Server Error''
# and other HTTP responses. OpenACS comes with some in the directory www/global.

        ns_param   NotFoundResponse   		"/global/file-not-found.html"
        ns_param   ServerBusyResponse	  	"/global/busy.html"
        ns_param   ServerInternalErrorResponse 	"/global/error.html"
        ns_param   ForbiddenResponse   		"/global/forbidden.html"
        ns_param   UnauthorizedResponse  	"/global/unauthorized.html"

# Directory listings -- use an ADP or a Tcl proc to generate them.
#
        #ns_param   directoryadp    $pageroot/dirlist.adp ;# Choose one or the other
        ns_param   directoryproc    _ns_dirlist           ;#  ...but not both! 
        ns_param   directorylisting  fancy                ;# Can be simple or fancy 
   
# for backport of ad_proc -- hqm
        ns_param acs_root_dir "/web/${server}"


# 
# ADP (AOLserver Dynamic Page) configuration 
# 
#ns_section "ns/server/${server}/adp" 
#        ns_param   map           "/*.adp"  ;# Extensions to parse as ADP's 
        #ns_param   map          "/*.html" ;# Any extension can be mapped 
#        ns_param   enableexpire  false     ;# Set "Expires: now" on all ADP's 
#        ns_param   enabledebug   false     ;# Allow Tclpro debugging with "?debug"

 

# ADP special pages 
        #ns_param   errorpage      ${pageroot}/errorpage.adp ;# Pretty-print ADP scripting errors

# 
# ADP custom parsers -- see adp.c 
# 
ns_section "ns/server/${server}/adp/parsers" 
        ns_param   fancy            ".adp"

# 
# Socket driver module (HTTP)  -- nssock 
# 
ns_section "ns/server/${server}/module/nssock" 
        ns_param   port         $httpport 
        ns_param   hostname     $hostname 
        ns_param   address      $address

# 
# Socket driver module (HTTPS) -- nsssl 
# 
#  nsssl does not load unless sslkeyfile/sslcertfile exist (above).
# 
ns_section "ns/server/${server}/module/nsssl" 
        ns_param   port        $httpsport 
        ns_param   hostname    $hostname 
        ns_param   address     $address 
        ns_param   keyfile     $sslkeyfile 
        ns_param   certfile    $sslcertfile

# 
# Database drivers 
# The database driver is specified here. PostgreSQL driver being loaded.
# Make sure you have the driver compiled and put it in {aolserverdir}/bin
#
ns_section "ns/db/drivers" 
    ns_param   postgres     ${bindir}/postgres.so  ;# Load PostgreSQL driver

# 
# Database Pools: This is how AOLserver  ``talks'' to the RDBMS. You need three for
# OpenACS: main, log, subquery. Make sure to replace ``yourdb'' and ``yourpassword''
# with the actual values for your db name and the password for it.
# AOLserver can have different pools connecting to different databases and even different
# different database servers.
# 
ns_section "ns/db/pools" 
    ns_param   main       "OpenACS Main Pool" 
    ns_param   log        "OpenACS Log Pool" 
    ns_param   subquery   "OpenACS Subquery Pool"

ns_section "ns/db/pool/main" 
    ns_param Driver postgres 
    ns_param Connections 5                  ;# 5 is a good number. Increase according to your needs
    ns_param DataSource localhost::openacs325   ;# Replace 'yourdb' with the name of your database in PG
    ns_param User nsadmin                   ;# User and password AOLserver will use to connect
    ns_param Password "nsadmin123" 
    ns_param Verbose Off                    ;# Set it to On to see all queries. Good for debugging SQL.
    ns_param LogSQLErrors On 
    ns_param ExtendedTableInfo On 
   # ns_param MaxOpen 1000000000            ;# Max time to keep idle db connection open
   # ns_param MaxIdle 1000000000            ;# Max time to keep active db connection open 

ns_section "ns/db/pool/log" 
    ns_param Driver postgres 
    ns_param Connections 5 
    ns_param DataSource localhost::openacs325
    ns_param User nsadmin 
    ns_param Password "nsadmin123" 
    ns_param Verbose On 
    ns_param LogSQLErrors On 
    ns_param ExtendedTableInfo On 
   # ns_param MaxOpen 1000000000 
   # ns_param MaxIdle 1000000000 

ns_section "ns/db/pool/subquery" 
    ns_param Driver postgres 
    ns_param Connections 2 
    ns_param DataSource localhost::openacs325
    ns_param User nsadmin 
    ns_param Password "nsadmin123" 
    ns_param Verbose On 
    ns_param LogSQLErrors On 
    ns_param ExtendedTableInfo On
   # ns_param MaxOpen 1000000000 
   # ns_param MaxIdle 1000000000 

ns_section "ns/server/${server}/db" 
        ns_param Pools          "*" 
        ns_param DefaultPool    "main"

#
# nscp: AOLserver Control Port - very useful for testing and evaluating.
# Uncomment the sample password below and do a "telnet localhost 9999"
# log in with "nsadmin", password "x", type "ns_crypt newpassword salt"
# and paste the new encrypted string below.
#
# Sample User="nsadmin", password="x" 
 set nscp_user              "nsadmin:t2GqvvaiIUbF2:"

# 
# Control port -- nscp 
#  nscp does not load unless nscp_user is a valid user. 
# 
ns_section "ns/server/${server}/module/nscp" 
        ns_param   port            9999 
        ns_param   address         "127.0.0.1" ;# LOCALHOST IS RECOMMENDED      

ns_section "ns/server/${server}/module/nscp/users" 
        ns_param   user            $nscp_user

# 
# Access log -- nslog 
# 
ns_section "ns/server/${server}/module/nslog" 
        ns_param   rolllog         true      ;# Should we roll log? 
        ns_param   rollonsignal    true      ;# Roll log on SIGHUP 
        ns_param   rollhour        0         ;# Time to roll log 
        ns_param   maxbackup       5         ;# Max number to keep around when rolling 
#
        ns_param file ${homedir}/log/${server}.log 
        ns_param EnableHostnameLookup Off
        ns_param LogCombined On
        ns_param LogRefer Off
        ns_param LogUserAgent Off
#
# nsjava - aolserver module that embeds a java virtual machine.  Needed to 
#          support webmail.  See http://nsjava.sourceforge.net for further 
#          details.
#

ns_section "ns/server/acs-pg/module/nsjava"
        ns_param   EnableJava          "off"  ;# Set to on to enable nsjava.
        ns_param   VerboseJvm          "off"  ;# Same as command line -debug.
        ns_param   LogLevel            "Notice"
        ns_param   DestroyJvm          "off"  ;# Destroy jvm on shutdown.
        ns_param   DisableJITCompiler  "off"  
        ns_param   ClassPath "/usr/local/jdk/jdk118_v1/lib/classes.zip:/usr/local/aolserver/bin/nsjava.jar:/home/nsadmin/mirror/openacs325/www/webmail/java/activation.jar:/home/nsadmin/mirror/openacs325/www/webmail/java/mail.jar:/home/nsadmin/mirror/openacs325/www/webmail/java" 

# 
# CGI interface -- nscgi, if you have legacy stuff. Tcl or ADP files inside 
# AOLserver are vastly superior to CGIs. You don't actually need the Interps 
# if your script calls the appropriate interpreter itself.
# 
ns_section "ns/server/${server}/module/nscgi" 
       ns_param   map "GET  /cgi-bin /web/$server/cgi-bin"
       ns_param   map "POST /cgi-bin /web/$server/cgi-bin" 
       ns_param   Interps CGIinterps

ns_section "ns/interps/CGIinterps" 
       ns_param .pl "/usr/bin/perl"
       ns_param .doit "/bin/bash"

# 
# Modules to load 
# 
ns_section "ns/server/${server}/modules" 
        ns_param   nssock          ${bindir}/nssock.so 
        ns_param   nslog           ${bindir}/nslog.so 
        ns_param   nsxml           ${bindir}/nsxml.so

#       ns_param   nsperm          ${bindir}/nsperm.so 
        ns_param   nscgi           ${bindir}/nscgi.so 
#       ns_param   nsjava          ${bindir}/libnsjava.so
#        ns_param   ats              Tcl

#
## nsssl: loads only if requisite files already exist (see top of this
# file). 
#
if { [file exists $sslcertfile] && [file exists $sslkeyfile] } { 
    ns_param nsssl ${bindir}/nsssle.so 
} else { 
    ns_log warning "nsd.tcl: nsssl not loaded because key/cert files do not exist."
}

# nscp: loads only if nscp_user is set (see top of this file).
if { $nscp_user != "" } {
#    ns_param nscp ${bindir}/nscp.so
} else {
    ns_log warning "nsd.tcl: nscp not loaded because user/password is not set."
}

#ns_section	"ns/server/${server}/ats"
#ns_param	ResourcePath		${homedir}/modules/tcl/ats/resources


# 
# To Source OpenACS Config File. You need a "yourservername.tcl" file in the
# OpenACS parameters dir. e.g.: say your server is "server1" then you'd need a
# file named /web/server1/parameters/server1.tcl
# (Hint: Rename ad.tcl to yourservername.tcl and edit it.)
#

source /web/${server}/parameters/${server}.tcl

ns_log notice "nsd.tcl: finished reading config file."

ns_log notice "HOMEDIR = $homedir"



Collapse
Posted by Kenny Chiang on
Gaizka,

I compiled postgres as follows:

./configure 
  --enable-locale 
  --enable-recode 
  --enable-multibyte 
  --enable-unicode-conversion 
  --with-maxbackends=64
  --with-tcl
  --with-tclconfig=/usr/local/lib
  --with-perl 
  --with-python
  --with-openssl
  --with-CXX 
  --enable-syslog

and created db's in UNICODE. My configuration file is same as the one posted by Henry except I changed the shift_jis to big5.

There is no upper/lower case in big5 (in fact, in Chinese). So I can not be helpful to test out your problem. sorry about that.

I did try patch the ad_return_template as you suggested but it still output juck characters as before. I think it is caused by the "ns_adp_parse -file" which does not honor the charset conversion as Henry suggested.

Henry,

I removed the default adp mapping to adp parser as you suggested and the "pure" .adp's work fine with this change. But the problem remains in the ad_return_template which is in use in serveral modules in openacs, such as ecommerce,gp and sdm etc. Do you have any suggestions on what to change to make use on the templated .adp pages called by ad_return_template ?

Collapse
Posted by Kenny Chiang on
Henry,

Contiune on testing the openacs modules, I found ad_return_top_page proc is not patched to handle charset conversion in your patches. ad_return_top_page is not used in a lot of modules, but for completness you may want to include in your patch set. The patch I did is as follows:

proc_doc ad_return_top_of_page {first_part_of_page{content_type "text/html; charset=big5"}} "Returns HTTP headers plus the top of the user-ivisible page.  Saves a TCP packet (and therefore some overhead) compared to using ReturnHeaders and an ns_write." {
    ns_startcontent -type $content_type
    set all_the_headers "HTTP/1.0 200 OK
MIME-Version: 1.0
Content-Type: $content_type
"
     util_WriteWithExtraOutputHeaders $all_the_headers $first_part_of_page
}

Collapse
Posted by Henry Minsky on
I added a patch to ad-style.tcl to support encoding conversion by ad_return_template. It is at

http://www.ai.mit.edu/people/hqm/openacs/ad-style.tcl.patch

Collapse
Posted by Roberto Mello on
Henry, I think Hafeez's patched version of your patch has the ad-style patch :)

Would you please coordinate this with him and come up with one tarball that we can keep on the contrib section of openacs.org? I will give you permissions to write over it. I made a readme for Hafeez's patch on the tarball that's in the contrib code right now. It explains what the files are and how to install them. You could included that in the tarball too.

My tarball is at the /new-file-storage on openacs.org.

Thanks.

Collapse
Posted by Kenny Chiang on
Henry, I appled ad-style.tcl patch and pointed my url to /ecommerce and came up with the error like:
[02/May/2001:12:09:52][1649.4101][-conn0-] Error: can't read "user_is_logged_on"
: no such variable
can't read "user_is_logged_on": no such variable
    while executing
"if { $user_is_logged_on } {
    set welcome_message "Welcome back $user_name!  If you're not $user_name, click <a href="$register_url">here</a>."
} ..."
    invoked from within chunk: 1 of adp: ns_adp_parse
[02/May/2001:12:09:52][1649.4101][-conn0-] Error: can't read "welcome_message": no such variable
can't read "welcome_message": no such variable
    while executing
"ns_adp_puts -nonewline  $welcome_message "
    invoked from within chunk: 2 of adp: ns_adp_parse
[02/May/2001:12:09:52][1649.4101][-conn0-] Error: can't read "search_widget": no such variable
can't read "search_widget": no such variable
    while executing
"ns_adp_puts -nonewline  $search_widget "
 invoked from within chunk: 3 of adp: ns_adp_parse
[02/May/2001:12:09:52][1649.4101][-conn0-] Error: can't read "recommendations_if_there_are_any": no such variable
Shouldn't the ns_adp_parse be eval in "uplevel 1" ?

I changed the code as follows but had some difficulty to get the quote working. can someone point out what is wrong in the code:

set mimetype [ns_guesstype $fully_qualified_template_filename]
set encoding [ns_encodingfortype $mimetype]

set fd [open $fully_qualified_template_filename r]
fconfigure $fd -encoding $encoding
set template [read $fd]
close $fd

if { $cache_p } {
  # build the http header, with a no cache pragma if cache_p is 0
    set http_header "HTTP/1.0 200 OK
MIME-Version: 1.0
Content-Type: $mimetype
Pragma: No-Cache

"
    eval "uplevel { ns_write "$http_header[ns_adp_parse -string "$template"]" }"
} else {
    eval "uplevel { ns_return 200 $mimetype [ns_adp_parse -string "$template"] }"
}

The error message of above code look like this:

[02/May/2001:13:52:30][1984.8201][-conn4-] Error: extra characters after close-quote
extra characters after close-quote
    while executing
"ns_write "HTTP/1.0 200 OK
MIME-Version: 1.0
Content-Type: text/html; charset=big5
Pragma: No-Cache

[ns_adp_parse -string "<ec_header>Welcome to [ec_s..."
    ("uplevel" body line 1)
    invoked from within
"uplevel { ns_write "HTTP/1.0 200 OK
MIME-Version: 1.0
Content-Type: text/html; charset=big5
Pragma: No-Cache

[ns_adp_parse -string "<ec_header>Welcom..."
    ("eval" body line 1)
    invoked from within
"eval "uplevel { ns_write "$http_header[ns_adp_parse -string "$template"]"
 }""
    (procedure "ad_return_template" line 68)

Collapse
Posted by Gaizka Villate on
First of all, thanks for your answers about the system encoding you are using.

Now, let's try to solve this problem:

Henry, you're right when you say that you can call ns_adp_parse on files directly, that you must convert them to string before.

The problem is that, i don't know why, AOLserver seems to parse ns_adp_parse -string with the plain parser, and ns_adp_parse -file with the fancy parser.

Also, you forgot the ns_startcontent.

And the ecommerce module requires the fancy parser (it uses its own tags).

So i've come up with this piece of code. It works, but it is quite ugly.

Anyway, here is the code, and let's see if somebody smarter than me can tell if it is ok.

 
    set mimetype [ns_guesstype $fully_qualified_template_filename]
    set encoding [ns_encodingfortype $mimetype]
 
    set fd [open $fully_qualified_template_filename r]
    fconfigure $fd -encoding $encoding
    set template [read $fd]
    close $fd
 
        set http_header "HTTP/1.0 200 OK
MIME-Version: 1.0
Content-Type: $mimetype
Pragma: No-Cache
 
"
 
    if { $cache_p } {
        eval "uplevel { ns_write "$http_header[ns_startcontent -type $mimetype][ns_adp_eval -parser fancy {$template}]" }"
    } else {
        eval "uplevel { ns_return 200 $mimetype [ns_adp_eval -parser fancy {$template}] }"
    }
 
 

Collapse
Posted by Kenny Chiang on
Gaizka,

<p>
Thanks for your code, it works for me too except I need to make a bit of modification as follows:
<p>
<pre>
    if { $cache_p } {
              eval "uplevel { ns_write "$http_header[ns_startcontent -type "$mimetype"][ns_adp_eval -parser fancy {$template}]" }"
          } else {
              eval "uplevel { ns_return 200 "$mimetype" [ns_adp_eval -parser fancy {$template}] }"
          }
</pre>
<p>
since my mimetype is "text/html; charset=big5" that needs to be enclosed in double quote.

<p>
Thanks again.

Collapse
Posted by Tilmann Singer on
Still can't get the handling of POSTed german umlauts right with
nsd8x. Kenny - you say post/get in Big5 encoding works fine, can you
confirm that the posted values are correctly stored in the db - do
they look as expected when queried with psql?

The error that I experience does _not_ appear in the browser: the
umlauts will be displayed correctly, but stored as two funny
characters in the db, according to the server log and to direct
querying with psql.

Here is a list of the steps that I took - please tell me what I am
missing.

Downloaded aolserver3_2+ad12_src.tar.gz

Downloaded aolserver-ad12-japanese-patches.tgz from
http://www.ai.mit.edu/people/hqm/openacs/ and followed the
instructions in the contained README file:

applied acs-aolserver-ad12.patch

copied 8bit.enc

make, make install

applied ad12-tcl.patch to the modules/tcl directory of the compiled
aolserver

Got openacs-3.2.5.tgz from openacs.org

created a db with createdb --encoding=UNICODE and loaded the openacs
datamodel

downloaded openacs-japanese-patches from
http://www.ai.mit.edu/people/hqm/openacs/ and replaced all occurencies
of shift_jis with iso-8859-1 as well as the line on the beginning
(which will go into tcl/00-ad-preload.tcl):

encoding system shiftjis

with

encoding system iso8859-1

applied that modified patch to the freshly unpacked OpenACS 3.2.5

Set those configuration parameters in my nsd.tcl:

ns_section "ns/parameters"
    ns_param  HackContentType 1
    ns_param  URLCharset      iso-8859-1
    ns_param  OutputCharset  iso-8859-1
    ns_param  HttpOpenCharset iso-8859-1

ns_section "ns/mimetypes"
        ns_param .html "text/html; charset=iso8859-1"
        ns_param .tcl "text/html; charset=iso8859-1"
        ns_param .adp "text/html; charset=iso8859-1"

Now when I start the aolserver and do a POST that contains german
umlauts (tested with /pvt/basic-info-update-2.tcl) the umlauts will be
converted into two funny characters. They do already show up in the
server log:

[07/May/2001:19:42:55][1541.9222][-conn2-] Notice: Querying 'update users
    set first_names = 'Günther',
    last_name = 'Schmidt',
[...]

(This is an uppercase A with tilde on it followed by a quarter sign in
first_names, in case this posting gets somehow converted)

before that there is this notice in the server log:

[07/May/2001:19:42:55][1541.9222][-conn2-] Notice: ns_getform using
encoding iso8859-1 for charset iso-8859-1

I also tried changing the line $AOLSERVER/modules/tcl/init.tcl

encoding system [ns_config ns/server/[ns_info server] SystemEncoding
utf-8]

to:

encoding system [ns_config ns/server/[ns_info server] SystemEncoding
iso-8859-1]

Don't know what effect that actually has but it was mentioned
somewhere on the forum.

Collapse
Posted by Kenny Chiang on
Tilmann,

I use the following shell script to start the aolserver:

#!/bin/sh
LD_LIBRARY_PATH=/usr/local/pgsql/lib:usr/local/lib      export LD_LIBRARY_PATH
PGCLIENTENCODING=UNICODE                                export PGCLIENTENCODING
LANG=zh_TW.Big5                                         export LANG
umask 022
exec /home/nsadmin/bin/nsd $*

and set the environment as below in my profile to run psql:

PATH=$PATH:/usr/local/pgsql/bin:~/bin         export PATH
LANG=zh_TW.Big5                               export LANG
PGCLIENTENCODING=BIG5                         export PGCLIENTENCODING

In this way you have nsd talks to the postgres in UNICODE and have psql speaks BIG5 (or in your case iso-8859-1). Since the data you POST thru aolserver is actually stored in postgres in UNICODE encoding and that is what you saw in the server log. OTH, if you want to query the db thru psql, then you have to set PGCLIENTENCODING to ISO8859_1 (I'm not sure if the spelling is correct, you have to check on postgres documents) to have the postgres client library to convert UNICODE to ISO8859-1, so you can display your charset correctly in your terminal.

-- Kenny Chiang

Collapse
Posted by Tilmann Singer on
Thanks for clarifying this. So it was just a configuration problem
with psql and had nothing to do with aolserver.

psql wants PGCLIENTENCODING set to LATIN1 to convert to iso8859-1

I have tried the suggestions in this thread, but am still seeing a problem.

The error I get concerns this:

<pre>
extra characters after close-quote
    while compiling
"eval "uplevel { ns_write "
    ("if" then script line 2)
    while compiling
"if { $cache_p } {
    eval "uplevel { ns_write "$http_header [ns_startcontent -type $mimetype] [ns_adp_eval -parser fancy {$template}]" }"
    } else {
      ..."
    ("if" else script line 26)
    while compiling
"if { [llength $available_templates] == 0 } {
    ad_return_error "No template available" "We can't find any template for presenting the output of this sc..."
    (compiling body of proc "ad_return_template", line 34)
    invoked from within
"ad_return_template
"
    ("uplevel" body line 154)
    invoked from within
"uplevel 1 $code"
    (procedure "source_with_encoding" line 11)
    invoked from within
"source_with_encoding $ad_conn(file)"

</pre>

Which I get regardless of whether I use Gaizka's suggested change, or Kenny Chiang's modification.  The target encoding is shift_jis.  The other modules that we "need" work, but they are not templated like ecommerce is.

Any help welcome.  I will be on #openacs on irc.openprojects.net for those of you with IRC access during this evening (Sept 27th, EST).

(repost due to not checking "HTML" in post form)I have tried the suggestions in this thread, but am still seeing a problem.

The error I get concerns this:

extra characters after close-quote
    while compiling
"eval "uplevel { ns_write "
    ("if" then script line 2)
    while compiling
"if { $cache_p } {
	eval "uplevel { ns_write "$http_header [ns_startcontent -type $mimetype] [ns_adp_eval -parser fancy {$template}]" }"
	} else {
	   ..."
    ("if" else script line 26)
    while compiling
"if { [llength $available_templates] == 0 } {
	ad_return_error "No template available" "We can't find any template for presenting the output of this sc..."
    (compiling body of proc "ad_return_template", line 34)
    invoked from within
"ad_return_template
"
    ("uplevel" body line 154)
    invoked from within
"uplevel 1 $code"
    (procedure "source_with_encoding" line 11)
    invoked from within
"source_with_encoding $ad_conn(file)"

Which I get regardless of whether I use Gaizka's suggested change, or Kenny Chiang's modification. The target encoding is shift_jis. The other modules that we "need" work, but they are not templated like ecommerce is.

Any help welcome. I will be on #openacs on irc.openprojects.net for those of you with IRC access during this evening (Sept 27th, EST).

Collapse
Posted by Henry Minsky on
I've beaten AOLserver and several ACS versions into handling Japanese charsets. I don't have a lot of time, but if you could post your
config file, and tell me what version of AOLserver and ACS (and what patches) you are using, I might be able to help out.

.adp files are generally parsed assuming a UTF8 encoding unless you use
my hack of reading the file into Tcl as a string first, with the encoding set specifically on the file descriptor. I have patches to
make ACS load .adp files in this way. You could also convert your
shift_jis .adp files into UTF8. I personally like to keep everything
in files in shift_jis though, because most tools like editors can
deal with it, whereas UTF8 is not widely supported by the backend tools. On the other hand, storing everything in UTF8 in the database
is the best way to go, hands down, in my opinion.

Hi Henry,

I am using 3.2.5 with your patches, 3.3.1+ad12 version of AOLserver.

I *think* the problem is in the part where the template is returned. I have modified ad-style.tcl per Kenny Chiang's message above. I have tried both ns_adp_parse -string and ns_adp_eval -parser fancy in the ad_return_template proc definition, both give the same result.

You can see the error message at http://205.247.253.66/ecommerce .

In my server.log, I see

[28/Sep/2001:14:28:18][23963.6798848][-conn44-] Notice: ns_getform using encoding shiftjis for charset shift_jis

followed by the error messages.

My nsd.tcl config file (passwords and comments removed for brevity):

ns_log notice "nsd.tcl: starting to read config file..."

set httpport               80
set httpsport             443 

set hostname            205.247.253.66 

set address              205.247.253.66 


set server                 "server1" 
set servername             "somewhere.ne.jp"


set homedir                 [file dirname [ns_info config]]
set bindir                  [file dirname [ns_info nsd]] 

set pageroot                /home/akira/openacs-3.2.5/www
set directoryfile           index.adp,index.html,index.tcl,index.htm

set sslkeyfile ${homedir}/servers/${server}/modules/nsssl/keyfile.pem

set sslcertfile ${homedir}/servers/${server}/modules/nsssl/certfile.pem 

ns_section "ns/parameters" 
        ns_param   home         $homedir 
        ns_param   debug        false 
        ns_param   MailHost     localhost
        ns_param   ServerLog    ${homedir}/log/server.log 
        ns_param   LogRoll      on
        ns_param   maxthreads 20
        #ns_param  minthreads 7 

        ns_param   HackContentType 1
        ns_param   URLCharset      shift_jis
        ns_param   OutputCharset   shift_jis
        ns_param   HttpOpenCharset shift_jis


ns_section "ns/threads" 
ns_param   mutexmeter      true      ;# measure lock contention 
ns_param   stacksize [expr 128*1024] ;# Per-thread stack size for hungry C modules

ns_section "ns/mimetypes" 
        ns_param   default         "*/*"     ;# MIME type for unknown extension 
        ns_param   noextension     "*/*"     ;# MIME type for missing extension 
        #ns_param   ".xls"         "application/vnd.ms-excel''
        ns_param .html "text/html; charset=shift_jis"
        ns_param .tcl "text/html; charset=shift_jis"
        ns_param .adp "text/html; charset=shift_jis"

ns_section "ns/server/${server}/tcl" 
        ns_param autoclose      "on" 
        ns_param debug          "false" 
 

ns_param library "/home/akira/openacs-3.2.5/tcl"

ns_section "ns/servers" 
        ns_param   $server     $servername 

ns_section "ns/server/${server}" 
        ns_param   directoryfile        $directoryfile 
        ns_param   pageroot             $pageroot 
        ns_param   globalstats          true     ;# Enable built-in statistics 
        ns_param   urlstats             true     ;# Enable URL statistics 
        ns_param   maxurlstats          1000      ;# Max number of URL's to do stats on
        ns_param   enabletclpages       true      ;# Parse tcl files in pageroot (dangerous)

        ns_param   NotFoundResponse             "/global/file-not-found.html"
        ns_param   ServerBusyResponse           "/global/busy.html"
        ns_param   ServerInternalErrorResponse  "/global/error.html"
        ns_param   ForbiddenResponse            "/global/forbidden.html"
        ns_param   UnauthorizedResponse         "/global/unauthorized.html"


        ns_param   directoryproc    _ns_dirlist           ;#  ...but not both! 
        ns_param   directorylisting  fancy                ;# Can be simple or fancy 
# 
# ADP custom parsers -- see adp.c 
# 
ns_section "ns/server/${server}/adp/parsers" 
        ns_param   fancy            ".adp"

ns_section "ns/server/${server}/module/nssock" 
        ns_param   port         $httpport 
        ns_param   hostname     $hostname 
        ns_param   address      $address


ns_section "ns/db/drivers" 

ns_param   postgres     /usr/local/aolserver/bin/postgres.so  ; # Load PostgreSQL driver

# 
# Database Pools: This is how AOLserver  ``talks'' to the RDBMS. You need three for
# OpenACS: main, log, subquery. Make sure to replace ``yourdb'' and ``yourpassword''
# with the actual values for your db name and the password for it.
# AOLserver can have different pools connecting to different databases and even different
# different database servers.
# 
ns_section "ns/db/pools" 
    ns_param   main       "OpenACS Main Pool" 
    ns_param   log        "OpenACS Log Pool" 
    ns_param   subquery   "OpenACS Subquery Pool"

ns_section "ns/db/pool/main" 
    ns_param Driver postgres 
    ns_param Connections 5                  ;# 5 is a good number. Increase according to your needs
    ns_param DataSource localhost:5432:db
ns_param User username                   ;# User and password AOLserver will use to connect
    ns_param Password "password" 
    ns_param Verbose On                    ;# Set it to On to see all queries. Good for debugging SQL.
    ns_param LogSQLErrors On 
    ns_param ExtendedTableInfo On 
ns_param MaxOpen 1000000000            ;# Uncommenting these two cause AOLserver to keep the
ns_param MaxIdle 1000000000            ;# db connection open. Can be a good thing, up tp your needs.

ns_section "ns/db/pool/log" 
    ns_param Driver postgres 
    ns_param Connections 5 
    ns_param DataSource localhost::db
    ns_param User username 
    ns_param Password "password" 
    ns_param Verbose Off 
    ns_param LogSQLErrors On 
    ns_param ExtendedTableInfo On 
ns_param MaxOpen 1000000000 
 ns_param MaxIdle 1000000000 

ns_section "ns/db/pool/subquery" 
    ns_param Driver postgres 
    ns_param Connections 4 
    ns_param DataSource localhost::db
    ns_param User username 
    ns_param Password "password" 
    ns_param Verbose Off 
    ns_param LogSQLErrors On 
    ns_param ExtendedTableInfo On
 ns_param MaxOpen 1000000000 
 ns_param MaxIdle 1000000000 

ns_section "ns/server/${server}/db" 
        ns_param Pools          "*" 
        ns_param DefaultPool    "main"


ns_section "ns/server/${server}/module/nslog" 
        ns_param   rolllog         true      ;# Should we roll log? 
        ns_param   rollonsignal    true      ;# Roll log on SIGHUP 
        ns_param   rollhour        0         ;# Time to roll log 
        ns_param   maxbackup       360         ;# Max number to keep around when rolling 

ns_section "ns/server/${server}/modules" 
        ns_param   nssock          ${bindir}/nssock.so 
        ns_param   nslog            ${bindir}/nslog.so 
        ns_param   nsperm          ${bindir}/nsperm.so 
#        ns_param   nscgi           ${bindir}/nscgi.so 
#        ns_param   nsjava          ${bindir}/libnsjava.so

#
## nsssl: loads only if requisite files already exist (see top of this
# file). 
#
if { [file exists $sslcertfile] && [file exists $sslkeyfile] } { 
    ns_param nsssl ${bindir}/nsssle.so 
} else { 
    ns_log warning "nsd.tcl: nsssl not loaded because key/cert files do not exist."
}

# 
# To Source OpenACS Config File. You need a "yourservername.tcl" file in the
# OpenACS parameters dir. e.g.: say your server is "server1" then you'd need a
# file named /web/server1/parameters/server1.tcl
# (Hint: Rename ad.tcl to yourservername.tcl and edit it.)
#

source /home/user/openacs-3.2.5/parameters/acs.tcl

ns_log notice "nsd.tcl: finished reading config file."