Forum OpenACS Development: cr_write_content and utf-8

Salut,

fconfigure $fd -translation binary

Well, I cannot comment to Malte's intentions, I guess this comment is not directly related to the fconfigure line. it is rather a by-product.

the problem is that, while binary translation as such is fine (anything else would be platform-bound with respect to eofs), it has itself a side-effect:

[...] and sets the encoding to binary (which disables encoding filtering) [...]

(quoting fconfigure's man http://docs.activestate.com/activetcl/8.4/tcl/TclCmd/fconfigure.htm#M11)

setting "-translation binary" implies (if not set explicitly) "-encoding binary", which is certainly not intended and corrupts character sets that need to be interpreted with a certain binary format (e.g. utf8).

so setting encoding explicitly to the system's default should suffice (to cover non-utf8 environments):

"fconfigure $fd -translation binary -encoding [encoding system]"

//stefan

Ok, thanks for reporting back ...

Can this be commited to CVS?

Yep, now that I know it works I will commit it to head, whether this (from my point of view major) fix goes into a stable branch shall be decided by oct.

I will also check for other occurrences of the issue ...

//stefan

Hi

I'm having a related problem but I don't know if it's my setup or an issue with OpenACS. I have CR items stored on the file system and when I use cr_write_content -string to write them to a different location on the file system, they appear to be getting corrupted. Here is my proc below. Note that I have played around with my own fconfigure line below, adding the -encoding flag too, but the file is still getting corrupted. Interestingly, I can get it to work by hacking cr_write_content to just use "-translation binary" (without the -encoding part).

To test this, upload a Word document or pdf to the content repository, then call the proc below in the developer shell (with the item_id of your uploaded file and the location you wish to put the new file), and then try to open the created file.

I'd be grateful if someone could try this and let me know if it's an OpenACS issue or something in my setup.

ad_proc -public pdf::write_file_to_file_system {
{-item_id:required}
{-filename:required}
} {
} {

#Initialise
set return_val 1

if { [catch {
set file_data [cr_write_content -string -item_id $item_id ]
set file_handle [open $filename "w"]
fconfigure $file_handle -translation binary
puts -nonewline $file_handle $file_data
close $file_handle
} errmsg] } {
ns_log Error "pdf::write_file_to_file_system item_id=$item_id filename=$filename : $errmsg"
set return_val 0
}

return $return_val
}

Correction:

Where it says 'When content_type is "file"', it should read 'When storage_type is "file"'.

Emmanuelle,

are you sure, you were running on your server system tclsh and aolserver with the same environment variables and linked against the same tcl shared libs? I see with both, Mac OS X and lenny/sid with aolserver 4.5.1 and 4.0 in tclsh and ds/shell always utf-8 for [encoding system].

background: During initialization, Tcl determines the default system encoding from the LC_* or LANG environment variables. If nothing can be found, it uses TCL_DEFAULT_ENCODING, which is set depending on the OS. For example, under Mac OS X the TCL_DEFAULT_ENCODING is utf-8. If configure can't determine anything, the final default system encoding is "iso8859-1". Later, Tcl's system encoding can be altered on the scripting layer via "encoding system ?XXX?" or from C via Tcl_SetSystemEncoding(). Aolserver 4.0.10/4.5.1 does not set it via Tcl or C, naviserver has a config variable named "systemencoding" and sets the encoding in init.tcl (if nothing specified, it defaults to utf-8).

note, that when you load a library file or a www/*tcl script that sets the encoding via "encoding system ...", it is set for the whole server (all threads). The system encoding is a global variable in the Tcl implementation. The only OpenACS package that sets the system encoding is lors-central (most likely, not a good idea).

It is a good idea to check the LANG variable in your startup script for aolserver and use in doubt something like LANG=en_US.UTF-8

Hope this helps and all the best
-gustaf neumann

Do you say that LANG of nsd is set to en_US.UTF-8 and the result of [encoding system] is "iso8859-1"?

Just in case: there is a difference between logging in as a user and running a command as a user. If you execute the command

set ::env(LANG)

in ds/shell, do you get "en_US.UTF-8" as result? What Tcl version are you using on the server in question?

Thanks Emmanuelle for trying the test case. Yes I think I'll probably use file copy for the cases where we store files on the file system (I haven't tested yet if the code works on systems where we store files in the database) but I thought I'd see if anyone here has a fix for the problem.

For completeness, here are the settings on my system:
set ::env(LANG) in ds/shell returns en_IE.UTF-8
encoding system in ds/shell returns utf-8
encoding system in tclsh returns utf-8
locale on the OS returns LANG=en_IE.UTF-8
echo $NLS_LANG on the OS returns .UTF8

thanks again
Brian

Thanks for the reply. I can't use file copy as the files may be stored in the database, so I have to code in a way that covers both cases.