Forum OpenACS Development: Re: Charset difference with NaviServer and OpenACS 5.9:

Collapse
Posted by Gustaf Neumann on
actually, i think i know what the problem is:
  • po uses its own function "im_report_write_http_headers" to output headers, rather than the proper OpenACS functions that configure the streams.
  • I would not be surprised, if the problem "goes away" when you use the OpenACS API: ad_return_top_of_page [1] rather than the hand-made version.
  • if you need more control, you might consider
    ns_headers 200 text/plain
    ns_write "Results: \n"
    foreach i {1 2 3} {
        ns_write "$i...'äüö' 'ÄÜÖ'<br>\n"
        ns_sleep 2
    }
    ns_write "Done."
    
  • for details on ns_headers, see [2]
all the best
-g

[1] http://openacs.org/api-doc/proc-view?proc=ad_return_top_of_page&source_p=1
[2] http://naviserver.sourceforge.net/n/naviserver/files/ns_write.html

Collapse
Posted by Neophytos Demetriou on
Not so sure it's a headers issue as the Content-Type header is correct for the given link/page. I would first check to make sure the data stored in the db is correct.

That said, ad_return_top_of_page uses ReturnHeaders that, in turn, uses ns_startcontent, which sets the connection outputEncoding (see Ns_ConnSetEncoding in conn.c). The default connection outputEncoding is suppose to be utf-8 anyway.

FWIW, the outputEncoding is utilized in Ns_ConnWriteVChars (in connio.c) to convert the response bytes using Tcl_UtfToExternalDString (if outputEncoding is not set to utf-8) i.e. converting each byte of a utf-8 character before it is sent to the browser.

Collapse
Posted by Gustaf Neumann on
neophytos,

the function used on this page (im_report_write_http_headers) is NOT using ReturnHeaders.

-g

Collapse
Posted by Neophytos Demetriou on
Thank you Gustaf, we are in agreement. My reply was meant to put more emphasis on the function of ns_startcontent (as opposed to the response headers) as it sets the connection outputEncoding and if that is not set to utf-8 (the default for outputCharset parameter) naviserver converts the actual response (via Tcl_UtfToExternalDString) before it is sent back to the browser. Converting a utf-8 string to external, converts each byte of a utf-8 character thus leading to the awkward result. Finally, I suggest checking the data in the db before delving deeper into encodings.
Collapse
Posted by Gustaf Neumann on
ns_startcontent is deprecated since 2007 (see [1] for the discussion). -g

[1] https://sourceforge.net/p/naviserver/mailman/message/8350950/

Collapse
Posted by Frank Bergmann on
Hi!

Thanks for the quick replies first of all.

I spent a few hours producing sample code and unsuccessfully trying to understand character encoding in AOL/NaviServer. Until I found out that I simply have to replace ns_startcontent with ReturnHeaders. Sorry, this didn't become clear from your discussion.

- Maybe you want to update the documentation of ns_headers. It does't say anything about "configuring the streams", which it should. I understood it's just a variant of ns_write.
- Why did't you update ns_startcontent to behave consistently in NaviServer?

Even following the SF discussion on ns_headers I don't understand why you had to break something, even if it is "old" or "inconsistent".

> This is all handled automatically now, and ns_startcontent should not be used.

Breaking stuff means increased costs for us (and others) to update. If you break stuff, then we'll need to fall back to the policy to update only if absolutely necessary. The last time we managed to stay 7 years in that mode...

Cheers,
Frank

Collapse
Posted by Gustaf Neumann on
It would have taken me much less time digging around in po code if you would have submitted a proper minimal bug report.
I have provided you with simple functions showing how streaming pages are supposed to work, using these should be quite straightforward for an experienced developer.

The function "ns_startcontent" was deprecated in NaviServer about 9 years ago, so i am not impressed by the the 7 years your code worked for aolserver :). The function was probably at no time working in NaviServer exactly as in aolserver (my guess is that the differences might be related with the various strange charset options in aolserver which are gone in NaviServer). It would have been probably better to remove the function in NaviServer 10 years ago, so the difference would have popped up sooner.

Anyhow, the right way for OpenACS applications is to use the abstractions provided by OpenACS and not using the low-level functions. These were introduced 15 years ago not without reason.

all the best
-g

http://openacs.org/forums/message-view?message_id=27658

Collapse
Posted by Frank Bergmann on
Thanks a lot!!

Frank