Forum OpenACS Development: Re: Ain’t No ☀shine

Thank you all for your comments!

Firstly, please allow me to mention, that the concrete examples that I have provided are somewhat artificial. Our initial problem was related to "not-usual" encoding problems (i.e. the browser rendered garbage instead of German umlauts). However, the problem existed only on certain "places" (not "everywhere" on the whole instance) and it did not show up on older instances running NaviServer 4.99.11/Tcl8.5 but only on newer instances with newer NaviServer versions (not only Tcl 8.6, also Tcl 8.5).

So, in the course of trying to grasp this, we played around much. It was kind of tricky, because for example ns_log always rendered a three-byte sun, whereas the immediately following ns_return returned a one-byte-broken-sun to the browser. However, ns_return did not have problems with three-byte suns per se. We suddenly realized that "touching" the variable (e.g. via append) seemed to solve the problem completely. In the course of this debugging session, we tried to write the content to a file, just for debugging purposes, and suddenly saw the strange behavior reported in this forum thread. Now we luckily had stumbled upon a reproducible snippet that produced a similar situation as we encounter (we were not able to produce a "damaged" variable that poses problems to ns_return any another way).

The surprising thing in my "fconfigure" example is, that the variable v is not "written", but only used as input source for the file writing process. And this fact alone changes the state of the internal representation in a way, that leads to the problematic behavior.

Just to be clear, let me again show the problem, this time with more details about responses. The following example from above (which does not use Tcl8.6 idioms) triggers the following response from a NaviServer 4.99.14 with Tcl 8.5.

set v "set v "SUN☀SUN"
set F [::open /tmp/sun w]; ::fconfigure $F -translation binary; ::puts -nonewline $F $v ; ::close $F
ns_return 200 text/plain $v

The browser opens the request as a download/attachment file, the name of which is "shell" and the content of which is the content of the v variable (with a 1-byte broken "sun"). mitmproxy's hex-output mode shows the following response from the server (the sun is a 00 here)

2017-11-15 10:44:29 POST https://example.com/ds/shell
                         ← 200 text/plain 7B 59ms

Server:          nginx
Date:            Wed, 15 Nov 2017 09:43:26 GMT
Content-Type:    text/plain
Content-Length:  7
Connection:      keep-alive
X-User-Id:       297369
X-Thread-Id:     7f2d6affd700
Accept-Ranges:   bytes

HEX VIEWER
0000000000 53 55 4e 00 53 55 4e                              SUN.SUN

However, when only changing binary to auto (or when omitting the whole second line), the request succeeds.

set v "SUN☀SUN"
#set F [::open /tmp/sun w]; ::fconfigure $F -translation auto; ::puts -nonewline $F $v ; ::close $F
ns_return 200 text/plain $v

The browser renders the expected result, and mitmproxy shows three bytes for the sun.

2017-11-15 10:45:41 POST https://example.com/ds/shell
                         ← 200 text/plain 9B 134ms

Server:          nginx
Date:            Wed, 15 Nov 2017 09:44:38 GMT
Content-Type:    text/plain; charset=utf-8
Content-Length:  9
Connection:      keep-alive
X-User-Id:       297369
X-Thread-Id:     7f2d6affd700

HEX VIEWER
0000000000 53 55 4e e2 98 80 53 55 4e                        SUN...SUN

So it cannot be a problem with Tcl 8.6 or higher, as it exists under Tcl 8.5 as well. Do you expect, that the HEAD version of NaviServer fixes the problems also with Tcl8.5?

The surprising thing in my "fconfigure" example is, that the variable v is not "written", but only used as input source for the file writing process. And this fact alone changes the state of the internal representation in a way, that leads to the problematic behavior.

It may be surprising because you were presented a broken string plus file download. As I tried to explain, this is/was a bad interaction between not shaping the value (bytearray) properly on its way out of Tcl and NaviServer sniffing on the presence of a bytearray, but because of its brokeness presenting an octet-stream.

The fact the values in Tcl (the value Tcl_Obj referenced by variable v) is transmogrified into different internal representations (e.g., different bytearrays without you noticing) with or without an external (string) representation, this is at the heart of Tcl.

Some of the surprise will go away when using [encoding convertto] as shown below because it will take care of not touching any shared value:

set v "\u2600"; # ☀
::xowiki::write_file /tmp/sun [encoding convertto utf-8 $v]
ns_return 200 text/plain $v

The value being written out to the file will distinct from the one ending up in ns_return. As I said, one must follow the protocol.