Forum OpenACS Development: Ain’t No ☀shine

Collapse
Posted by Michael Aram on
Dear all,

can anyone explain to me what is happening here: The following code (ds/shell) works as expected in my browser:

set v "<SUN></SUN>"
ns_return 200 text/plain $v

The same is true for those snippets, they "work" in the sense that they are a workaround for the bug below:

set v "<SUN></SUN>"
::xowiki::write_file /tmp/sun $v
ns_return 200 text/plain ${v}v

set v "<SUN></SUN>"
::xowiki::write_file /tmp/sun $v
append v " "
ns_return 200 text/plain $v

However, the following snippet makes NaviServer deliver the file as "attachment" to the browser

set v "<SUN></SUN>"
::xowiki::write_file /tmp/sun $v
ns_return 200 text/plain $v

NaviServer 4.99.14/Tcl8.5 and 4.99.15/Tcl8.6 unexpectedly return a file (where the sun is truncated to a one-byte character), whereas NaviServer 4.99.11/Tcl8.5 behaves as expected (no file returned, output rendered in the browser)

Collapse
2: Re: Ain’t No ☀shine (response to 1)
Posted by Iuri Sampaio on
Michael,

Although I can't tell you what it is, you could start writing ns_logs Notice within the ad_proc ::xowiki::write_file

Perhaps they clarify what has been done precisely.

Collapse
3: Re: Ain’t No ☀shine (response to 2)
Posted by Iuri Sampaio on
Here, I have the same behavior
nsmain: NaviServer/4.99.14 starting^[[0m

It seems stdout has been wrongly assigned to file output instead of html renderized, as it is in the first samples ( i.e. The ☀ shines!) rsrsrs

Collapse
4: Re: Ain’t No ☀shine (response to 3)
Posted by Michael Aram on
Just to be clear: The problem does not show up, without the three-byte UTF-8 sun. So the following does work:

set v "SUN"
::xowiki::write_file /tmp/sun $v
ns_return 200 text/plain $v

Moreover, ::xowiki::write_file per se is not the culprit. It seems to be the binary translation of fconfigure. The following works as expected,

set v "☀"
set F [::open /tmp/sun w]
::fconfigure $F -translation auto
::puts -nonewline $F $v
::close $F
ns_return 200 text/plain $v

while the following does not:

set v "☀"
set F [::open /tmp/sun w]
::fconfigure $F -translation binary
::puts -nonewline $F $v
::close $F
ns_return 200 text/plain $v
Collapse
6: Re: Ain’t No ☀shine (response to 4)
Posted by Stefan Sobernig on
Hi Michael!

I admit, I might not get the problem here, fully. Maybe it is an unwanted interaction between file I/O (the way xowiki::write_file works, defaulting to binary with requiring the caller to handle the input?) and ns_return which sniffs for the internal representation of a Tcl_Obj.

But, for me, your example runs just fine when treating the Tcl_Obj (the value of v) properly, as Tcl intends:


set v "\u2600"; # ☀
set v [encoding convertto utf-8 $v]
::xowiki::write_file /tmp/sun $v
set v [encoding convertfrom utf-8 $v]
ns_return 200 text/plain $v

Maybe we can clarify first what is surprising to you?

* ::xowiki::write_file should properly deal with the re-coding (stringrep -> bytearray)? This can become hairy when the provenance of the data is not clear. Better leave it with the caller, and document it?

* I am a little distant to NaviServer internals right now, so I am not an authority here, but for my taste, ns_return (w/ and w/o -binary) is maybe too bold to sniff on the byterarray nature (mainly, because it makes a huge difference whether it is a said pure bytearray without stringrep or a bytearray with stringrep).

HTH,
Stefan

Collapse
5: Re: Ain’t No ☀shine (response to 1)
Posted by Gustaf Neumann on
What happens is the following:
  • tcl8.6 has a more aggressive sharing of Tcl_Objs than tcl8.5
  • in tcl8.6, $v passed to ns_return is of type bytearray (meaning "binary data")
  • the conversion to binary happens in the xowiki function, when the content of $v is written via "-encoding binary".
  • interestingly, the length of the bytearray is reported from Tcl via Tcl_GetByteArrayFromObj() is 1, although the "black sun" is actually 3 bytes long.
  • the conversion to the bytearray can be done more easily with the roundtrip over base64 than writing to a file via xowiki (see below)

One can easily test the behavior by commenting in/out the base64 round-trip in the code below:

set v "☀"
#set v "*"
set v [binary decode base64 [binary encode base64 $v]]
ns_return 200 "text/plain" $v
i am not sure, what the best solution is, and whether this is a bug in tcl8.6 (the intended semantics of "number of chars" vs. "number of bytes" is not always clear to me). i will look closer, when time permits.

What can you do in the meanwhile? The easiest is to use "*" instead of "☀" :), or can certainly force a conversion from bytearray by string append operations (as you showed above).

Collapse
7: Re: Ain’t No ☀shine (response to 5)
Posted by Gustaf Neumann on
Michael,

if possible, get the tip version of NaviServer from bitbucket. it should work with your example with tcl 8.7a1 and probably as well with tcl8.6.*. My example did not work for other reasons, the example below is supposed to work. Handling binary data in tcl is still a pain (and a moving target).

set v "☃"
set v [encoding convertfrom utf-8 [binary decode base64 [binary encode base64 [encoding convertto utf-8 $v]]]]
ns_return 200 "text/plain" $v
Collapse
8: Re: Ain’t No ☀shine (response to 1)
Posted by Michael Aram on
Thank you all for your comments!

Firstly, please allow me to mention, that the concrete examples that I have provided are somewhat artificial. Our initial problem was related to "not-usual" encoding problems (i.e. the browser rendered garbage instead of German umlauts). However, the problem existed only on certain "places" (not "everywhere" on the whole instance) and it did not show up on older instances running NaviServer 4.99.11/Tcl8.5 but only on newer instances with newer NaviServer versions (not only Tcl 8.6, also Tcl 8.5).

So, in the course of trying to grasp this, we played around much. It was kind of tricky, because for example ns_log always rendered a three-byte sun, whereas the immediately following ns_return returned a one-byte-broken-sun to the browser. However, ns_return did not have problems with three-byte suns per se. We suddenly realized that "touching" the variable (e.g. via append) seemed to solve the problem completely. In the course of this debugging session, we tried to write the content to a file, just for debugging purposes, and suddenly saw the strange behavior reported in this forum thread. Now we luckily had stumbled upon a reproducible snippet that produced a similar situation as we encounter (we were not able to produce a "damaged" variable that poses problems to ns_return any another way).

The surprising thing in my "fconfigure" example is, that the variable v is not "written", but only used as input source for the file writing process. And this fact alone changes the state of the internal representation in a way, that leads to the problematic behavior.

Just to be clear, let me again show the problem, this time with more details about responses. The following example from above (which does not use Tcl8.6 idioms) triggers the following response from a NaviServer 4.99.14 with Tcl 8.5.

set v "set v "SUN☀SUN"
set F [::open /tmp/sun w]; ::fconfigure $F -translation binary; ::puts -nonewline $F $v ; ::close $F
ns_return 200 text/plain $v
The browser opens the request as a download/attachment file, the name of which is "shell" and the content of which is the content of the v variable (with a 1-byte broken "sun"). mitmproxy's hex-output mode shows the following response from the server (the sun is a 00 here)
2017-11-15 10:44:29 POST https://example.com/ds/shell
                         ← 200 text/plain 7B 59ms

Server:          nginx
Date:            Wed, 15 Nov 2017 09:43:26 GMT
Content-Type:    text/plain
Content-Length:  7
Connection:      keep-alive
X-User-Id:       297369
X-Thread-Id:     7f2d6affd700
Accept-Ranges:   bytes

HEX VIEWER
0000000000 53 55 4e 00 53 55 4e                              SUN.SUN
However, when only changing binary to auto (or when omitting the whole second line), the request succeeds.
set v "SUN☀SUN"
#set F [::open /tmp/sun w]; ::fconfigure $F -translation auto; ::puts -nonewline $F $v ; ::close $F
ns_return 200 text/plain $v
The browser renders the expected result, and mitmproxy shows three bytes for the sun.
2017-11-15 10:45:41 POST https://example.com/ds/shell
                         ← 200 text/plain 9B 134ms

Server:          nginx
Date:            Wed, 15 Nov 2017 09:44:38 GMT
Content-Type:    text/plain; charset=utf-8
Content-Length:  9
Connection:      keep-alive
X-User-Id:       297369
X-Thread-Id:     7f2d6affd700

HEX VIEWER
0000000000 53 55 4e e2 98 80 53 55 4e                        SUN...SUN
So it cannot be a problem with Tcl 8.6 or higher, as it exists under Tcl 8.5 as well. Do you expect, that the HEAD version of NaviServer fixes the problems also with Tcl8.5?
Collapse
9: Re: Ain’t No ☀shine (response to 8)
Posted by Stefan Sobernig on
Hi Michael!

Do you expect, that the HEAD version of NaviServer fixes the problems also with Tcl8.5?

If you make sure that you massage the value (Tcl_Obj) that you pass to/ receive from properly with [encoding convertto] and [encoding convertfrom], the examples that you showed will work whatever Tcl version you run. This is not some issue of some Tcl version you are looking at. It is a more general requirement when you turn Tcl (internal) values into an external value; and vice versa.

browser opens the request as a download/attachment file
This is a symptom of a subtle interaction of failing in complying with the above value protocol plus the way NaviServer (presumably post 4.99.11) sniffs on the value type. It is not related to Tcl. If one fails to comply with the above, will not be presented a broken string as an octet-stream (file download), but a broken string rendered by the browser. No win, just a shift.

So, in short (haven't checked but that is my understanding):

- use [encoding convertto] and [encoding convertfrom] in a disciplined manner when handling your values to an IO channel that does not run the transformation on its own (this is what you observe as the diff between -translation auto/binary, -translation binary implies -encoding binary, meaning don't touch the outgoing value).

- if you want to have the unexpected octet-stream go away in the responses on broken strings, you should update NaviServer, but you have to get the above right anyway. But emphasis is on the first item.

HTH, Stefan

Collapse
10: Re: Ain’t No ☀shine (response to 8)
Posted by Stefan Sobernig on
The surprising thing in my "fconfigure" example is, that the variable v is not "written", but only used as input source for the file writing process. And this fact alone changes the state of the internal representation in a way, that leads to the problematic behavior.
It may be surprising because you were presented a broken string plus file download. As I tried to explain, this is/was a bad interaction between not shaping the value (bytearray) properly on its way out of Tcl and NaviServer sniffing on the presence of a bytearray, but because of its brokeness presenting an octet-stream.

The fact the values in Tcl (the value Tcl_Obj referenced by variable v) is transmogrified into different internal representations (e.g., different bytearrays without you noticing) with or without an external (string) representation, this is at the heart of Tcl.

Some of the surprise will go away when using [encoding convertto] as shown below because it will take care of not touching any shared value:

set v "\u2600"; # ☀
::xowiki::write_file /tmp/sun [encoding convertto utf-8 $v]
ns_return 200 text/plain $v
The value being written out to the file will distinct from the one ending up in ns_return. As I said, one must follow the protocol.
Collapse
12: Re: Ain’t No ☀shine (response to 10)
Posted by Michael Aram on
Thank you, Stefan, for your extensive clarifications. Your approach might be the correct way to deal with this kind of stuff when programming in plain Tcl, and maybe as well in special cases when implementing NaviServer/OpenACS applications. However, in the typical case of development "inside the framework (NaviServer/OpenACS/XO* Packages)", there is usually no need to convert values from or to UTF-8. The framework deals already with this, so most attempts to "convert" something result in over-converting something.

Actually, the developer who faced this bug in the first place, tried to fix the problem using "convert", and even thought she had fixed it (using [encoding convertfrom [ns_conn encoding] $result]). However, this "fixed" the problem only for German umlauts, not for the three-byte sun (which was not part of the initially observed problem).

During code review I saw this "conversion-based fix" and immediately had the suspicion that there must be something wrong at some other place. Now that we have found the bug, or at least a solution within the framework, everything is fine! Thank you!

Collapse
11: Re: Ain’t No ☀shine (response to 1)
Posted by Gustaf Neumann on
The short answer to your question, whether or not the tip version of NaviServer will work on that kind of data is: test it! I have set up a test environment with Tcl 8.5, and yes, it works.

The whole situation with Tcl byte arrays is tricky (to use a polite word). I am pretty sure, you do not want to know the all details but since you brought it up, here it goes. Probably, i have not got everything completely correct, and for sure, details change between Tcl versions.

A "bytearray" is an internal type of an Tcl_Obj for interpreting the internal representation of a Tcl value (not necessarily a Tcl variable). There are situations, when a Tcl byte array is really representing binary data (e.g. content of an image), then there are situations, where byte arrays are used, where content has a string representation in UTF-8, but which does not fit into Tcl's internal UCS-2 representation (e.g. >2 byte UTF-8 characters), and situations, where one has a subset of UTF-8 that fits into UCS-2. Depending on this situation, Tcl has an internal machinery trying to detect, when it is safe to use a byte array directly, depending on the fact, whether or not the byte array has a string rep (the latter is called a pure byte array). Unfortunately, it is also possible to create a string rep in situations, where one should not have a string rep (e.g when the one interactively enters a command returning a pure byte array, or when the Tcl C-API function Tcl_GetStringFromObj() is called on such an Tcl_Obj, eg. in a ns_log or some other command used for debugging. In these cases, it happens easily, that a wrong representation is chosen. Here comes the converto/convertfrom into play, which mostly here for converting to/from the UCS-2 rep and producing proper pure Tcl byte arrays.

For a Tcl application developer, it is not so easy to know, when convertto is necessary. Some good Tcl guys are working to improve the situation.

For example, starting Tcl 8.7a1, Tcl has now two types of Tcl byte arrays, the classical one and a so called "proper" bytearray, replacing the former pure bytearray, and making it more robust against creating string reps. I addition to these, there were many changes to address various bug reports from that area.

Just the indicator "german umlauts are working or not" is not a good indicator, whether the encoding is right. Here is a pure Tcl example, showing the "umlaut" correctly (since it is a 2-byte char) but the black sun is damaged. The code without the converto operations works only fine for "a" and "ü" (UTF-8 2 char), bot not for "☀" (UTF-8 3 char).

foreach v {"a" "ü" "☀"} {
   puts "v <$v> v1 <[binary decode base64 [binary encode base64 $v]]>"
} 
with convert operations at the right places, it works fine.
foreach v {"a" "ü" "☀"} {
   puts "v <$v> v1 <[encoding convertfrom utf-8 [binary decode base64 [binary encode base64 [encoding convertto utf-8 $v]]]]>"
}
Nobody is happy about this, people are working to make it better on Tcl 9.

Hope this helps
-gn

Collapse
13: Re: Ain’t No ☀shine (response to 11)
Posted by Michael Aram on
Thank you very much for your answer. I am interested in "all the details", thank you for the extensive answer. I did not immediately test your fix, as you were explicitly refering to Tcl versions 8.6 and 8.7. I simply thought, if you dont expect it to work for Tcl 8.5, for whatever reason, there is no point in setting up a HEAD-version test environment.

Anyhow, now I have compiled a NaviServer that has the same versions for all its components except for NaviServer itself (from 4.99.14 to 4.99.16d10/HEAD), and I can confirm that the problem disappears when using the HEAD version that includes your fix. The problem also disappears, when doing an append myvar "".

So, finally, the problem seems to be solved. I simply add the workaround and wait for the next NaviServer release.

Thank you, again, for your help!!

PS: Actually, the "culprit" in the "real code" was an ns_log line a few lines above (my log 3-[encoding convertfrom [ns_conn encoding] $interactionParams(preview)]). The problem disappears as well, when I simply remove that line. It seems to have a similar effect as the -translation binary in the artificial example.