Forum OpenACS Q&A: Can't exec a command in tcl page

Collapse
Posted by Antonio Pisano on
Hello everyone,

I am trying to exec libreoffice in a tcl page so to convert a csv into an xls and return that to the user, but the server keeps complaining that "child process exited abnormally".

I have checked the command into the bash shell and into the tcl shell on the same server and it works properly. I think this could have something to do with the ns_proxy (which is in place)

For reference, the command I issue is this:

libreoffice --headless --convert-to xls /tmp/fileBXXCcF.csv

where /tmp/fileBXXCcF.csv is a comma-separated csv

What could that be? Any hint will be appreciated.

Antonio

Collapse
Posted by Benjamin Brink on
Antonio,

Try using a full pathname for "libreoffice", such as /usr/local/libreoffice but with the location of libreoffice.

It may be that the server's environmental variables are not set during startup, so the server cannot find "libreoffice".

cheers,

Ben

Collapse
Posted by Antonio Pisano on
Thanks Ben, it sounded reasonable, but unfortunately it is not the case: using the full path won't help...
Collapse
Posted by Antonio Pisano on
I've also tried issuing 'libreoffice --headless --help' as command, with just the relative path and it works on the server (shows the options properly).

I think it must be something more subtle related to execution into proxy...

Collapse
Posted by Benjamin Brink on
Examine the server log. Does the error message provide more detail?
Collapse
Posted by Antonio Pisano on
This is what the log says... nothing new...

child process exited abnormally
while executing
"exec libreoffice --headless --convert-to xls /tmp/filedNva4p.xls /tmp/filedNva4p"
invoked from within
"ns_proxy eval $handle "exec $call""
("uplevel" body line 2)
invoked from within
"uplevel $code"
invoked from within
"with_finally -code {
set return_string [ns_proxy eval $handle "exec $call"]
} -finally {
ns_proxy release $handle
}"
(procedure "proxy::exec" line 4)
invoked from within
"proxy::exec -call $args"
(procedure "exec" line 1)
invoked from within
"exec libreoffice --headless --convert-to xls ${tmpfile}.xls $tmpfile"
("uplevel" body line 212)
invoked from within...

Calling the very same exec command in tclsh by the command line (on the same server) works as expected.

Collapse
Posted by Antonio Pisano on
The real command is this, took the wrong part of the log.

exec libreoffice --headless --convert-to xls /tmp/fileBXXCcF.xls /tmp/fileBXXCcF.csv

Same error though.

Collapse
Posted by Dave Bauer on
Does the operation complete successfully?

We had this problem with many commands, where the output of the command causes Tcl exec to think the command has failed when it was fine.

We catch the exec and look for specific error responses in the output as necessary.

Many sites I have worked on used a modified exec replacement that took care of the catch automatically.

Collapse
Posted by Gustaf Neumann on
A common cause is that the executed script outputs to stderr. This leads per-default to the same failure like a non-zero return code. I would recommend to redirect stderr to stdout (adding the the script 2>@1)

A longer discussion is e.g. here: http://wiki.tcl.tk/1039

Collapse
Posted by Neophytos Demetriou on
The following seems to do the trick for me (even though I would have preferred to go with what Gustaf says):

    exec -- /bin/sh -c "$cmd || exit 0" 2> /dev/null

Another way is to use ns_proxy as follows:

    set handle [ns_proxy get exec_proxy]
    set result ""
    if { [catch { set result [$handle "exec {*}${cmd}"] } errmsg] } {
        ns_log notice "proxy exec: errmsg=$errmsg"
    }
    ns_proxy release $handle
    return $result

You might want to check OdfConverter for DOCX. I had used it with some success in MediaBox a long time ago. I remember having some issues with headless OpenOffice at the time (circa 2006-2007) and thus I was using the DocumentConvert.py --- it's kind of old but I have uploaded anyway here:

https://openacs.org/storage/?folder_id=2551358

Collapse
Posted by Jim Lynch on
One question... won't libreoffice and ms office, gnumeric, etc accept a CSV? Why convert into a closed format when the open one should work just fine?

Just Sayin...

-Jim

Collapse
Posted by Jim Lynch on
When you tested the commands using bash, did you check the exit status every time?
Collapse
Posted by Antonio Pisano on
Thanks for the many hints guys. Sadly no luck with this by now...

I need to convert csv to xls because I need an export function which could Just Work for a customer of ours. They expect a file to be loaded in their system to be an xls, so...

Anyway, I have tried redirecting standard error (and everything else) using &> /dev/null at the end of the command, but didn't work out (same error).

Catching the exec and ispecting its output shows a CHILDSTATUS error, but this we knew already.

I have also thought it could be a matter of different current folder, so I changed the command to

/usr/bin/libreoffice --headless --convert-to xls /tmp/fileugDbjI.csv --outdir /tmp

to force output folder to be /tmp. As before, the command works fine in the shell (return code is 0)

Using exec to invoke a shell as Neophytos suggested brings to the very same behaviour.

I am out of clues... think I will try to reproduce the problem on another server to see if it is due to this particular machine/configuration or if there is something going wrong with the proxy

Collapse
Posted by Neophytos Demetriou on
FWIW, the following works for me - I've tried it from nscp but should work just as well from elsewhere / a TCL page:

set cmd "/usr/bin/libreoffice --headless --convert-to xls /tmp/test2.csv --outdir /tmp"

set handle [ns_proxy get exec_proxy]
set result ""
if { [catch { set result [$handle "exec {*}${cmd}"] } errmsg] } {
      ns_log notice "proxy exec: errmsg=$errmsg"
}
ns_proxy release $handle

Collapse
Posted by Antonio Pisano on
I've tried your snippet, but the problem persists: logs report the usual error and the xls is not created.

Would you mind, at your convenience, to issue the command into a ds/shell or a minimal tcl webpage?

Collapse
Posted by Gustaf Neumann on
The hint with the CHILDSTATUS rings a bell. In case Neo's suggestion does not help, one more idea for debugging. Write a wrapper (e.g. named /tmp/convert.sh) like the following
#!/bin/sh
echo xxx `/usr/bin/id`
/usr/bin/libreoffice --headless --convert-to xls SOMEFILE.csv --outdir /tmp
echo "xxx $? exit code"
fill in a .csv tile for testing and call the wrapper via
set result [exec bash /tmp/convert.sh]
i would not be wondering, when you get on your machine now a useful result now, showing a non-zero exit code from libreoffice. Maybe there is a permission problem under the userid of the server? Handing CHILDSTATUS in Tcl requires some special, which i don't see in nsproxy. Probably nsproxy has to be improved to return better errors.
Collapse
Posted by Gustaf Neumann on
I've compared the cases, where tcl-exec and/or nsproxy-exec calls a program with a non-zero return status. In both cases, the behavior is the same, one gets the return status via the global variable ::errorCode. The following snippet shows the result code from the called program:
set cmd {/usr/bin/libreoffice ... --outdir /tmp}
if {[catch {set result [exec {*}$cmd]} errorMsg]} {
   lassign {*}$::errorCode code id value
   if {$code eq "CHILDSTATUS"} {set result "program exited with code $value"}
}
so, the nsproxy behavior is technically correct, just the error message from tcl could be better. What return code do you get in your installation?
Collapse
Posted by Bjoern Kiesbye on
Hi Antonio,

as others already mentioned, if exec results in an error thrown, it is not a reliable indicator that the executed Program did not finish successful. I have the same problem with wget, when updating the product catalog images. Using it from within tcl exec always throws an error, even if the file was downloaded successful. After wget has finished I check if the tmp file exists, to determine if wget finished successful, see below. I would try to redirect the output of libreoffice to a file, like &>/tmp/libreoffice.log to hopefuly get more details about the actual error.

Out of curiosity, you say you executed the command in a shell and it worked, did you execute the command as the same user running Aolserver?


foreach cur_url $image_urls {
set temp_picture_name [lindex [split "$cur_url" "/"] end]
set temp_picture_file "/tmp/${temp_picture_name}"



if [catch {exec $wget --directory-prefix=/tmp/ "$cur_url"} errmsg] {
ns_write "wget faild : $errmsg"
}

ns_write "Check if Picture exists!"

if {[file exist $temp_picture_file]} {
ns_write "Picture exists!"
......
}
}

Collapse
Posted by Antonio Pisano on
Thanks Bjoern,

yes, we considered the possibility of a "fake error" from the command, but this doesn't seem to be the problem. The command actually fails and no output file is created.

I've tried to redirect all the output to a temp file using &>, but the command fails without creating any...

Yes, the command works as expected on the same server both from bash and tclsh.

I was wondering: is there some limitation in privileges for ns_proxy? /tmp/ is world readable, but maybe there are other things ns_proxy is not allowed to do...

Collapse
Posted by Benjamin Brink on
Gustaf writes:

"Maybe there is a permission problem under the userid of the server."

I've had this happen before.

Antonio, what happens if you use a different directory using server's owner and group same as openacs system, such as a openacs-4/tmp/ ?

Collapse
Posted by Gustaf Neumann on
ok, i've installed libreoffice on a ubuntu machine and tried to reproduce Antonio's problem. As suspected in https://openacs.org/forums/message-view?message_id=4168810, the command "/usr/bin/libreoffice ..." exists with exit code "77". Google tells me that the exit code of libreoffice means roughly "insufficient permissions", but unfortunately, libreoffice gives no more details. After some more digging around, it seems that libreoffice requires write permissions on the directory to which the from the environment variable HOME points!

The environment variables are inherited from the shell from where NaviServer was started to nsd and nsproxy. When nsd is started from the startup scripts, it is not so unlikely that HOME points something like "/root", and that the userid under which the server is running has no write permissions. This might explain as well, why e.g. Neophytos got no errors, maybe since he started nsd from his userid.

The solution is to set in the startup script the environment variable HOME to something writable for the userid under which the server runs, e.g. the home directory of the server instance.

all the best
-gustaf neumann

Collapse
Posted by Bjoern Kiesbye on
Hi Antonio,

Gustaf is right with his assumption, try running the command without the --headless flag which suppresses all output including useful warnings and errors. If possible in /ds/shell in case you have developer support installed.
This is the error I got issuing the command:


ERROR:
No protocol specified
Failed to open display
[Java framework] Error in function createSettingsDocument (elements.cxx).
javaldx failed!
Warning: failed to read path from javaldx
No protocol specified
No protocol specified
mkstemp("/root/.execoooaVoik8") failed: Keine Berechtigung
while executing
"exec /usr/bin/libreoffice --convert-to xls /tmp/filedNva4p.xls /tmp/filedNva4p"
("uplevel" body line 1)
invoked from within
"uplevel 1 $script"

Collapse
Posted by Jeff Rogers on
Another option that is still an open format but gives much more control than csv is creating as Office 2003 XML format spreadsheet. I've uploaded a simple program to do this to https://openacs.org/storage/?folder_id=4169041

The program as written doesn't add any formatting to the data, treating it all as plain strings, but it would be easy enough to add appropriate column widths, bolded column headers, etc. The format is documented at http://msdn.microsoft.com/en-us/library/aa140066%28office.10%29.aspx Or you can open the spreadsheet in excel (or LibreOffice), reformat it as you like, and resave it in the same format (i.e., agree to save it in the old format when you get the pop-up), and simply examine what has changed w.r.t styles, fonts, document info, etc.

As far as it just working, I've found that if you save it as a ".xml" file (e.g., by content-disposition header), it will just work in excel (just don't remove the "mso-application" PI from the xml). However if the user just selects "open with excel" when downloading it, they will get a popup saying it's in the wrong format and they have to hit ok before opening it (because the browser sees fit to rename it to whatever.xml.xls), at least in firefox. Peversely, if you specify the mimetype as "application/xml" then the browser doesn't recognize it as an excel file and will open it with "xml editor", which recognizes it as a spreadsheet and opens it with excel with no complaints. And mockingly, it works cleanly in IE either way. All of this is on my machine of course, YMMV.

Collapse
Posted by Antonio Pisano on
Uau, this was subtle!
Thanks a lot to everybody for testing and finding out!

All the best

Antonio

Collapse
Posted by Gustaf Neumann on
The updated version of with_finally includes in the branch oacs-5-8 includes the exit-code and the pid of the child process in the error message in case we have a CHILDSTATUS error. This should avoid threads like this one in the future.