Forum OpenACS Development: Re: NaviServer 4.99: Issues with "rename"?

Collapse
Posted by Gustaf Neumann on
Maybe you are running into a conflict with the exec definition of OpenACS [1]. Probably, ::exec_orig is not part of the blueprint. I do strongly recommend not to mess around with exec, especially not in the way you are using it (it is unsafe from tcl, var globbing + "eval" dangers). Why are you not leaving exec alone?

[1] http://openacs.org/api-doc/procs-file-view?path=packages%2facs-tcl%2ftcl%2fproxy-procs.tcl&version_id=4877495&source_p=1

Collapse
Posted by Frank Bergmann on
Hi Gustaf,

]po[ runs on both Linux and Windows (and other stuff if necessary). As part of ]po[ we use a lot (>10) of Linux commands like find, ifconfig, dot, ...

The easiest way to keep the Windows and Linux versions in sync is to use exactly the same commands on the TCL side, and have a custom version of "exec" that performs some path corrections etc. This way the TCL part is exactly identical. This is very, very important for testing etc.

We woul have to touch all "exec" calls in the entire system and replace them by "im_exec", if we don't manage to get a custom version of exec going.

I'll check the link below to check if we can work around the issues with OpenACS exec.

Thanks!
Frank

Collapse
Posted by Gustaf Neumann on
The exec implementation in OpenACS is built around nsproxy (in cases where nsproxy is available). The reasons for using nsproxy are reliability (tcl's exec has several problems with concurrency, which might lead to hangs and crashes, hopefully solved in 8.5.19) and resource consumptions (every [exec] calls fork, which requires to "copy"/preallocate the full vmsize, which leads in combination with zippy malloc easily to out-of-memory problems, when OpenACS is large and the memory is little (e.g. 4GB). These exec problems are issues in unix versions, therefore using nsproxy is highly recommended on unix systems to avoid problems. If PO replaces [exec] by its own breed, it reintroduces the mentioned problems leading to limited reliability.

what do you mean by "path corrections"? paths pointing to the correct binaries? are you aware about util:which? If PO uses only 10 external progs, the right approach is to locate these during startup which util::which, store the full paths to the binaries in variables (best in the blueprint) and use these variables in exec without runtime overhead.

-g
[1] http://openacs.org/api-doc/proc-view?proc=util%3a%3awhich&source_p=1

Collapse
Posted by Frank Bergmann on
Hi!

> right approach

The priority is to get ]po[ V5.0 out and not to spend any more time on infrastructure. The "exec" method has never caused any problems in the past, so there ain't nothin' to fix if there ain't no problem (not sure about the spelling...)

It seems I spent too much time in Spain and the US, so that I doubt any "right approach" concept, unless there is a business case for it :-)

Frank

Collapse
Posted by Maurizio Martignano on
Dear Frank and Gustaf,
let me add some few points to this discussion.
When calling an external program/utility we basically need two things:
1. know where the program is (so that we can call it)
2. pass to it the proper parameters (in case they are needed).
Up to now in the discussion you have covered only point 1.
But suppose our program/utility requires as parameter a file name.
In this case file naming conventions are also important "inside" the command line parameters.
Things become even trickier when we used mixed models of file naming conventions. I'll try to explain: in Linux all executable binaries follow the same files naming conventions. In pure Windows, when all the executable binaries are all native Windows binaries, they all follow the same files naming conventions.
On the contrary when some of the binaries are Windows native and others are Cygwin (or MinGW) binaries they are not guaranteed to follow the same files naming conventions.

Hope it helps,
Maurizio

Collapse
Posted by Maurizio Martignano on
I'm adding an example to explain my point.
Let's take the following simple C Program:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

#define LEN 2048

char command[LEN+1];

int main(int argc, char *argv[]) {
    int i;    
    for (i = 1; i < argc; i++) {
        strcat(command, argv[i]);
        strcat(command, " ");
    }
    printf("Executing command = %s...\n", command);
    system(command);
    printf("Done!\n");
    return 0;
}

Let's compile it both as Native Windows (exec_win) and as Cygwin-64 (exec_cyg) and let's run it from within Cygwin-64...

Maurizio@MMNASUS /cygdrive/c/smart/prog/provec
$ exec_win which ls
/usr/bin/ls
Executing command = which ls ...
Done!

Maurizio@MMNASUS /cygdrive/c/smart/prog/provec
$ exec_cyg which ls
Executing command = which ls ...
/usr/bin/ls
Done!

Maurizio@MMNASUS /cygdrive/c/smart/prog/provec
$ exec_win ls c:\\tmp
Executing command = ls c:\tmp ...
Done!

Maurizio@MMNASUS /cygdrive/c/smart/prog/provec
$ exec_cyg ls c:\\tmp
Executing command = ls c:\tmp ...
ls: cannot access 'c:tmp': No such file or directory
Done!
Collapse
Posted by Gustaf Neumann on
i have spent a lot of time about handling exec issues of tcl in aolserver and naviserver, which usually only happen under heavy usage, when the server have log uptimes, etc. I am just pointing out consequences of design decisions, which the person rewriting "exec" for application specific needs might not have been aware off.

The "right approach" is the one which reduces on the longer range the maintenance cost and reliability complaints....

Collapse
Posted by Frank Bergmann on
Hi!

> reduces ... maintenance costs

That's a perfect business case :-) Well, the point in ]po[ is that most "execs" are used in administrator functions like periodic integration with external system etc. To my knowledge there is no exec in operational procedures, except for the GrahpWiz "dot" calls in the acs-workflow package. This may be the reason why we didn't have any issues under high load etc. so far.

Concerning Maurizio's post:
I'm aware that there may be incompatibilities in the arguments of an exec command, and not only in the path to the binary. However, all CygWin binaries use the Linux "/" forward slash notation that is also accepted in TCL. So that is why we're on pretty save ground if we keep everything to the Linux standard.

> proxy-procs.tcl

I haven't tested yet (easter holidays...), but I'm quite confident that I can fix our current issues with the knowledge about this new proxy::exec. I understand the arguments given by Gustaf in favor, and I believe this is great work. It might even be worth the pain to change the respective ]po[ code. I'll check and may come back here on the forum.

Thanks a lot for the great work!
Frank