Forum OpenACS Development: Re: Image resizing, exec and server problems

Your requirements call for a scalable and robust solution. ns_proxy (AOLSERVER 4.5+) and, therefore, an upgrade is probably one solution, but still, i would consider the use of exec in your case as a smell of "bad design".

I am wondering whether you ever considered using tcl event loop? It is common sense in mono-threaded tcl environments to neglect exec entirely, and go for pipe indirection through open. To talk turkey in your case:

proc done args {...}
proc is_readable {fid} { ... }
set pipe [open "|zip -9 test.zip 454.pdf"]
fileevent $pipe readable [list is_readable $pipe]
vwait ::done

What is so smelly about exec? Well, I am not an insider, nor have I a complete picture of Tcl internals in the field of tension of *nix/win thread models, BUT I do know that TCLs exec is built upon fork() and forking from within threads (at least in *nix) is critical (due to requirements on callstack set-up, if I am not completely mistaken). Besides, exec is inherently blocking I/O. So, the fileevent/open solution above avoids both issues.

see http://wiki.tcl.tk/880

The only tricky thing (at first glance) is how to use async I/O within the multi-threading environment of AOLSERVER. To give you an example, you are not advised to use vwait from within a AOLServer connection thread. Without vwait, the connection thread (i.e. its interpreter) will finish script execution, close the connection and it will be recycled. Still, you can use it and there is even a design fit with another requirement of yours: "a couple of hundred (perhaps) zip downloads"

Well, I guess you will use the bgdelivery feature that comes with XOTcl enhanced OpenACS environments to free resources for new connections while processing downloads in the background?

see https://openacs.org/xowiki/Boost_your_application_performance_to_serve_large_files!

In that case, you can hijack the bgdelivery infrastructure to deal with the archiving/ tarball business. You can place zipping jobs in the standing/ persistent bgdelivery thread which are then, upon completion, directly delivered back.

The nice thing of this design is that you (a) avoid exec, (b) use async I/O in the back, and (c) it is deployable both under AOL 4.0.10 and 4.5.

you seem to have your head around this pretty tightly.

I'd wish, in the meantime I learnt that the pipe indirection through open also involves a fork(). But, still, async i/o + bgdelivery make sense and provide agility. The tricky thing is to avoid background zombie processes.

I will try to come up with a little solution using bgdelivery that can be tested.

As long as it works for you and you tested (with a minimum effort) that it scales (to whatever extent) than it is apparently fine (at least).

However, another critical thing is the kind of exec call itself (redirector, background, ...) and the code surrounding it. Malte never posted the critical exec line, at least to my knowledge.

In other words, if it works for you. Go ahead!

My option referred more to use a non-blocking variant and link it to the background delivery feature (a larger picture of design).

I agree while there are resources on comp.lang.tcl etc. that report on the threaded fork() issue, I cannot find a concise statement on the issue's state (fixed, open, unsolvable).

Short intro: http://wiki.tcl.tk/9775