Forum OpenACS Development: Image resizing, exec and server problems

Alternatively I can use the one provided by AOLserver now and put it into my install directions, which is probably preferred. Tom, if you look at http://cognovis.de/developer/en/aolserver_install would it be included by default or would I need to get it separately (which is what I assume)? If yes, where from and where to put it to? Additionally does it need special treatment in the config.tcl file?

Tom, have you had a look at the image:: procedures written by Dave? I think a comparison what they do and at which level (inside TCL or outside with imagemagick, alternatives) they interface would be interesting.

a) Edit the sourcecode for image generation and use ns_proxy there. Use a specific ns_proxy pool for image generation only

b) Write a generic "exec_proxy" procedure which works just as exec, but executes through an ns_proxy. I could probably add background capabilities to that (so you could push it to the background and have errors reported in the error.log or as user messages). Replace exec with exec_proxy in the places where it matters.

If I go with "b" is it fine to just use the default configuration for the "exec" proxy or should I provide parameters to tune that? Should I put this in acs-tcl?

Now.... I could rename "exec" to "real_exec" using the rename command. Interestingly the proxy still requires the call to go to "exec", so I might not even have to do the rename.

Sadly I have no clue how I can write a procedure which checks if proxy::exec exists and then calls it with all the arguments, as exec allows for an arbitrary number of arguments and I don't know how to model that in ad_proc.

1. add the following entry to your etc/config.tcl:

   # ...
   if {[ns_info version] >= 4.5} {
     if {[file exists ${bindir}/nsproxy.so]} {                
       ns_param	nsproxy		${bindir}/nsproxy.so        
     }        
     ns_limits set default \
         -maxupload [ns_config ns/server/${server}/module/nssock maxinput]
   }
   # ...
   

   Note, I added it to the 4.5+ branch as it won't be available below. You also need to check for its very existence, otherwise, the you will experience a crash. that would be the option for using malte's "ns_proxy proxy".
2. as you outlined above, an on-demand / or lazy initialisation in your code (convenient for testing purposes). in xotcl jargon, as I assume that you stick with it:

   Class X -proc init args {
      if {[info command ns_proxy] eq "" && \
    	    [file exists [ns_info home]/bin/nsproxy.so]} {
          load [ns_info home]/bin/nsproxy.so
          # create a proxy pool 
          ns_proxy config my_pool
        }
        next
   }
   

   Provided that you class object is defined in a *-procs.tcl file, the constructor "init" will only be called once, upon initial sourcing in the driver thread. The "info command" expression takes care for its deployment in www/* scripts.

I'm thinking about user-portraits, for instance, having a "photo.jpg" original file, let' say 1024x768 pixels, you could create other 4 files: medium, small, thumbnail and square versions: photo-m.jpg, photo-s.jpg, photo-t.jpg, photo-sq.jpg

I'm assuming your application is not image resizing? Since Malte never identified what the problem was, I would ignore it until you run into trouble.

Just keep your exec calls simple.

Your requirements call for a scalable and robust solution. ns_proxy (AOLSERVER 4.5+) and, therefore, an upgrade is probably one solution, but still, i would consider the use of exec in your case as a smell of "bad design".

I am wondering whether you ever considered using tcl event loop? It is common sense in mono-threaded tcl environments to neglect exec entirely, and go for pipe indirection through open. To talk turkey in your case:

proc done args {...}
proc is_readable {fid} { ... }
set pipe [open "|zip -9 test.zip 454.pdf"]
fileevent $pipe readable [list is_readable $pipe]
vwait ::done

What is so smelly about exec? Well, I am not an insider, nor have I a complete picture of Tcl internals in the field of tension of *nix/win thread models, BUT I do know that TCLs exec is built upon fork() and forking from within threads (at least in *nix) is critical (due to requirements on callstack set-up, if I am not completely mistaken). Besides, exec is inherently blocking I/O. So, the fileevent/open solution above avoids both issues.

see http://wiki.tcl.tk/880

The only tricky thing (at first glance) is how to use async I/O within the multi-threading environment of AOLSERVER. To give you an example, you are not advised to use vwait from within a AOLServer connection thread. Without vwait, the connection thread (i.e. its interpreter) will finish script execution, close the connection and it will be recycled. Still, you can use it and there is even a design fit with another requirement of yours: "a couple of hundred (perhaps) zip downloads"

Well, I guess you will use the bgdelivery feature that comes with XOTcl enhanced OpenACS environments to free resources for new connections while processing downloads in the background?

see https://openacs.org/xowiki/Boost_your_application_performance_to_serve_large_files!

In that case, you can hijack the bgdelivery infrastructure to deal with the archiving/ tarball business. You can place zipping jobs in the standing/ persistent bgdelivery thread which are then, upon completion, directly delivered back.

The nice thing of this design is that you (a) avoid exec, (b) use async I/O in the back, and (c) it is deployable both under AOL 4.0.10 and 4.5.

you seem to have your head around this pretty tightly.

I'd wish, in the meantime I learnt that the pipe indirection through open also involves a fork(). But, still, async i/o + bgdelivery make sense and provide agility. The tricky thing is to avoid background zombie processes.

I will try to come up with a little solution using bgdelivery that can be tested.

As long as it works for you and you tested (with a minimum effort) that it scales (to whatever extent) than it is apparently fine (at least).

However, another critical thing is the kind of exec call itself (redirector, background, ...) and the code surrounding it. Malte never posted the critical exec line, at least to my knowledge.

In other words, if it works for you. Go ahead!

My option referred more to use a non-blocking variant and link it to the background delivery feature (a larger picture of design).

I agree while there are resources on comp.lang.tcl etc. that report on the threaded fork() issue, I cannot find a concise statement on the issue's state (fixed, open, unsolvable).

Short intro: http://wiki.tcl.tk/9775