Forum OpenACS Development: blueprint and *-init.tcl files

Collapse
1: blueprint and *-init.tcl files
Posted by Gustaf Neumann on 09/20/05 11:07 PM
the bootstrap-loader loads - somewhat simplified - during startup first the tcl/*-proc.tcl files, then the tcl/*-init.tcl files, and produces then apparently the blueprint (containing all tcl functions) used for initializing connection threads.

I have noticed the following problem: when i create a (disconnected) thread from a function invoked from a tcl/*-init.tcl file, this thread will be initialized with a blueprint not containing the functions from the tcl/*-proc.tcl files. Is this intended? Since i do this from xotcl, I could handle this situation in the thread initialization from xotcl by calling _ns_savenamespaces at an appropriate place.

Should i submit a bug-report concerning bootstrap-loader and apm-procs, or should i fix this in xotcl-core. Can any apm-procs guru comment on this?

Collapse
2: Re: blueprint and *-init.tcl files (response to 1)
Posted by Andrew Piskorski on 09/21/05 10:42 AM
Gustaf, AFAIK that behavior is neither intended nor desirable, it's a bug. (I think all the APM code pre-dates AOLserver 4.x, which might have something to do with the problem.)

Btw, in AOLserver 4.0.10 but without OpenACS, I do spawn detached threads from tcl/*.tcl files during AOLserver startup, and those detached threads do have all my custom Tcl procs defined. So whatever is going on with your problem, it sounds like it's probably specific to OpenACS.

A related problem is the "reload changed files" feature from the OpenACS APM. It can re-load changed Tcl procs, but it does so only in AOLserver connection threads - other threads, like the scheduler thread, see new procs only if you restart AOLserver. That's probably fixable by simply changing that feature to use ns_eval instead of its current and ancient method of hooking into the OpenACS request processor. That's been discussed before but AFAIK no one has actually tried it.

Collapse
3: Re: blueprint and *-init.tcl files (response to 1)
Posted by Gustaf Neumann on 09/22/05 02:35 PM
By looking into the code there seems to be a lot of magic going on in the loading process, some of this seems optimizable. Using ns_eval instead of the rp-hack won't be sufficient to change the tcl-code in already running (reused) connection threads (with maybe long threadtimeouts). one could certainly think about shutting all connection threads down, when the blueprint is changed, but this is another issue.

i am as well not very happy about the loading order of tcl/*-procs, as i would like to be able to define classes in one package and use these in another package, not worrying about the alphabetical order of the package names. Has anybody thought about using the package dependencies of apm for determining the loading order? within a package, one could add a flag in ad_library to override the alphabetical order...

For the time being, i have fixed the original problem within the xotcl thread support.

Collapse
4: Re: Re: blueprint and *-init.tcl files (response to 3)
Posted by Andrew Piskorski on 09/22/05 02:52 PM
Gustaf, why wouldn't ns_eval work for conn threads? I haven't actually tried it, but according to its docs, ns_eval exists specifically for such situations - "causes args to be sourced as a script in all Tcl interpreters".
Collapse
5: Re: blueprint and *-init.tcl files (response to 1)
Posted by Gustaf Neumann on 09/22/05 08:54 PM
Andrew, you are right. ns_eval does the right thing and queues the script for all interpreters, so idle threads are updated as well (i remember a verison of ns_eval, which was just updating the blueprint, my fault).

so apm_load_any_changed_libraries could be probably replaced by something like a thread checking the watched files and doing simply a ns_eval {source somefile}.

Collapse
6: Re: blueprint and *-init.tcl files (response to 1)
Posted by Malte Sussdorff on 09/23/05 11:53 AM
I have to admit that I'm unsure how to translate what has been said here into something we can fix in the APM, so Andrew and Gustaf, if you could take a minute (or more) to change this in OpenACS 5.2 or describe in more details what needs to be done so someone else can change it, this would be great.
Collapse
7: Re: blueprint and *-init.tcl files (response to 1)
Posted by Gustaf Neumann on 09/28/05 03:38 PM
I will try to paraphrase, allthough I do not feel as an expert on the request processor. apprently the following happens now:

At the begin of every request processing (in every connection thread) the following happens: 

      if ! rp_performance_mode ...
         apm_load_any_changed_libraries
apm_load_any_change_libraries does (simplified) the following:
  • foreach "watch", compare remembered dates with file dates
  • if changed, reload the files in the current thread
Certainly, scheduled procs are not directly affected by the reload. It is not ideal that the check is performed before the processing of the user request (users get the impression of slowness) and that it happens in every connection thread.

It would be certainly better to perform this checking in a scheduled sweeper process which does checking once and loads the files via "ns_eval [list source $path/$file]" which cares about updating the blueprints in all interpreters in the background.

Collapse
9: Re: Re: blueprint and *-init.tcl files (response to 7)
Posted by Andrew Piskorski on 09/28/05 07:04 PM
Gustaf, more importantly, if you use ns_eval instead, then all re-loading of procs should happen once when needed, and you should be able to remove that apm_load_any_changed_libraries hook entirely. No additional scheduled sweeper procs should be necessary, ns_eval should take care of it all.

The ns_eval comments say that all Tcl interps will sync with the new proc state "on their next _ns_atalloc", but I'm not sure what that means. So depending on precisely how ns_eval works, it may be desirable, in the the request processor, to replace the current apm_load_any_changed_libraries hook with some code that insures that the current thread has processed any and all pending ns_eval updates, by triggering one of these _ns_atalloc events or something like that. And since that's probably very fast, it should always run, regardless of whether rp_performance_mode is true or not. (And if necessary a similar light-weight hook could probably be installed somehow for all non-conn Tcl threads as well.)

Basically, it looks like apm_load_any_changed_libraries is just an old ugly hack which only exists because either its author wasn't aware of ns_eval, or ns_eval wasn't available way back then.

Collapse
8: Re: blueprint and *-init.tcl files (response to 1)
Posted by Dave Bauer on 09/28/05 03:50 PM
If ns_eval reloads the procedures in scheduled proc interpreter we should switch to using it.

I think in most cases rp_performance_mode_p will be true on an production system and we would skip that step.

The reason it seems to run on every request is for developer convenience. With a scheduled procedure, a reload might not happen immediately. Of course there is a way to force a reload by the developer instead of using the "watch" feature.

Collapse
10: Re: blueprint and *-init.tcl files (response to 1)
Posted by Gustaf Neumann on 09/28/05 11:07 PM
Gustaf, more importantly, if you use ns_eval instead, then all re-loading of procs should happen once when needed, and you should be able to remove that apm_load_any_changed_libraries hook entirely. No additional scheduled sweeper procs should be necessary, ns_eval should take care of it all.

For reloading, you are right. For watching files, where you want to have the interp updated as soon the file is changed on the disk, you need something like a scheduled procedure to compare dates. If the sweeper checks all two seconds, that should be good enough. in any case, only one ns_eval "source ..." is needed.

Collapse
11: Re: Re: blueprint and *-init.tcl files (response to 10)
Posted by Andrew Piskorski on 09/30/05 05:16 AM
Yes, you're right of course - I forgot about "watching" files, because I never use that feature.