Forum OpenACS Development: Can we use namespace variables to avoid upvar?

In acs-content-repository/tcl/filter-procs.tcl there are several procedures that create a namespace variable, and then other procedures in that namespace share the value of that variable. It doesn't have to be passed around or upvar'ed.

It is possible to export a namespace variable and import it into another procedure. I am not the expert on tcl namespaces, but it seems like it would be a good idea to investigate using namespace variables more in the toolkit. It is much clearer what is happening, and it is much more standard Tcl style.

ACS/OpenACS started on Tcl 7 which did not support namespaces. Now that we require a version of AOLserver that used Tcl 8.x it could help clean up the code to use namespaces more.

Collapse
Posted by Tom Jackson on

Yes you can do it, obviously a lot of time could be spent in this area which would benefit code simplification. The problem is the code most in need of simplifying is difficult to understand! It may prove to be too tedious, but all new work should consider the benefits.

Collapse
Posted by Jon Griffin on
There are some issues regarding AOLserver and namespaces. Mainly, the way vars are created and not deleted. Things like tcllib are basically impossible to use as it uses namespace vars and AOL doesn't destroy them.
Of course, namespaces are no problem in many contexts.
Collapse
Posted by Tilmann Singer on
Jon, are you saying that if I create and set a namespace variable like this within a tcl script on aolserver:

namespace eval ::mylib {
  variable some_var
  set some_var foo
}

that ::mylib::some_var will persist over connections and not be deleted after the end of the connection of the thread's interpreter such as other vars would be?

Do you know in particular if that is true for aolserver 4? That would be quite a serious bug I think.

Collapse
Posted by Dave Bauer on
Jon,

As far as I know this namespace cleanup has been fixed for quite a while. Maybe it is not in AOLserver 3.3+ad13, but it should be fixed in the newer AOLserver  versions.

I'll double check this information to be sure.

Collapse
Posted by Jon Griffin on
Dave,
That is not true, I had a long exchange with the AOLServer team about this and the consensus was that they don't really see a need to initialize reused threads.
SO yes the problem is still there. This only affects vars not namespaces in general.
Still a big minus in that we could get rid of passing vars explicitly if it was working correctly. And, yes tcllib will not work correctly either.
Collapse
Posted by Tilmann Singer on
Indeed, the following script yields 'variable exists' after a few reloads on aolserver 4:

set s ""
if { [info exists ::mytest::some_var] } {
    set s "VARIABLE EXISTS!"
}
namespace eval ::mytest {
    variable some_var
    set some_var "foo"
}
append s "some_var: $::mytest::some_var"
doc_return 200 text/plain $s

Jon, was that long exchange you mention on the aolserver list?

Collapse
Posted by Jon Griffin on
I remember at least starting the conversation on the list. It was probably all there.
It was maybe 9 months ago???
Collapse
Posted by Andrew Piskorski on
There is no conceivable way that not cleaning up namespace and global variables before the connection thread gets re-used could ever be considered a feature, it is just faster and simpler (and is the way AOLserver has worked all along). The decision there is simply one of standard Tcl namespace semantics vs. the existing simpler faster AOLserver code.

Therefore, if someone implements namespace cleanup in a manner which is clear, efficient, and optional, I can't imagine why the AOLserver maintainers wouldn't accept it. AFAIK the problem is that no one has attempted to implement it.

Collapse
Posted by Alfred Werner on
Does the ttrace package resolve any of this? The README in the aolserver directory seems to indicate that it's designed to optimize thread initialization.
Collapse
Posted by Andrew Piskorski on
Alfred, I don't think Zoran intended ttrace to address this per-thread namespace cleanup issue at all. In fact I think what it does is sort of exactly opposite. If I remember right, ttrace hooks into Tcl's 'uknown' command to find procedure definitions and things which don't exist yet in the current thread. The benefit of that is it lets you greatly decrease AOLserver server and thread startup times in some cases, and maybe it can tend to reduce the memory footprint too (I don't remember).

On the other hand, What Jon G. and others would like here, is some sort of removal of left over global and namespace variables from the thread when each connection request completes its processing.

It wouldn't hurt to ask Zoran about it though, he's likely to have some ideas.

Collapse
Posted by Jun Yamog on
Hi,

So this means that we can't still use namespace vars to transfer values?  It has been confirmed by Tils that it still not cleans up the vars.

Collapse
Posted by Dave Bauer on
ttrace does allow definiton of cleanup callback procedures, so it is possible it makes it easier to support namespace var cleanup without hacking aolserver.
Collapse
Posted by Tom Jackson on

What is wrong with simply initializing the variable? I guess the question is when and how. Unforutnately ACS, and now OpenACS decided to hog the first preauth filter position with the rp_filter (now rp_resource_filter). It would be nice to move both filters to the postauth position, after authorization takes place, although the resource filter could maybe run in preauth, since it doesn't require auth.

Then packages could run initilization code in preauth, and use trace filters to handle cleanup. You might also have the opportunity to load new packages which protect the site and that should run before database access gets involved (possibly aborting the connection). Right now the only way to do this is to hack in a filter into the private init.tcl file.

Collapse
Posted by Tilmann Singer on
The problem is that in the package initalization code the variables can potentially get set to default values, such as in http://www.wjduquette.com/tcl/namespaces.html under "Rule 9". This would only happen on the _first_ call of package require, not on subsequent ones when the thread has already loaded the package. There seems to be no way for a general mechanism in aolserver or openacs to find out _how_ these variables should get reset.

Apparently the only solution is to be careful when using libraries with aolserver that contain state information and manually clean up by calling custom reset procs if provided by the package or resetting variables manually (ugly) before using a tcl package in a new connection.

Collapse
Posted by Tom Jackson on

In AOLserver there is a convenient proc ns_atclose which might be used to do what you want. Here is an example of it being used with namespace vars:


namespace eval ::twt::db::transaction {

    variable initialized 0
    variable ids
 
    namespace import ::twt::log::*
}


proc ::twt::db::transaction::init { } {

    ns_atclose ::twt::db::transaction::reset
    
    variable initialized 1
    variable ids
    array unset ids

}

proc ::twt::db::transaction::reset { } {

    # This runs after the connection closes

    variable initialized
    variable ids

    # Need to rollback any open transactions
    foreach id [array names ids] {
	
	set transactionInfo $ids($id)
	set provider [lindex $transactionInfo 0]
	set transactionRollbackProc [set ::twt::db::${provider}::transactionRollbackProc]
	$transactionRollbackProc $id
    }
    
    array unset ids
    set initialized 0
}

The namespace vars are reset for each connection. Ns_atclose is guaranteed to run even if the connection returns an error, it also runs at the end of a scheduled procedure, if the code were to be used there.

Collapse
Posted by Tilmann Singer on
Yes, ns_atclose might solve the 'when' issue, but not the 'what', which is unsolvable in a general way - for every custom library something different needs to be done. E.g. when using the tcl http package and using ::http::config -useragent to set the user agent variable, there seems to be no other way to reset it to the default value than manually. That's simply something to be aware of when using package require.
Collapse
Posted by Andrew Piskorski on
So, naive question, but what the heck does tclsh do exactly for "namespace cleanup"? If Tcl packages are leaving all sorts of bogus state hanging around in namespace variables, what magic is tclsh invoking to automatically clean up after them?
Collapse
Posted by Tom Jackson on

Andrew, I'm guessing: exit.

Tillman, I think my example solves the 'what' in a very general way. Each procedure which could be called by an application needs to make sure the variables have been initialized, which happens once per request. At the end, the callback 'reset' does more than just unset some variables. It handles all cleanup for the namespace. The namespace allows you to easily manage this process. Some namespace variables you might actually like to hang around across requests, so there is no 'blind' solution to this issue, you may not want defaults restored in every case.

Collapse
Posted by Ivan Lazarte on
It is highly disappointing that namespace variable persistence isn't seen as an issue, as importing tcl extensions (like the above mentioned tcllib) practically becomes impossible.

Watching the logs, I've sort of put together that it falls like this : many requests fall under one connection, and one connection ends the life of a namespace variable.

I guess it has to do with the child interpreter that aolserver uses?

Why hasn't anyone addressed this before? or for Aolserver 4x ?

Collapse
21: Tcl namespace cleanup (response to 1)
Posted by Andrew Piskorski on
I haven't carefully investigated any of these issues, but reading this thread again, the AOLserver "namespace cleanup" issue seems to fundamentally be a Tcl package or library problem, not an AOLserver problem.

Tcl apparently provides no consistent method or convention for initializing and re-setting package state, other than completely destroying the Tcl interpreter and then creating a new one. So this is going to be a problem for any long lived Tcl interpreter.

What do the Tcl maintainers think about this? What other projects are using long-lived server-like Tcl interpreters, and how have they addressed this problem?

Collapse
Posted by Dave Bauer on
Andrew,

The reason why namespace vars are an issue in aolserver is that the interpreters are resued. The work pretty much like globals. AOLserver has a way to clean up globals as the end of a request.

Its not really a problem, and I think most existing Tcl packages do something itelligent with namespace vars. It is just something to be aware of when using the variables inside AOLserver.