Forum OpenACS Development: Re: Usage of XoTCL objects vs. database queries

There are as always various questions involved on caching and persistance layers, which are independent of XOTcl:
- do all objects fit into memory (or fit into the memory of every thread)?
- how is the cache coherency problem solved? If the objects do not change, everything is simple. If there are multiple caches (e.g. multiple connection threads, or one level higer: a cluster configuration), how is the invalidation protocol defined, in case of aolsever: how is the notification between threads implemented. the easier approach is to use nsvs (not very fast) or a thread as object pool interfacing the database. if every access to the objects data goes over a single interface, this is doable in various ways, if the object's data is changed via direct database operations, it becomes an issue to inform the affected objects.

The speed of access to the objects in memory depends certainly on your search operations. The access over the object name is very fast, sequential search is certainly slower (but often fast enough). Finally, nothing prohibits the developer to setup and maintain an index via hash a hash table (e.g. associative array or maybe a dict in tcl 8.5).

I'm afraid but I get the impression that you confuse some matters here ...

tread-safety vs. resource sharing in concurrency modes

my assumption was that the XoTCL Objects are thread safe within AOLserver.

XOTcl _as C extension_ is perfectly thread-safe in the sense of being deployed in thread-enabled tcl interpreters. What you refer to is the ability of sharing XOTcl objects as resources under concurrency, e.g. between several threads. And this is not possible, though this is more related to underlying Tcl than XOTcl. in the context of the tclthread extension, there is resource sharing under certain limitations (shared thread vars + shared channels, for instance). The issue here is that you cannot share commands and procs, as far as I know due to the current setup of 1-interpreter-per-thread. This also applies to XOTcl objects which certainly built upon the same low-level infrastructure as Tcl procs and commands. Currently, there is no BLACKBOARD kind of architecture in a scripted multi-threaded tcl applications realisable. Zoran, however, also discusses a BLACKBOARD-like architecture based on naviserver over at naviserver-devel (can't find the link right now).

Future Tcl will come with with another concurrency technique called coroutines that will allow concurrency programming based on a model that is not that strict in terms of resource sharing.

I hoped there was an easy way to share an object among all threads and all clusters and that an object

as far as i know, you experimented with streaming/ serialising objects into ns_cache (another thread-shared space), didn't you?

Would ttrace help here (http://wiki.tcl.tk/2770) ? Probably not, because it still requires me to make the object known to the thread and creating a resource copy in that thread.

You already hinted it, ttrace addresses a different issue. ttrace does not realise sharing of commands, procs (and objects), it solves the problem of the eager initialisation of interpreters. Based on Tcl's unknown mechanism, it sources procs on demand. This helps to address the issue of AOLServer/ Naviservers extensive blueprinting need ... unfortunately, it provides challenges to tcl's (and xotcl's) introspection mechanism which assume an entirely initialised surrounding ...

But it's not the same as using the same object copy in memory, as each thread has it's own copy of the ns_cache object and I can manipulate each of them independently.

Indeed, you don't have a shared state, compartimentalised into objects, available. If you try to achieve that, you may choose a different strategy: Realise object representations by ns_set et al. ... XOTcl slots provide an elegant way of introducing such an representation strategy, instead of accessing per-object state (in the strict sense), the slot-provided accessor/mutator methods write to and read from ns_sets ... Well, this implies other remedies and might not be scalabale open-end, but may be a valid optimisation strategy ... Objects residing in each thread are then mere PROXIES to a single shared state, under a single, db-provided identity.

Malte, of course Tcl has such a place, it's called "nsv" (in AOLserver; in the Tcl Threads Extension the same thing is called "tsv").

Of course, nsv/tsv supports only base Tcl types, strings and in some cases more complicated values like lists (with Zoran's clever perforamance hacks to avoid type shimmering). Can all the XOTcl objects you care about be serialized to from nsv/tsv? Efficiently? Between the two of them, Gustaf and Zoran should be able to tell you for sure.

Another option is to dedicate one single thread (and its associated Tcl interpretor) as your "XOTcl db" thread. Then have other threads send it messages with requests to read/write data. Everything will be serialized through that single "db" thread of course. In theory that could be much less scalable than the hash bucketized multi-threaded design of nsv/tsv, but in practice it might be simpler and faster, especially if the XOTcl objects are large but the read/write requests are small.