Forum OpenACS Development: xotcl object cache behavior?

Collapse
Posted by Stan Kaufman on
I'm trying to understand how xotcl/xowiki work, and here's some behavior that I can't tell if it's expected, a bug, or what.

Using the latest (xotcl 0.42; xowiki 0.35), mount a new xowiki instance. Navigate to the index page, and you'll see "This is the default start page of XoWiki...etc etc" along with three portlets (Recently Changed, Last Visited, Most Popular).

Note the xowiki's root object id aka content_folder (11375 in my case -- its Page Title is ::1375 and is the only page you see now other than "index"). Find the package_id for this xowiki instance (11367 in my case).

Now open a Shell (using DS) and enter the following:

set package_id 11367
set xowiki_folder [::xowiki::Page require_folder -package_id $package_id -name "xowiki"]
set o [::xowiki::Object instantiate -item_id $xowiki_folder]
$o info class

The Shell will output "::xowiki::Object" as you expect. Why would I want code that does nothing more than that? Well, I don't really; what I ultimately want to do is to programmatically grab the root object (from the xowiki instance package_id) and then edit the text of that xowiki root object and save it. But in trying to figure out why I can't seem to do that, I've found that I get stuck even at this step of getting the root object and confirming that I've got it.

So here's the weird part. After running that code once in the Shell, go back to the index page of the xowiki instance and refresh the page. Probably each of the portlets will show an error from the adp: "can't read "package_id": no such variable". Now refresh the page again -- once or several times, as many as it takes for the portlets to display correctly again. Sometimes it takes only a single refresh, sometimes more. Keep refreshing; every few times the index page will show errors and then it won't -- without any other actions besides refreshing the page.

Now go to the Admin page where you'll see a list of the objects in the xowiki instance -- only 2 for now: an ::xowiki::Page and our ::xowiki::Object. Click on the root Object instance and you'll see a server error because package_id can't be read. Backup in the browser and click the instance again. Now there will be no server error and you'll see the "Index of XoWiki Objects" page with our root object in the list. Now backup in the browser again and click the instance again; the server error will return. Repeat as often as you like; every other trip to the page will return a server error.

Clearly something is going on behind the scenes with this object -- but what? Somehow it periodically forgets and then remembers its package_id (and maybe other things?). I'm guessing this has to do with the ns_cache xotcl_object_cache, since various instprocs flush it when saving objects. But the code I ran in the Shell doesn't save any object; it just retrieves some info.

Any ideas what's going on? Gustaf, can you explain a bit about how we use the object cache, and when we need to flush it, etc? Or is this due to something else? Thanks in advance for any explanations that can help us know how to understand and use xotcl!

Collapse
Posted by Gustaf Neumann on
Just a short answer, it is quite late here already, and i have to get up early...

you are creating an object without an package_id (it is not fetched from the database, but via the environment, where the object exists). The object, you are creating is kept in the interpreter thread (you do not use volatile). it is possible to run e.g. require_folder_object several times; in order to avoid multiple creations of the same object in one session, the code only creates an object, when it is not there. Since you have created a broken object, you are in bad shape. since the broken object exists in only one of possibly many connection threads, and you use most probably for different queries different connection threads, you get
the strange behavior. This has very little to do with the caching.

there is another tricky part about the folder object in xowiki: since folders do not have revisions and do not allow subclassing, xowiki creates an ::xowiki::Object with some item_id and the **name** containing the folder_id. this can easily lead to confusions.

assume, you have a folder 16059, you might have
a folder object with the id 16068 and the name ::16059
The next two commands do create the object and show it to
you (you can run this from the shell).

::xowiki::Package instantiate_page_from_id -item_id 16068
::Serializer deepSerialize ::16068

the next three commands make the object volatile,
append some text to the object and save it

::16068 volatile
::16068 append title "... kilroy was here"
::16068 save

Be aware that i am actively changing the code, since
i am trying under the hood various approaches to improve modularity and to remove redundancies. in about one week i will go on holiday, then there will be a more stable period of the code...

Collapse
Posted by Stan Kaufman on
Thanks for your reply, Gustaf. I see now that ::Generic::CrItem instantiate doesn't actually load the object from the database, though the docs suggest it does: "returns object containing the attributes of the CrItem". So with a broken object, weirdness inevitably would follow.

I don't want to delay the active work you're doing on these packages -- and much appreciated work, too -- so maybe I should just delay my explorations until a later date.

Still, permit me a few more questions. I am having problems with your suggested test code. I have an xowiki instance with id 11385. Its root content_folder is id 11393, and the content_folder's ::xowiki::Object is id 11395 (and its name is ::11393). When I try what you suggest, here's what happens in the DS Shell:

set pkg_id 11385
set fldr_id 11393
set obj_id 11395
::xowiki::Package instantiate_page_from_id -item_id $obj_id
::Serializer deepSerialize ::$obj_id

returns this error:

ERROR:
Query did not return any rows.
    while executing
"db_1row get_class "select content_type as object_type from cr_items where item_id=$item_id""
    invoked from within
"ns_cache eval xotcl_object_type_cache  [expr {$item_id ? $item_id : $revision_id}] {
      if {$item_id} {
	db_1row get_class "select content_type as ..."
    (procedure "instantiate" line 4)
    ::Generic::CrItem->instantiate
    invoked from within
"::Generic::CrItem instantiate -item_id $item_id -revision_id $revision_id"
    (procedure "instantiate_page_from_id" line 4)
    ::xowiki::Package->instantiate_page_from_id
    invoked from within
"::xowiki::Package instantiate_page_from_id -item_id $obj_id"
    ("uplevel" body line 9)
    invoked from within
"uplevel 1 $script"

Thinking perhaps you meant to pass the folder -- not the folder object -- to ::xowiki::Package instantiate:

set pkg_id 11385
set fldr_id 11393
set obj_id 11395
::xowiki::Package instantiate_page_from_id -item_id $fldr_id
::Serializer deepSerialize ::$fldr_id

reveals that presumably that's not what you meant:

ERROR:
invalid command name "content_folder"
    while executing
"$object_type instantiate -item_id $item_id -revision_id $revision_id"
    (procedure "instantiate" line 13)
    ::Generic::CrItem->instantiate
    invoked from within
"::Generic::CrItem instantiate -item_id $item_id -revision_id $revision_id"
    (procedure "instantiate_page_from_id" line 4)
    ::xowiki::Package->instantiate_page_from_id
    invoked from within
"::xowiki::Package instantiate_page_from_id -item_id $fldr_id
"
    ("uplevel" body line 9)
    invoked from within
"uplevel 1 $script"

I see that ::xowiki::Package instantiate_page_from_id calls ::Generic::CrItem instantiate, sets the package_id and folder_id, and does some other stuff -- but doesn't appear to get database data in the way that "select * from xowiki_objectx where object_id = $object_id" would. Probably I'm still misunderstanding something basic here...

Anyway, thanks for any pointers. I'm trying to grok "The Xotcl Way" since I have several packages developed for OpenACS 3.x that am porting to 5.x and want to build with xotcl.

Collapse
Posted by Gustaf Neumann on
Stan, are you sure that 11395 is the item_id and not the revision_id? it can be easily confused, since the the entries in the class specific extension tables (such as xowiki_page or xowiki_object) are revision_ids and name page_id or object_id.

::Generic::CrItem instantiate loads the object from the database (or cache), bat this is only a partial initialization . btw, in the next version i'll get the package_id as well this way... i wanted to keep it mostly functioning with 5.1, but one should give user reasons to upgrade anyway...

Collapse
Posted by Stan Kaufman on
Aha!! It *was* the revision_id. I mistakenly looked at the object_id -- and not the item_id -- in the xowiki_objectx view -- which was particularly stupid since the view provides a column named "item_id". Using the correct item_id now works without error.

Thanks again for your helpful explanations and ingenious code!

Collapse
Posted by Stan Kaufman on
One last question on this: is there an API to find this root folder object's id -- which is the item_id as you pointed out? In the following snippet:

set package_id 11385
set xowiki_folder [::xowiki::Page require_folder -package_id $package_id -name "xowiki"]
set obj_id [db_string obi "select distinct item_id from xowiki_objectx where object_package_id = $package_id"]
set o [::xowiki::Package instantiate_page_from_id -item_id $obj_id]
$o volatile
$o set title "My New Title"
$o save

obj_id is found via the "select distinct" query -- which seems to me like a hack, but at that point we don't have a real object to introspect. Seems like a ::xowiki::Package proc get_root_folder_id {package_id} method would be useful -- or maybe it's already there by another name and I'm too dim to see. Many thanks, Gustaf, as always!

Collapse
Posted by Stan Kaufman on
Ooops, I meant a ::xowiki::Package proc get_root_folder_object_id {package_id} method -- ie one that returns the id of the root folder's associated object.
Collapse
Posted by Gustaf Neumann on
Stan, there are currently two approaches implemented to determine the folder_id (needed in different situations):
  1. In oder to determine the (root) folder_id from a package (e.g. in order to fetch xowiki pages from this folder)

    set Package [::xowiki::Package create ::$package_id]
    set folder_id [$Package folder_id]

  2. In order to determine the folder_id from a page:

    set folder_id [$page parent_id]
As said before, this describes the situatiton in the current release 0.34, it might change in the future (when e.g. subfolders are introduced, etc). i am currently working to make it easier to get the connection context (setting up package-id, user-id etc for e.g. testing) there will be calls to take care of these issues as well and setup the package objects etc. automatically). if someone reads this posting later, things might have changed.

It took me yesterday a couple of hours to find a bug where i had several code pieces with ::namespace:cmd (note the missing second colon), so i need some break or better glasses. i am currently finishing a ~1000 page book, and i am doing the xowiki stuff mostly as a recreational activity. On the 18th i am going on vacation, until then i try to finish up the pending things.

best regards
-gustaf

Collapse
Posted by Stan Kaufman on
Gustaf, those steps return the folder_id -- but not the id of the folder's associated object -- which is what is needed to instantiate the object itself, as you explained earlier:

assume, you have a folder 16059, you might have
a folder object with the id 16068 and the name ::16059
The next two commands do create the object and show it to
you (you can run this from the shell).

::xowiki::Package instantiate_page_from_id -item_id 16068
::Serializer deepSerialize ::16068

Is there somewhere in the api that returns the id "16068" in your example? In any case, I see how to work around this for now, and I realize that you're actively working on all this. I really appreciate all your ongoing contributions! You are amazingly productive!

By the way, is your book entitled "Using XOTCL with OpenACS"? That would be a most excellent title! 😊

Collapse
Posted by Gustaf Neumann on
sorry, i thought the question was to obtain the folder id, i misread your posting. There is not a big need for an api, since once a package is instantiated, the folder_object is as well. you just need the package_id to get all you want. here is, how you get the interesting ids:

set P [::xowiki::Package create ::$package_id]
set folder ::[$P folder_id]
set folder_id [$folder set folder_id]
set folder_item_id [$folder set item_id]
set folder_revision_id [$folder set revision_id]

btw, in the next release, you can do

$folder serialize

instead of the Serializer::...

The new code is stabilizing, and has some cool features. For example, i added some code to instantiate values from an sql query into an object. normally you get the return variables into the current scope of a function. In order to implement this, only one line was needed

::xotcl::Object instforward db_1row -objscope

db_1row is a forwarder applicable to all objects that makes the instance variables of an object to appear as local variables (the flag -objscope). So, if you do a

$obj db_1row query_name "select some_atts from ..."

the return variables are set as instance variables of the object. This is quite important for dynamic sql queries, where there is no control over the output variables, where it is quite easy to overwrite local variables. the standard set of db-commands allows already array output, but for oo style of code, the above variant is much more convenient and saves copying of values...

Collapse
Posted by Stan Kaufman on
Gustaf, this is *brilliant*! Thanks again!