Forum OpenACS Q&A: Re: Lazy site node caching

Collapse
Posted by Malte Sussdorff on
I realized the hard way that the code has a couple of issues left, mainly with the number of times it is hitting the database and the number of times the side_node cache is updated for a specific node_id.

The main problem here is that from the URL you cannot detect if you are looking at a site_node or a folder within a package.

So, if you go from top to bottom, I already implemented a check for "has_children_p". Sadly this does not help if you work with multiple subsites e.g /malte. /malte/register would look at /malte, see it has children (e.g. /malte/photo), therefore try to load /malte/register from the database, which will obviously fail because /register is a directory, not a site_node.

Next idea is to try and load the children of a site_node not mounted under / and /dotlrn/, /dotlrn/clubs, /dotlrn/communities (you get the picture). Then I could say "children_loaded_p" and if I come from the top and hit a "children_loaded_p" in the site_node cache then I assume that all children have been loaded and go the usual way of finding a site_node, which assumes that it is in the site_node cache.

Only if this fails I would try to reload the site_node cache for the last node (from bottom to top) that says "children_loaded_p". This is due to the reason that you might want to mount a new package under an already loaded site_node, which would then not be found by the cluster.

ANy other idea is highly welcome though, but I am out of ideas for how to make good assumption if I need to hit the database for a URL like /malte/forums/moderate/edit or not.

Collapse
Posted by Malte Sussdorff on
I created a new version of lazy site node caching which roughly follows what I have been describing. Remaining issues are though:

a) Unnecessary hits to the database. Due to the fact that files can reside under /packages/acs-subsite/www and be mounted under that package, we always hit the database unless we would announce for the / that we loaded all children. Which excactly is not the purpose of lazy caching. So the occasional hit to the DB is in there. Same is true for directories under /www.

To solve this issue I have come up with a list of reserved names under / which will not trigger a trip to the database. At the moment these are the usual suspects "resources images image x o file admin". As this is a list of names you are not allowed to use when mounting a package, maybe we can get this list complete and also make sure that noone comes with the idea of mounting a package with that name in the first place?

b) Much more testing needed. I have it now on a busy production site, but this is using acs-subsite a lot and does not make use of dotlrn. So maybe someone with a busy .LRN site could test it again? Also I'm not sure if all the issues reported by Gustaf have been fixed.

Collapse
Posted by Jose Agustin Lopez Bueno on
Hi, Malte.

We are testing a bit your code.
Only we have detected some problems:

-The images in
http://www.xx.xx/dotlrn/configure
for up, down, delete, ... are not shown
until we refresh the cache with
http://www.xx.xx/acs-admin/cache

Same behaviour with urls
portal/admin/portal-config?portal_id=xxx
below
http://www.xx.xx/dotlrn/admin/portal-templates

-We are geting the next messages in the log:
[30/May/2007:09:38:51][23199.1106975072][-conn:20-] Notice: Loading from the database /dotlrn/classes/c032/12254/c07c032a12254gH c07c032a12254gH 4890528
[30/May/2007:09:38:54][23199.1106975072][-conn:20-] Error: Tcl exception:
can't read "node(url)": no such element in array
while executing
"string equal $node(url) "[ad_conn url]/""
(procedure "rp_filter" line 92)
invoked from within
"rp_filter preauth"
...

Regards,
Agustin

Collapse
Posted by Malte Sussdorff on
The first one is really strange. It should find the images no problem. What are the links to these images? And did you use the latest version from HEAD (as I initialize all non subsite packages)? Your error message does not indicate that 😊.

Furthermore, run the server in Debug mode. THis will help much more in getting to the root of the problem. As for the portal stuff, please find out where the code with "cn't read node(url)" is actually comming from, which procedure was called where that tried to find the node.

Collapse
Posted by Emmanuelle Raffenne on
Hi Agustin and Malte,

I suspect this problem comes from the site_nodes library that both dotlrn and new-portal implement (yes! it's duplicated in those 2 packages...). This should be removed and the calls to those procs replaced by calls to site-node ones (acs-tcl).

I've planned to work on this some time during the next 2 weeks. Not sure I will have time though.

Collapse
Posted by Malte Sussdorff on
Emma, thanks for this information. I guess it really is time for a cleanup 😊. If you have time that would be wonderful, especially as I don't have a larger .LRN installation on which to test this.
Collapse
Posted by Emmanuelle Raffenne on
Hi,

I've finished the cleanup of site_nodes and committed on HEAD. The duplicated libraries have been removed and calls to site_nodes::* replaced by calls to site_node::

I just did some smoke tests: dotlrn install, create communities, add applets, create users and add them to communities, etc. (where those calls were used) and everything seem to work correctly.

Hi, Emma!

Thanks for the work!

I have put the code in our development environment but I got
the next problem:

When I add one applet to one community appears Ok
but when you click the url of the new applet (in admin)
you get the error:

The requested URL was not found on this server.

When you restart Openacs, the url is working.

Regards,
Agustin

Collapse
Posted by Emmanuelle Raffenne on
Agustin,

Did you reinstall first? It won't fix what you describe but the dotlrn installation was broken because of site_nodes.

Anyway, I've noticed this behavior too. I was about to post about it after doing more tests.

On my side, I get a "file not found" when I go to the admin pages, but I don't need to restart aolserver, just to reload the page (I'm using firefox 2 on debian). It happens also on openacs when applications are added, so I guess there are still issues with caching.

Malte, any ideas?

Collapse
Posted by Emmanuelle Raffenne on
Agustin, Malte,

I've tested again. Here are the steps I followed:

- Standard install of dotlrn from HEAD
- created a new community
- went to calendar index page of the new created community: page displayed
- went to the admin index page of calendar in the new created community: File not found
- reload the page: page displayed
- went to the admin index page of forums in the new created community: File not found
- reload the page: page displayed

There are no error messages in logs.

Note: Same occurs in Openacs when I mount an application, users pages are correctly displayed but I get a "file not found" the first time I go to the admin page of the application.

After that, I've replaced acs-tcl/tcl/site-nodes-procs* with the last revision of oacs-5-3 branch and followed the same steps:

- created a new community
- went to calendar index page of the new created community: page displayed
- went to the admin index page of calendar in the new created community: page displayed
- went to the admin index page of forums in the new created community: page displayed

Collapse
Posted by Torben Brosten on
Hi Malte,

Just a heads up.

I tried using the notes package and got the "file not found" for all the user pages except the one at the mount point (which apparently redirects to /add).

notes package mounted at /notes-test1 from a fresh db using cvs head (June 19) and no parameter changes from defaults.

Swapping out the site-nodes-procs* from oacs-5-3 (and reloading via apm) fixed the problem.

cheers,

Torben

Collapse
Posted by Torben Brosten on
..well.. fixed the notes problem anyway.

After re-installing the db using the oacs-5-3 version of site-nodes-procs* (to rule out any residual side effects), I see this:

/api-doc/ file not found
/api-doc/index file not found
/api-doc/index? the usual page shows up.

but that probably doesn't help much for test cases with the current code.

Am trying again with current cvs, since there appears to be some recent changes in those files..

Collapse
Posted by Torben Brosten on
current version works.. Wonderful! and thank you =)
Collapse
Posted by Emmanuelle Raffenne on
Torben,

By current version, do you mean 5.3?

I have the "file not found" problem but with HEAD code, not with 5.3 one.

Collapse
Posted by Torben Brosten on
Sorry for the confusion, Emmanuelle.

Current (June 30th) version of head is working fine for me. I'm using aolserver 4.0.10, pg8.1 on OS X 10.4.10.

Collapse
Posted by Malte Sussdorff on
Though I need to work on a lot of things for the next release of OpenACS (acs-mail-lite being the primary culprit) and ]po[ (upgrade to latest OpenACS version), please post *any* issues you have with the HEAD version of site-node caching here (I am not subscribed to bug-tracker, sorry...). Just make sure you run off the latest version from HEAD.
Collapse
Posted by Malte Sussdorff on
The /admin bug is fixed now in HEAD at least in the instances where I tested it 😊.
Collapse
Posted by Emmanuelle Raffenne on
Thanks Malte. Seems to work fine now.
Collapse
Posted by Gustaf Neumann on
Malte,

can you look at Test site_node::update_cache
test/admin/testcase?testcase_id=site_node_update_cache&package_key=acs-tcl&view_by=testcase&category=&quiet=0
Two tests are failing in the CVS head version. Seems as still some updates are needed.

Many thanks
-gustaf