Forum OpenACS Q&A: Re: Lazy site node caching

15: Re: Lazy site node caching (response to 1)

Posted by Malte Sussdorff on 04/04/07 07:59 PM

I realized the hard way that the code has a couple of issues left, mainly with the number of times it is hitting the database and the number of times the side_node cache is updated for a specific node_id.

The main problem here is that from the URL you cannot detect if you are looking at a site_node or a folder within a package.

So, if you go from top to bottom, I already implemented a check for "has_children_p". Sadly this does not help if you work with multiple subsites e.g /malte. /malte/register would look at /malte, see it has children (e.g. /malte/photo), therefore try to load /malte/register from the database, which will obviously fail because /register is a directory, not a site_node.

Next idea is to try and load the children of a site_node not mounted under / and /dotlrn/, /dotlrn/clubs, /dotlrn/communities (you get the picture). Then I could say "children_loaded_p" and if I come from the top and hit a "children_loaded_p" in the site_node cache then I assume that all children have been loaded and go the usual way of finding a site_node, which assumes that it is in the site_node cache.

Only if this fails I would try to reload the site_node cache for the last node (from bottom to top) that says "children_loaded_p". This is due to the reason that you might want to mount a new package under an already loaded site_node, which would then not be found by the cluster.

ANy other idea is highly welcome though, but I am out of ideas for how to make good assumption if I need to hit the database for a URL like /malte/forums/moderate/edit or not.

16: Re: Lazy site node caching (response to 15)

Posted by Malte Sussdorff on 05/27/07 01:47 PM

I created a new version of lazy site node caching which roughly follows what I have been describing. Remaining issues are though:

a) Unnecessary hits to the database. Due to the fact that files can reside under /packages/acs-subsite/www and be mounted under that package, we always hit the database unless we would announce for the / that we loaded all children. Which excactly is not the purpose of lazy caching. So the occasional hit to the DB is in there. Same is true for directories under /www.

To solve this issue I have come up with a list of reserved names under / which will not trigger a trip to the database. At the moment these are the usual suspects "resources images image x o file admin". As this is a list of names you are not allowed to use when mounting a package, maybe we can get this list complete and also make sure that noone comes with the idea of mounting a package with that name in the first place?

b) Much more testing needed. I have it now on a busy production site, but this is using acs-subsite a lot and does not make use of dotlrn. So maybe someone with a busy .LRN site could test it again? Also I'm not sure if all the issues reported by Gustaf have been fixed.

17: Re: Res: Re: Lazy site node caching (response to 16)

Posted by Jose Agustin Lopez Bueno on 05/30/07 09:42 AM

Hi, Malte.

We are testing a bit your code.
Only we have detected some problems:

-The images in
http://www.xx.xx/dotlrn/configure
for up, down, delete, ... are not shown
until we refresh the cache with
http://www.xx.xx/acs-admin/cache

Same behaviour with urls
portal/admin/portal-config?portal_id=xxx
below
http://www.xx.xx/dotlrn/admin/portal-templates

-We are geting the next messages in the log:
[30/May/2007:09:38:51][23199.1106975072][-conn:20-] Notice: Loading from the database /dotlrn/classes/c032/12254/c07c032a12254gH c07c032a12254gH 4890528
[30/May/2007:09:38:54][23199.1106975072][-conn:20-] Error: Tcl exception:
can't read "node(url)": no such element in array
while executing
"string equal $node(url) "[ad_conn url]/""
(procedure "rp_filter" line 92)
invoked from within
"rp_filter preauth"
...

Regards,
Agustin

18: Re: Res: Re: Lazy site node caching (response to 17)

Posted by Malte Sussdorff on 05/31/07 09:49 AM

The first one is really strange. It should find the images no problem. What are the links to these images? And did you use the latest version from HEAD (as I initialize all non subsite packages)? Your error message does not indicate that 😊.

Furthermore, run the server in Debug mode. THis will help much more in getting to the root of the problem. As for the portal stuff, please find out where the code with "cn't read node(url)" is actually comming from, which procedure was called where that tried to find the node.

19: Re: Res: Re: Lazy site node caching (response to 17)

Posted by Emmanuelle Raffenne on 05/31/07 10:40 AM

Hi Agustin and Malte,

I suspect this problem comes from the site_nodes library that both dotlrn and new-portal implement (yes! it's duplicated in those 2 packages...). This should be removed and the calls to those procs replaced by calls to site-node ones (acs-tcl).

I've planned to work on this some time during the next 2 weeks. Not sure I will have time though.

20: Re: Res: Re: Lazy site node caching (response to 19)

Posted by Malte Sussdorff on 06/01/07 09:56 AM

Emma, thanks for this information. I guess it really is time for a cleanup 😊. If you have time that would be wonderful, especially as I don't have a larger .LRN installation on which to test this.

21: Re: Res: Re: Lazy site node caching (response to 20)

Posted by Emmanuelle Raffenne on 06/07/07 10:39 PM

Hi,

I've finished the cleanup of site_nodes and committed on HEAD. The duplicated libraries have been removed and calls to site_nodes::* replaced by calls to site_node::

I just did some smoke tests: dotlrn install, create communities, add applets, create users and add them to communities, etc. (where those calls were used) and everything seem to work correctly.

22: Re: Res: Re: Res: Re: Lazy site node caching (response to 21)

Posted by Jose Agustin Lopez Bueno on 06/08/07 10:13 AM

Hi, Emma!

Thanks for the work!

I have put the code in our development environment but I got
the next problem:

When I add one applet to one community appears Ok
but when you click the url of the new applet (in admin)
you get the error:

The requested URL was not found on this server.

When you restart Openacs, the url is working.

Regards,
Agustin

23: Re: Res: Re: Res: Re: Lazy site node caching (response to 22)

Posted by Emmanuelle Raffenne on 06/08/07 11:33 AM

Agustin,

Did you reinstall first? It won't fix what you describe but the dotlrn installation was broken because of site_nodes.

Anyway, I've noticed this behavior too. I was about to post about it after doing more tests.

On my side, I get a "file not found" when I go to the admin pages, but I don't need to restart aolserver, just to reload the page (I'm using firefox 2 on debian). It happens also on openacs when applications are added, so I guess there are still issues with caching.

Malte, any ideas?

24: Re: Lazy site node caching (response to 22)

Posted by Emmanuelle Raffenne on 06/08/07 01:31 PM

Agustin, Malte,

I've tested again. Here are the steps I followed:

- Standard install of dotlrn from HEAD
- created a new community
- went to calendar index page of the new created community: page displayed
- went to the admin index page of calendar in the new created community: File not found
- reload the page: page displayed
- went to the admin index page of forums in the new created community: File not found
- reload the page: page displayed

There are no error messages in logs.

Note: Same occurs in Openacs when I mount an application, users pages are correctly displayed but I get a "file not found" the first time I go to the admin page of the application.

After that, I've replaced acs-tcl/tcl/site-nodes-procs* with the last revision of oacs-5-3 branch and followed the same steps:

- created a new community
- went to calendar index page of the new created community: page displayed
- went to the admin index page of calendar in the new created community: page displayed
- went to the admin index page of forums in the new created community: page displayed

26: Re: Res: Re: Lazy site node caching (response to 17)

Posted by Torben Brosten on 06/30/07 10:01 AM

Hi Malte,

Just a heads up.

I tried using the notes package and got the "file not found" for all the user pages except the one at the mount point (which apparently redirects to /add).

notes package mounted at /notes-test1 from a fresh db using cvs head (June 19) and no parameter changes from defaults.

Swapping out the site-nodes-procs* from oacs-5-3 (and reloading via apm) fixed the problem.

cheers,

Torben

27: Re: Res: Re: Lazy site node caching (response to 26)

Posted by Torben Brosten on 06/30/07 12:26 PM

..well.. fixed the notes problem anyway.

After re-installing the db using the oacs-5-3 version of site-nodes-procs* (to rule out any residual side effects), I see this:

/api-doc/ file not found
/api-doc/index file not found
/api-doc/index? the usual page shows up.

but that probably doesn't help much for test cases with the current code.

Am trying again with current cvs, since there appears to be some recent changes in those files..

28: Re: Res: Re: Lazy site node caching (response to 27)

Posted by Torben Brosten on 06/30/07 12:43 PM

current version works.. Wonderful! and thank you =)

29: Re: Res: Re: Lazy site node caching (response to 28)

Posted by Emmanuelle Raffenne on 07/02/07 10:34 AM

Torben,

By current version, do you mean 5.3?

I have the "file not found" problem but with HEAD code, not with 5.3 one.

30: Re: Res: Re: Lazy site node caching (response to 29)

Posted by Torben Brosten on 07/03/07 09:16 AM

Sorry for the confusion, Emmanuelle.

Current (June 30th) version of head is working fine for me. I'm using aolserver 4.0.10, pg8.1 on OS X 10.4.10.

31: Re: Res: Re: Lazy site node caching (response to 29)

Posted by Malte Sussdorff on 07/03/07 09:43 AM

Though I need to work on a lot of things for the next release of OpenACS (acs-mail-lite being the primary culprit) and ]po[ (upgrade to latest OpenACS version), please post *any* issues you have with the HEAD version of site-node caching here (I am not subscribed to bug-tracker, sorry...). Just make sure you run off the latest version from HEAD.

32: Re: Lazy site node caching (response to 24)

Posted by Malte Sussdorff on 08/03/07 09:50 AM

The /admin bug is fixed now in HEAD at least in the instances where I tested it 😊.

33: Re: Lazy site node caching (response to 32)

Posted by Emmanuelle Raffenne on 08/06/07 09:44 AM

Thanks Malte. Seems to work fine now.

34: Re: Lazy site node caching (response to 32)

Posted by Gustaf Neumann on 09/17/07 12:14 PM

Malte,

can you look at Test site_node::update_cache
test/admin/testcase?testcase_id=site_node_update_cache&package_key=acs-tcl&view_by=testcase&category=&quiet=0
Two tests are failing in the CVS head version. Seems as still some updates are needed.

Many thanks
-gustaf