Forum .LRN Q&A: Speed of dotLRN portal and applets

Collapse
Posted by Peter Alberer on

I would like to know about other peoples experiences with the speed of the dotLRN portal system.

In our system where we have a few objects in the content repository already (9000) in total 30000 acs_objects, especially the portal pages seem to be rather slow. Of course this is due to the fact that the underlying packages are using some complex queries. With very low load on the system the pages take quite some time to load (measured time to render the portal page in tcl file): (no changes made to the pages)

  • Main Page 1sec
  • Calender Page 600ms
  • File Storage Page 9sec
Especially the fs-portlet is nearly unusable (the main query "select_folders" takes more than 8 seconds).

The machine is a dual P4 Xeon 1,7Ghz with 4GB Ram, using postgres 7.2. Vacuumdb is running every night. Could this be some kind of misconfiguration or are others seeing similar performance problems?

Many TIA
Collapse
Posted by Caroline Meeks on
Sloan has done a number of performance fixes and I don't know how many of them have made it back into the core and been translated into postgres.

The biggest performance boost is from caching permission_p.  You will only see this improvement on reload, but on a system full of users it makes a huge difference. Is that in your version of dotLRN?

On fs-portlet is the problem on user level or for a group?

Collapse
Posted by Dave Bauer on
Peter,

You might want to check the bugtracker setp for dotlrn here: http://dotlrn.openforce.net/bugs/

And if the problem is not addressed post a new report.

Collapse
Posted by Peter Alberer on

Hmm, i tried to incorporate the permission_p caching, but then there were a lot of errors in the log file concerning filters (tclop: invalid return code from filter proc). There are probably additional changes necessary that i overlooked.

But these changes would only affect the tcl procedures and quite often the problem is with db-queries that use db-procs to check permissions. The caching of permission_p will not speed up the db-procs.

The performance problem seems to affect the group file-storage as well.

Collapse
Posted by Ben Adida on
Yes, the big issue with portal-based tools is that a lot of queries are being
performed. That explains your main page problem.

The calendar page time is unacceptable though, and requires optimization
(that's just one module). The file storage page is even more unacceptable.
Would be good to turn on developer support and see what queries are
causing this slowdown.

In general, for portals and OpenACS, I think it's time to investigate a data
caching system with domain-based invalidation. For example, cache data
related to calendar id N, and invalidate the cache for calendar id N only when
you add an event to that calendar. This isn't too difficult technically speaking,
but it does require some non-negligible amount of work.

This kind of caching approach could be applied to permission_p, although I
think the problem with permission_p is a bit deeper and needs to be
investigated a bit more in the direction that Don's been mentioning recently
(making sure the views are done correctly, simplifying the permission_p call
itself, etc..).

Collapse
Posted by defunct defunct on
Don't know if this is too simple.... but as a quicker performance fix, would it not be possible to use memoize functionality, to ensure no re-running of queries functions that haven't changed?
Collapse
Posted by Don Baccus on
I have some financial support for working on both caching and permissions for 4.7.  I've been talking to Berklee about ideas I have to improve the simple caching db api pI did for Greenpeace (which Berklee uses heavily in their customized deployment of dotLRN).  Berklee's made some improvements of their own, too.

Clearly there's no time to fix things for 4.6.

Peter, the memoize caching avoids repetitive calls to the db for a particular user.  The initial page load is still slow of course but subsequent views are heavily cached.  Not just permissions but some vital portal stuff are being cached on SSV2.  Sloan has as a goal trying to some of this moved into the "real" dotLRN code base.

Simon, memoize is great for procs that return values from the db.  The caching db api I hacked together for GP caches db_multirows, db_list, db_list_of_list and the like and is very simple to use (just add "-cache" to the db_* calls you want cached).  While it's not general purpose as written for Greenpeace, our work there and Berklee's use makes it clear that it can be made to work very well and very easily.

Of course we'll have public dicussion before any final decision is made on just how to most effectively add caching.

However ... caching only reduces the time needed to generate a query result after the first query result is produced.  The first query needs to be reasonably fast, too - and that's where permissions really lets us down.  Simple permissions scale well, but group-based permissions don't (and dotLRN heavily depends on relational segments, which can be thought of "subsets of groups").  IMO OpenForce made the right decision in using relational segments, they work very well.  They were more or less ambushed by the unknown fact that group/relseg perm checking scales much less well than direct permissions.