Forum OpenACS Development: OpenACS 4 Permissions scalability

Collapse
1: OpenACS 4 Permissions scalability
Posted by Don Baccus on 01/22/03 08:50 PM
Folks might be interested in a document I just uploaded covering OpenACS 4 permissions scalability issues and a possible solution.

I've asked Dave Bauer if it's possible to open up this edit-this-page generated document for comments. If he gets back to me with a "yes" answer then I'll do that and would ask folks to comment there rather than here ...

Collapse
2: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Don Baccus on 01/22/03 08:54 PM
OK Dave's turned on comments (thanks so much, Dave!)

I'd suggest general questions be posted here, but would really like to see any specific suggestions or observations posted as comments to the page in order to keep all the good stuff in one place.

BTW the mini-"populate" package I wrote to generate test data will be made available shortly, I'm asking some of my fellow CVS/openacs.org server babysitters to suggest the right place to put non-mainstream packages like this.  We should get it figured out in the next few hours or tomorrow and I'll post after I upload it.

Collapse
3: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Deds Castillo on 01/23/03 03:43 AM
The results look promising.  Would really be great to incorporate it into current live sites who suffer the permissions system overhead.

Since we are talking about permissions, it would actually be great if users are granted permissions on the "Add a Comment" feature. =)

Is there someone who volunteers to replicate Don's tests on a faster machine with larger RAM? and possibly on SCSI disk?  I think the results Don posted are already significant enough, but it would be interesting to see this tested on various hardware.

Collapse
4: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Dave Bauer on 01/23/03 03:58 AM
Registered users are granted general_comments_create on that page.
Collapse
5: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Deds Castillo on 01/23/03 04:06 AM
That's odd.  I'm getting a security violation.  I'm pretty sure I'm logged in as I'm able to write this.  Can someone else try it.  I've tried it both on Konqueror and Mozilla.
Collapse
6: Re: OpenACS 4 Permissions scalability (response to 5)
Posted by Bart Teeuwisse on 01/23/03 04:11 AM
Deds, it is not you. I get the same security message.

/Bart

Collapse
7: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Dave Bauer on 01/23/03 04:45 AM
OK, i think I fixed the permission, but only on that page. I am not sure why that permission is not inherited from the content folder. The context_ids all appear to be set correctly.
Collapse
8: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Dave Bauer on 01/23/03 04:47 AM
That page lives inside an acs-subsite, and I am not sure how setting permissions to 'registered_users' will flow down throught the subsite.
Collapse
9: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Don Baccus on 01/23/03 05:45 AM
Regarding testing on more RAM ...

I did monitor things to make sure my test results weren't being skewed by disk I/O.  I'm in no way offended by the fact that Deds has raised the issue, but the datasets I'm talking about are small enough so it isn't an issue.

After all, when I was a kid 384MB was unimaginable for a warehouse of magnetic tapes :)

Good question, though.  Results were CPU, not I/O bound.  I made sure of that before collating them.

But ... various architectures and implementations of architectures do exhibit different execution profiles.  My laptop has a P500 Celeron with 128KB L2 cache.  Straight Coppermine PIIIs should perform better in the high calculation case, AMD Athalon's > Thunderbird even better.  P4's?  I don't know, have never used one.

Still ... I don't think architectural differences won't change the story on the gross scale.  "a lot better" is "a lot better" even if one or the other CPU chip defines "a lot" differently.  Obviously the experimental queries are doing much, much less work.

It also indirectly raises the issue of maintaining an expanded "party_member_map" table and its indices.  There are only two columns in the table, so under Oracle the cost should be fairly minimal.  PostgreSQL, however, has a fairly steep per-row penalty that makes the storage impact more severe than  under Oracle.  And of course for both RDBMSs there is the need to store the index.

I think we all accept the fact that the most intense and busiest sites built with OpenACS may require Oracle.  We mostly care that the 99% of sites that don't fall into that "busier than hell" category can run with Postgres.  I think my tests show that both can be made to scale much better than they do today even if the Oracle numbers are considerably better.

And of course if any of you want to experiment and improve upon either version of the function, and can beat me - hurray!  Kick my ass!  The faster the better, that's all that can be said!

Collapse
10: Re: OpenACS 4 Permissions scalability (response to 9)
Posted by Jeff Davis on 01/23/03 11:48 AM
For another datapoint, I have a test db with 300k acs_objects, 42k users, 8 groups, 7 rel_segments, 125 acs_privileges, and 84k rows for acs_permissions and party_member_map. This is the typical openacs install with all packages installed with registered users + a few sitewide admins but no real permissioning beyond the default. On that site (once the disk cache is populated) the time to check for a direct permissions check is 24ms and for indirect permissions it is 8.5ms (this is on an amd XP 1800+ with 1gb memory).

Curiously, this reproduces the result on oracle where the direct permission check is a fair bit slower than the indirect permission check (something not in Don's postgres results).

Collapse
11: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Don Baccus on 01/24/03 03:56 AM
Curious, indeed, because the numbers seem counter-intuitive and I double and triple checked to make sure I hadn't messed anything up in my testing.

The fact that you get the same result is a curiousity but a relief, as well.

Your computer's also a lot faster than my laptop! :)

Collapse
12: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Jun Yamog on 01/24/03 04:42 AM
Hi,

I wonder if Oracle CBO will have any impact on the queries?  Also PG vacumm analyze?

It looks good that we can improve on performance.

Anyway if needed to see how it scales, I can try it on my P4.  On a side note, going from P3 to P4 OACS does scale linearly.  So does CCM, CCM does get more juice if JDK 1.4 is used.  Although its a one off apache bench out of curiosity benchmark.

Collapse
13: Re: OpenACS 4 Permissions scalability (response to 12)
Posted by Jeff Davis on 01/24/03 10:13 AM
I vacuum analyze the test db regularly so that's not an issue with the numbers I posted. The default on oracle is to use the CBO. I have seen some queries like this where the rule based optimizer is faster (sometimes a lot faster) but it's hit and miss (and I think dependent on quirks of both the data and the rule based optimizer). One other hint which sometimes made a difference for oracle was to say /*+ FIRST_ROWS */ (I think thats right, though it's from memory).

I don't know what people think about trying to tweak the queries like that since it's all such a black art (to me at least).

Collapse
14: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Frank N. on 01/25/03 09:57 AM
Don't know if the following applies to the benchmarks, but I just learned that indices in PG always grow in size, ie. the entries for deleted rows are not removed, not even by VACUUM. This is apparently the penalty for being able to do a VACUUM on a live db.

The unofficial way of thoroughly cleaning up a PG db is to first do a VACUUM, then dropping indices and subsequently recreating all of them. Wether this can be done on a live db depends on the given application.

Has anyone using OpenACS had to deal with these issues? Apparently heavily used PG db's use a cron job to thoroughly purge the empty space at regular intervals, including recreating all the indices.

Frank.

Collapse
15: Re: OpenACS 4 Permissions scalability (response to 1)
Posted by Barry Books on 02/13/03 02:06 PM
I have about 5 million acs_objects . So far with tweeks I havn't had any real problems except when trying to get permissions on large data sets. In looking over my data I've noticed that the vast majority of permissions are granted to groups. Also most users are not in very many groups. I suspect this is true for most implementaions and led me to the following thought.

What if you could only grant permission on groups and then change permssion_p to query based on what groups you are in instead of who you are? It seems like this would reduce the number of rows in the permssion views because they would only contain groups. For the times you need to grant permission to just one user you could create a group with one user in it (this could be done by default when the user is created).

I haven't really given this much thought, but I think it would be as flexible as the current model. Does anyone think it would be faster in the real world?