Forum OpenACS Q&A: ACS classic Tcl module repository up

Hi folks, just to let you know, in the advent of the purchase of ArsDigita, we've put up a module repository for 4.X tcl modules at developer.ybos.net. Anyone can upload, so please feel free. Ybos has uploaded a bunch of code to it, but we'll be uploading more in the next few weeks, as we sort through things like the six instances of the news module we have lying around, for instance.

Additionally, we put up all the modules from arsdigita.com, (about 150 of them). We have a few goals:

  1. Make sure that the community has an easy time finding and downloading ACS modules
  2. Respond to complaints we're not releasing any of our code
  3. Provide a central location for ACS-tcl modules
We've also ported the ACS 4X-Tcl to Oracle 9i, which was an interesting experience. Our updated core modules will work with 9i as well as 8i, but we haven't re-modulized all of them yet. We'll be releasing a tarball with the 9i changes incorporated soon. (Ask tigre@ybos.net)

Finally, we're happy to support openACS modules as well. If the .info XML spec hasn't changed, they will upload just fine. We have space in the database for recording which database the packages work with, but haven't implemented the code to check the .info file for that information, and act accordingly. Are you folks doing something like

<database-support>
 <database>postgres-7.2</database>
</database-support>
in the .info files? If someone is willing, maybe you could upload a few openACS modules, and we'll make sure that the importer groks them correctly.
Collapse
Posted by Talli Somekh on
Peter, since the OACS code's been around for the past six months, why don't you guys download and find out how our files are organized, then work with the dozens of developers and companies our there who already have set a standard?

talli

Collapse
Posted by Peter Vessenes on
Talli, we just put up a free service that will help anyone works with the ACS 4, and offered to support you, and your company further by making sure that it works with your chosen spec. What am I missing?
Collapse
Posted by Talli Somekh on
I guess I'm missing everything, except for your disingenuousness. But I expect others have something to say, too, so I'll let them continue this discussion for me.

talli

Collapse
Posted by Peter Vessenes on
Also, I forgot to mention that we are hiring ACS developers: developer.ybos.net/hiring. I mentioned this at ArsDigita, but it was edited out of my original post.
Collapse
Posted by Michael A. Cleverly on
Strange that aD edited it when they're effectively out of business, their born-again-java core product discontinued, and all but a skeletal staff laid off... wierd.
Collapse
Posted by Peter Vessenes on
It seemed like a "corporate" decision -- Bryan Che, who is a totally nice guy, was very polite to me via e-mail. That wouldn't be a fun job, editing people's postings due to your corporate policy.
Collapse
Posted by Don Baccus on
Thanks for putting this up, Peter.  We haven't gotten our own repository up yet so having this service is very useful.

The .info file does indeed spec database support as you're assuming in your XML snippet.

The OpenACS core and APM support legacy ACS 4 Classic packages as well as those extended to work with our core.  I just moved a large ACS 4 Classic site to OpenACS 4 as an experiment with no problem.  There are some minor differences in acs-messaging and acs-mail, but I think those could be replaced with their ACS 4 Classic versions if someone didn't want to do the minor hacking necessary to use our slightly modified stuff.

OpenACS 4 incorporates a bunch of bug fixes and some (though not many) important performance enhancements, especially in the area of permissions checking.

So ... in an ideal world you folks at Ybos might explore loading up your custom packages written for ACS 4 Classic on OpenACS 4.  You might find it painless.  Or perhaps very slightly painful.  I doubt it would take someone more than a day or so (given a machine that runs Oracle reasonably fast!).

It would be an interesting experiment.

The reason I mention this is that OpenACS 4 packages running on the ACS 4 core will break (the APM will choke).  So maintaining a repository with application packages from both code bases would confuse someone using ACS 4.

On the other hand, someone running the OpenACS 4 core should be able to download a package and our APM will recognize it as being Oracle-only if it's an ACS 4 Classic package, and Oracle/PG if it is one of our fully-ported OpenACS 4 packages.

I'm rambling but hopefully this makes some sense?

Collapse
Posted by Peter Vessenes on
Don, this does make sense. I'll update the parser to do the database check, thanks for that info.

On loading into OpenACS 4, yes, it would be really interesting. I like the idea of moving some stuff over just to see how it is. The prospect of a large merge is a little daunting, but it would be pretty easy to move over individual packages, and see how they go.

My thought on the repository was that one would eventually be able to select a preferred architecture, and see packages which support it, which sounds in line with your concerns.

I'll post a follow up here with a little more info on how that goes. Out of curiosity, have you folks done any 9i porting yet?

Collapse
Posted by Titi Ala'ilima on
I just moved a large ACS 4 Classic site to OpenACS 4 as an experiment with no problem. There are some minor differences in acs-messaging and acs-mail, but I think those could be replaced with their ACS 4 Classic versions if someone didn't want to do the minor hacking necessary to use our slightly modified stuff.

If you've kept track of what those hacks are, it mightn't be much of a problem to make our standard classic stuff play well with your setup.

OpenACS 4 incorporates a bunch of bug fixes and some (though not many) important performance enhancements, especially in the area of permissions checking.

At least some of which we've incorporated into the classic version, but I'd be interested to see what else you guys have cooked up beyond what I've seen on the bboards. I have a few more ideas brewing myself, but I haven't had time to test them all out.

If there's one thing we could probably all use a great answer for it's the permissions checks.

Collapse
Posted by Don Baccus on
The stuff you've seen on the boards is most of it.  I've also chased slow "acs_object_party_object_map" queries and have replaced most of them with "all_object_party_object_map" queries.  Postgres is universally faster with the latter view, Oracle seems to at least almost, and maybe always, faster with it.

You probably won't have those because the code's littered with them, unlike the LAUSD (Jon Griffen et al) change to permission_p() which was a one-place drop-in improvement.

I think the major thing we did was to move posting stuff from acs-messaging to acs-mail, and the API should be the same but with a different package name.  We did that because we needed to rip out dependence on Oracle's Java SMTP (which Postgres doesn't have) and wanted all SMTP stuff in one place.

There's a minor change to the workflow API, too, involving notification callbacks but I doubt many people are impacted with that because I get the impression that not many people have worked with workflow (they should, though, especially as it improves!)

Collapse
Posted by Titi Ala'ilima on
I'm actually hoping to axe both of those views in favor of something a little more manageable.  Large systems (thousands of users and hundreds of thousands of objects) make those views pretty hard to use.
Collapse
Posted by Titi Ala'ilima on
And ditto Don on the workflow.  Takes some getting used to, but boy is it powerful.
Collapse
Posted by Peter Vessenes on
One thing we've done is implemented an nsv_shared permissions cache, which is flushed on certain actions. It's not particularly smart, and it's not particularly fancy, but it's sped up a number of queries remarkably well.

The syntax changes from

db_multirow query query_with_perms {
select o1.object_id, o2.name from acs_objects o1, acs_objects o2
where o1.blah = o2.blah
and acs_permission.permission_p(:user_id,o1.object_id,'read') = 't'
}
to
db_multirow .. .. {
select o1.object_id, o2.name from acs_objects o1, acs_objects o2
where o1.blah = o2.blah
} {
if {![yb_object_permission_p $object_id "read"]} {
   continue
}
Which looks, oh, so ugly, but turns out to be much faster. This seems to be due to the fact that in the first case, oracle will do full table scans on any query with a pl/sql call in the where clause. The second case, the optimizer uses the where clause intelligently, because we've taken away the confusing bit.

The method also relies on an undocumented feature of the db_multirow proc, which will execute the string after the query, a-la db_foreach.

Collapse
Posted by Don Baccus on
That feature's documented in the code, actually (boy, I'm a smart-ass). Not the most convenient place for it...
select * from
  (select o1.object_id, o2.name from acs_objects o1, acs_objects o2
   where o1.blah = o2.blah) foo
where 't'= (select acs_permission.permission_p(:user_id,foo.object_id,'read') from dual)
will awkwardly force a smarter scan on the inner query. Andrew Grumet found this out while working on improving the performance of file storage.

The simpler "'t' = acs_permission..." (without the subselect/dual) will still do a full table scan.

Oracle's bleeping stupid at times.

Collapse
16: Permissions (response to 1)
Posted by Titi Ala'ilima on
Dave Rodriguez clued us into the query gymnastics to speed up permissions checks.  However, it doesn't seem to work all the time.  In some cases we've resorted to the distasteful method of selecting out all possible matches, with the permission value, and screening them out in Tcl.

I did have some success creating procs which return a "collection" of ids, which can then in turn be used like a table.  This is very Oracle-specific, and is only practically useful in certain circumstances, particularly answering the question "who has privilege X on object Y?"

Collapse
Posted by Peter Vessenes on
The query is helpful in the join cases, so thanks! I didn't know about that. We'll have to test it against some of our other permission situations. For sure the cache is faster than the database checking perms against site nodes (which happens once per page, or once per gif, if the request processor doesn't ignore images).
Collapse
Posted by Don Baccus on
Yeah, but if it's done in a single query you can always cache the whole thing ...

PostgreSQL allows SQL functions to return rowsets with the result being used like a table, too, so the tricks mentioned in regard to returning collections in Oracle should be applicable in PG as well.  I already use this feature in our tree (hierarchical query) code.

Collapse
Posted by Roberto Mello on
What I don't understand is why Ybos has a new "release" of ACS Tcl when OpenACS has been the official maintainer of the Tcl tree for months, with lots of work to complete and improve it.

Having two different trees seems like just a waste of effort to me, unless it's just a marketing stunt.

Collapse
Posted by Peter Vessenes on
Hi Roberto,

We don't consider it a marketing stunt, we've just been doing what's seemed to be best for our customers this year. When we started modifying ACS 4.2 Tcl, the OpenACS people were just starting to think about the architecture. Check some posts I made here over a year ago to see what I mean -- here or here, although the second one seems to have lost some of the posts.

We're totally for OpenACS, and as we have time, we're working to roll our stuff back into the OpenACS. It's not just a clone of the OpenACS functionality, though -- that would be totally useless. We've been working on other areas, usually application and usability areas of the ACS. Because of that it's made some internal sense so far for us to go with what we've had.

As I said, we're totally for the OpenACS! As we have resources, we're working on the integration. If you're not too busy, and if you think what we've done might be helpful to the OpenACS, maybe you'd like to help. All our code is available for download on the developer.ybos.net site.

In any event, in the interim, we want to release what we have. If you like it, great! If not, no problem. But, we hope you like it! We do.

Collapse
Posted by Ben Adida on
Peter,

I welcome all additional participation to the OpenACS
community, and, as it seems that Ybos is beginning to
participate, I'd like to say "Welcome." I can understand people's
concerns regarding your motivations given that your
contributions began only after ArsDigita's demise. In the end,
however, the open-source community works by judging your
actions. So welcome to OpenACS, and hope to see your team
contribute to the code in the near future.

In passing, I would like to correct some of the misinformation
present in your posts here and in other places.

OpenACS did not *begin* architecture last year. We began *
reviewing* and *improving* the existing ACS 4.2 architecture.
Don't forget that OpenACS started with ACS 4.2 as a base and
only improved from there. We've made many improvements in
the consistency of APIs, speed of many modules, database
compatibility. In fact, we made *certain* that we never took
functionality out of our code so that, at any point in time, our code
was always a superset of ACS 4.2 (in terms of functionality,
stability, and scalability). We worked hard on database
compatibility because it is a critical part of our project. We would
never have the thriving community we have without PostgreSQL
support.

We've also become more stringent about how we name our
releases. We refused to call our first release anything more than
an alpha, because we consider that the ACS 4.2 code as
delivered was pre-alpha, incomplete, very buggy, and lacking in
scalability. We took code that we improved from ACS 4.2, and
called it alpha. And we continually made it available over CVS.

The naming became more honest. The code only got better.
Wine doesn't have to be expensive or French to be good.

Now we're about to release a long-awaited beta. That's because
the code is far better than ACS 4.2: it contains large numbers of
bug fixes, speed improvements, and truly works on PG and
Oracle. We're psyched about it. And we look forward to your
contributions so that hopefully OpenACS can soon be Oracle 9i
compatible, on top of Oracle 8i and PostgreSQL.

Collapse
Posted by Peter Vessenes on
Hi Ben,

Thanks for the welcome, and the corrections! I didn't mean to communicate that you started from scratch at all, thanks for the clarification. We're currently playing with OACS, and have just released an 8i -> 9i patch which think will also work with OACS, and leave the postgres stuff alone. Titi's Message here