Forum OpenACS Development: Notes from OpenACS 6 Design Discussion in Heidelberg

We had an OpenACS 6 design session at the great .LRN conference in Heidelberg a couple of days ago. Here are my somewhat unstructured notes from the meeting. The most active participants in the meeting were (not surprisingly) Lars Pind, Jeff Davis, and Don Baccus

- Object orientation - each object type should have a predefined set of Tcl methods such as ::create, ::get, ::edit, ::delete, ::render, ::list, ::rss_feed etc. We might want to consider using an object oriented variant of Tcl (such as incr Tcl or XOTcl).

- We want to reduce the usage of PL/SQL code so most procedural logic for OpenACS objects will be in Tcl code. This will speed up development, reduce code duplication and ease upgrades.

- We want to auto generate SQL for the standard cases and leave escape routes for the more complicated ones. This means there will be a way for package developers to manually write SQL queries when they need to. Reasons for using manually written queries may be that the SQL generator cannot provide the data set needed or because its SQL is not performant enough.

- Object services such as ratings are not object type attributes and typically apply to all visible objects in the sytem. We would like to see Jeff's KMS system with its object services integrated as soon as possible.

- We want our view templates (such as blog view, calendar view, list view etc.) to be object type agnostic so that the website admin can mix and match views with object data as he pleases. For example the blog view may require a data source with the attributes subject, body, date and author but it doesn't dictate this data to belong to a specific object type (i.e. the blog view can list other things than blog postings, such as photos). A data source can draw its data from one or multiple data types. We are missing input contracts for adp:s. We need to converge the form builder, ad_page_contract etc. data types. Challenge - do we want sortability to be in the view/datasource contract?

- Challenge - how do we maintain state for portlets while avoiding URL name collisions? For example suppose you have two blog portlets on a page you then want to be able to change the sorting of one of the portlets without affecting the other portlet.

- Request processing. We could mount objects of any type in the site map if the objects know how to render themselves.

- We would like to provide an admin interface (a CMS backend) where interaction designer draws up page flow and wire frames, the designer designs the pages

- Portlets take a set of parameters and uses a certain adp (view) and Tcl (datasource) template. Portlets register themselves with the portal system so that they are available for selection by the page builder in the page admin UI. To make working portals more attractive and worthwhile we need more high quality look and feel options (using CSS) and a better portal UI (we can look at services like TypePad for ideas).

- Caching of portlets should ideally be done at the package level since the package knows when its data has changed and the view cache needs to be flushed. The portal system cannot easily know when to cache. We need a good way to cache adp output. The fact that different users have different view permissions makes caching of object listings very difficult. If all listed objects share the same parent the permission caching problem can be circumvented if instead of checkig read permission on every single object we only check read permision on the parent of the objects.

- For scalability the database tends to become the ultimate bottleneck since you can always add more frontend web servers. Multiple web servers has the drawback that caching becomes more difficult.

- Page rendering. Content management systems (Typo3, Plone etc.) have a backend with a page tree that allows Interaction and Graphic designers define pages, layouts and links between pages. Implementing this on OpenACS may not require changing the request processor. The page map could be defined in an XML file in the filesystem.

Posted by Dave Bauer on
Peter said: " Page rendering. Content management systems (Typo3, Plone etc.) have a backend with a page tree that allows Interaction and Graphic designers define pages, layouts and links between pages. Implementing this on OpenACS may not require changing the request processor. The page map could be defined in an XML file in the filesystem."

Uh? Am I missing something or is this already covered by the Content Repository, and eventually by moving parent_id up to the object level? There is no user interface for it, and its something high on my list of things to do especially in relation to "- We would like to provide an admin interface (a CMS backend) where interaction designer draws up page flow and wire frames, the designer designs the pages"

I am sorry to be negative, but why would we want an XML file in the filesystem to define relationships between objects in the database?

Posted by Barry Books on
One of the things that sets ACS apart from other systems is it assumes a database and pretty much puts everything into it. I would prefer even the code was in the database but currently that's a bit difficult. Files are easier to move around, the database is better for storage, relations and updating.

The tcl functions listed above are interesting. What I'd like to see also is ::putxml and ::getxml for objects. That would make it easier to transport objects via xml and files (or http) and still allow for creating the initial objects via xml and files. I think this would solve what I see as some of the current conflict in ACS. It wants to be a dynamic system that allows users to add attributes to object via a web interface and everything automagically works, yet nothing quite works that way because the initial overhead is high because there is no way to easily create the objects required and transport them around.

Posted by Jun Yamog on

The first 3 points seems to be a direction to create some sort of object persistence layer.  Although I am not fully against it, it moves us farther away from the roots of ACS.  That is make your sql get the data, etc.

There are also negatives on it as none sql aware developers may abuse the system.  The one object manipulator is not much of a problem, the problem are the ::list api.

What is the end goal we would like to achieve is making this items?

Where is OpenACS going to?  Do we want scalability, rapid application development, better integration, etc?

Maybe by making an over all technical goal we can prioritize the items.

Posted by Lars Pind on
Hi Dave, Barry, and Jun,

First of all, we're just trying to have a fruitful discussion on where we collectively want to head. I personally have some ideas, because I've thought a lot about it over the past 6 months, but I'm stilling having trouble articulating it and getting the details sorted out. Which is why I wanted to discuss this in person, which is, to say the least, much easier.

The general idea is to make OpenACS more of what it was intended to be (from OpenACS 4 and forward). Not less.

Also, the idea is obviously to build on what's there.

In a number of places, we have 3-4 half solutions to the roughly the same problem. In this case, the intent is not to do a fourth or fifth half solution, but rather to try and synthesize the existing solutions into one coherent solution.

Now, to some specifics:

Dave: The underlying idea is that if OpenACS is going to be both in the eLearning space, in KM, and in whatever-website-you-happen-to-build-for-your-grand-mom business, then it will have to allow for customizing the user interface. For example, if you have a site that only needs one forum, it should be trivial to get rid of the page listing all forums in a package instance.

Currently that requires manually changing files in the packages/forums/www/ directory, which means you're now going to have trouble upgrading.

Second, the ability of users or superusers to tweak and make changes to the user interface is a major differentiator in the .LRN market. None of the big players support this (for obvious reasons), and everybody is doing it - Heidelberg, Mannheim, Bergen, Vienna, Sloan.

Now, my thinking is that browsing objects in the CR is different from browsing pages in the system. In a traditional CMS system, in the back-end, you'd have one page per news item, for example. In OpenACS, we have one news-item page, and it takes a parameter of which news-item to display. I want to keep things the OpenACS way, because that's what a dynamic web application platform should look like, but I'd like to (a) make it easy for people to customize UI to suit their users, and (b) do so without forking code, and (c) have a back-end UI to help them.

Now, we talked about where to store such a thing, noting that there would be advantages to being able to use version control for this (it's essentially UI development), and Jeff suggested just storing it in a /local directory.

Barry: OpenACS doesn't currently put code in the file system. And there's a trade-off, as you mention. I don't know what the right place to draw the line is.

Jun: A number of us here in the community are painfully aware of the ACS Java experience, and have no intention whatsoever of repeating them.

I'm not sure how you draw that conclusion from what's written, though.

The ::list method mentioned, in my mind, should know how to build the query to get all the desired objects in one query, not get each object through a separate query. You can still choose to develop like you do today, or you can hand-tune the query.

As for moving away from PL/SQL, that was something that came up during the conference. There was broad agreement that PL/SQL is hard and time-consuming to develop and debug, and that it's mostly not necessary at all. You can easily auto-generate it, write the same code in Tcl intead, or write a single API that does it for you. There's no talk of change in functionality, just in choice of programming language.

Yes, some people may abuse the system, but we also have poorly written queries today.

Look at some typical applications like forums, file-storage, blogger, calendar ... There's a lot of code that could be auto-generated.

Of course, as Jeff pointed out, we might not want to make it too easy, we all still want to have jobs ...


Posted by C. R. Oldham on
<blockquote> - We want to reduce the usage of PL/SQL code so most
procedural logic for OpenACS objects will be in Tcl
code. This will speed up development, reduce code
duplication and ease upgrades.

Maybe it was outside the original scope of OpenACS, but having all the PL/SQL code in the database means that if you have to interface with the datamodel *outside* of OpenACS for whatever reason, you can.

Now in this day and age of XML-RPC and Web Services maybe this isn't an issue anymore, but it still seems that it will be easier to interface legacy systems through the PL/SQL layer rather than the Tcl layer.

Posted by Jeff Davis on
One idea was to autogenerate the methods for new and delete etc that are almost entirely boilerplate anyway or to use the views + rules (like CR uses) for that. As it stands most of the packages don't implement much more than that anyway and having to write those by hand was the sort of thing we wanted to dispense with.

You would make creating an object type declaratory and the requisite tables, plpgsql, and views would be created from that (and lest anyone think this is to ambitious I would point out the CR already implements exactly this but only for CR derived object types).

Posted by Tom Jackson on

This is what my query-writer does: takes object name, attributes, etc. and generates the entire pl/sql or plpgsql for creating the object type plus insert/update/delete plpgsql. One problem with plpgsql pl/sql is that you have to drop and reload the functions before things work. It also now has an option to just use tcl/dml statements (plus a tcl layer acs_object.create function). The tcl/dml option (per object) allows you to just alter your datamodel, add the field to query-writer, and the insert/update code keeps up. Query-writer only works on objects directly derived from acs_objects. It also has hooks for callbacks in certain places, like after parameter validation, after the regular dml/pl is executed, etc.

One thing I have found difficult to make easy is to move objects created on one system to another one. Ofcourse this is only done by the developer, but takes a few minutes and a little checking to make sure it is done right. Having a file dump format for your objects is an essential feature.

On pg, query-writer now has a wizard where you just supply the table name to begin. It queries the system tables to figure out what columns need to be added, and walks you through the process. Takes about five minutes to get a table working, even if it isn't an object_type table.

Posted by Barry Books on
The code I was refering to is all the tcl/adp code. My main point was you have to bootstrap the system some how but after that I'd rather have everthing in the database.

It would be very nice to have a standard set of SQL and TCL routines for every object type. I also like the idea of auto generating them because it ensures they are there and they they consistantly do the same thing. I think having an object::init function for custom logic would be nice for objects that need that.

It seems ACS is being pulled in a couple different directions. At first it valued SQL and scalability. Now it seems headed for customization rapid development and database portablity. I personally prefer the later. Scaling can generally be solved with hardware/tuning and hardware is pretty cheap these days. I also think if you could customize without hacking the code upgrading would be alot easier.

At first it valued SQL and scalability. Now it seems headed for customization rapid development and database portability.
In my opinion, OpenACS has always highly valued all of those things, and there is no inherent conflict between them, either. It's only a question of what areas people currently want to put effort into improving.
Scaling can generally be solved with hardware/tuning
Barry, that's a statement with basically null meaning. First of all, scaling is very much not "generally" solvable by hardware. (Lousy software and lousy queries can easily trump all hardware.) Secondly, many folks have hacked on pieces of OpenACS over the years to make it scale (permissions in particular come to mind), and that's precisely what "tuning" is!

So yes, scaling can generally be solved, somehow. But doing so is not necessarily easy, and it's far, far better that that solving go into the standard toolkit, rather than every high-volume OpenACS site having to fix it all themselves. The only reason getting an OpenACS site to scale reasonably well today is reasonably easy, is because of all the scalability work that has already gone into the toolkit.

Posted by Jun Yamog on

Sorry did not mean to have a negative discussion.  Just would like to point out some of the possible pitfalls of some items.

I think having create, edit, etc. standard manipulators is good and helpful.  I would lean on having default routines if those methods are not overridden instead of auto generated routines. I guess there are the very similar.  I would lean on the pregenerated methods be not easily visible by the developers and that you need to create overrides rather than editing existing pregenerated methods to override.  So in order to override a standard method you create rather than edit existing auto generated standard methods.

Posted by Don Baccus on
Andrew's right ... but there has been a paradigm shift in the OpenACS community - some of us, at least - to use Tcl APIs where possible in order to REAGAIN the ability to rapidly generate applications.

The write-the-API-in-PL/SQL works OK if all you're just using one RDBMS, but supporting both makes it a pain in the rear.

In practice it either slows down development considerably or (as is becoming all to common in our community) only one RDBMS  rather than both are being supported by developers.

Also the ability to automate the generation of forms, displays, etc has been diminished in both ACS and OpenACS 4.x code by laziness - attributes aren't getting properly defined because it's tedious and time-consuming.  A Tcl object definition API can automatically generate all the SQL stuff needed, including tables (the type system does this today via PL/[pg]SQL though I doubt more than 10% of our OpenACS programmers are aware of it) and attributes.

Jun - we're not talking about generating a persistent layer, just making certain things easier.  As Jeff mentions, one thing we might do would be as simple as defining the type views for all objects as is already done for CR objects.  Likewise autogeneration of new/delete functions (or generating the inserts in straight SQL) would end abuse of those procedures (some types do more in "new" than they should, for instance, which means you have to *look* at a type's "new" procedure to know how much it is doing and how much you have to do afterwards).

Consistency, ease of programming, the moving of tedious tasks like type attribute definitions into Tcl ... these are the kind of rapid development things we're talking about.

No more radical than the development of the recent and I believe uncontroversial developments of Tcl APIs to define service contracts or portlets for instance.

More and better support for rapid development would be very welcome, and the specifics Don touches on above mostly sound good to me. (Except for auto-generation of the SQL DDL for data models; so far I'm leary of that.) It certainly would be very nice to write PL/SQL only when you really want to write PL/SQL, rather than being forced to do so repeatedly for boilerplate code.

However, you OpenACS gurus who will be designing most of this stuff, please keep in mind all the discussions and reported problems with that other recent rapid development tool - ad_form.

Historically, the switch from OpenACS 3.x to 4.x seems to have given substantial additional power at the cost of a moderate increase in complexity and decrease in development speed. (And from what I've heard from those who've actually used it, ACS 5 Java probably gave a huge increase in complexity and a great decrease in development speed, for nebulous or un-realizable increases in power. Last I heard anyway; perhaps it has improved since then.)

If OpenACS 6.x could improve on the power of OpenACS 4.x and 5.x while beating the rapid development and short learning curve of OpenACS 3.x, that would really be slick. Given what others have talked about here, I suspect it can be done, too.

As for "improving the power", I don't see any mention of Lars' "object soup" ideas. What happened with that one? Was it discarded as a bad idea, tabled for later discussion for a later version, or?

Also, Peter's original list above doesn't seem to have "best of breed Content Management toolkit" on it at all. I find that surprising, given all the past discussion of how hugely useful it would be to dig out all the existing almost-never-used OpenACS Content Management functionality and re-factor it into one cohesive, integral, widely used whole.

Posted by Don Baccus on
CMS is an application layered on top of the OpenACS core itself, which is why there was no explicit discussion of ripping out the good parts and repackaging a more useful version as part of this discussion.  This doesn't mean it's not important.

What we have discussed is the integration of the CR/CMS and acs object attributes and associated operations that work on them (automatic form generation, automatic table generation, etc).  Separately there's been some work done to integrate Jun's BCMS Tcl API to the CR (which exposes some of the CMS-type functionality) into the core.

All of this should help those who want to work on an A+ CMS solution.  Meanwhile, Daveb has been working on his XCMS UI application and of course Jun has provided the Basic CMS (bcms) package for simple CMS needs.