Forum OpenACS Q&A: What should be included in OACS 4.6

I think any new major release needs to be a balance of restrospective and quality based work, minor enhancements and so forth.. but also needs some new, creative, 'cutting edge elements.

The world's moving on, the ACS is still largely based on ideas and requirements from 1997 (Ala Mr. Greespun's book) and important and vital though that is I think we also need to be looking to the 'next thing'

I think the comunity needs to identify some area(s) not well understood or addressed at the moment to attract in new thinkers, move the kit forward, and steal an edge on the alternative.

Who know what this might be... VoiceXML perhaps, Mobile Video Streaming or perhaps location based services... answer on a postcard

In summary I'm saying a lot of efforts gone into this toolkit, much of it compensatory for AD's misdemeanours, and perhasp now we should entertain some new, more bleeding edge, elements.

Simon

We have given the world a revolutionary cool and new product (OACS 4.5) with multiple database support, stable version of the good starts AD has made, and new development of all the stuff that wasn't good. The next release should be more about stability than about new features. I dont want us to go down the road, where we drag some bugs with us in the toolkit all the time and then they will be assimilated as normal behaviour.

BTW, bugs also mean bugs in the UI. If a page just doesn't work like a normal user would suppose it to work (either because of language, misleading links, badly formated form {like ask for first name, address, last name in this order}).

I see a 4.7 release end of the year which incorporates all the stuff we have developed on in the various projects (dotLRN, Greenpeace CMS, AD sharenet [okay, might be just wishful thinking], full internationalization support). And who is stopping anyone to add really cool other features (aka packages).

IMHO we should have two kind of releases. One release (e.g. odd numbers) which introduce mainly new functionality, but is not thoroughly tested (e.g. because it was written, but no client has used it in production). The even releases would focus on fixing all the bugs found in the odd release, once it has been given a stress test. I'm not in favour of weeks of in-house testing. Once it works as it should in a well defined, no all including, testcase, release it. Other people will find bugs and urgent enhancements you never thought of.

Just my $0.02

Regarding Simon's comment ... I look at our 4.6 release as being a pretty modest effort to stablize what exists, hopefully add dotLRN elements (at minimum the core changes needed for dotLRN - one reason we need to move quickly is that dotLRN doesn't work with 4.5), and again hopefully internationalization.

As far as "whiz-bang no one else is doing this" work ... if we could provide internationalized packages translated to a couple of languages that would put us a leg up on any open source web/db toolkit I'm aware of.  I know it would sure attract the attention of many folks in Europe, that point was hammered home at our meeting in Amsterdam (you should've come, Simon, England lost anyway ...)

Let me stress that there's no intention of letting a 4.6 release drift for months, either.  We need to set some firm goals, execute, and roll the sucker out the door and move on.

I think "4.7" will be the release in which we can realistically discuss "whiz-bang" cool stuff.  Of course any one wanting to work on something cool immediately is more than welcome to do so ...

As far as the comment regarding SOAP ... this would be a great little project for someone to work on.  It's modest enough so that such work could be integrated in 4.6 if it happened soon enough.

A side note - ex-ArsDigitan Jeff Davis has been bug-bashing like a fiend the last few days.  That's really great. If anyone else has the time to do so and wants to join in, e-mail me and I'll get folks in touch with each other.

I don't want to try to maintain separate downloads on the current site because we don't have an adequate tool to organize them.  Once we get the new site up we can easily unsynchronize releases of the core set of packages and everything else.  Then we can better distribute the management load of maintaining and releasing non-core packages, which would certainly make my life easier.  The lock-step approach we've been taking, forcing everything to release at once, is a PITA.  I think 4.6 will all come in one big lump but I want it to be the last release of this sort.

• make 4.6 more of a stability and polishing release.
• make some of development guide, like standard UI, guide on developing packages, guide on extending the data model, etc. So that in the future the packages can break free of the core with good guidance.
• put in new features that is being used on other projects like dotLRN.
• remove or merge some packages.

• Moving back the datamodel closer to relational data model not this pseudo object model. Though it has benefits but I think making it more relational will also make the Tcl thinner. In CCM it does make sense to make it ala object model. But I think the power of aolserver and OpenACS is the database object model. We will probably just use Tcl as a layer to abstract the different databases.
• Have the packages output a uniform HTML or XHTML, standard have to be set. And have the subsite template either parse it or tranform it. This way packages can drastically look different from one instance to another.

Yes streamlining the data model.  I am not thinking of revamping it drastically.  Maybe it just the way we got it from aD.  There are concepts of inheritance, objects, blah blah blah.  Its too object oriented thinking that makes sense on Java not Tcl.  But I think we should go away from that thinking and concept.  Like a cr_revision inherits from cr_items which inherits from acs_objects.  That's just cr_revisions is related to cr_items which is related to acs_objects. That way we can also remove some objects on acs_objects because there is no relation needed.  Why would you need to put an acs_object for a cr_item and cr_revision?  Does not make sense if you think relational.  They are related anyway... so join/relate them.

Maybe if we think more relational we can streamline the data model that suited for aolserver and tcl.  We can probably market the toolkit as tried and true solution, using relational model which has been tested for over 30 years.  Ofcourse that may have a side effect of being a unsexy platform.  But I think currently we are unsexy but sure kicks butt.  I hope some marketing guys can help out on this if this is a good choice.

Im sorry if I keep on bringing up CCM but somehow I am bound to compare them with OACS since am working on both.  Well in CCM they use Persistence Data Layer.  This is to simplify the complexity of the data model underneath so that other developers may not be aware of this.  Do you think SQL and relational model is hard to understand?  I think its one of the easist data modeling to understand.  But it will get difficult if we move away from the very essense of relational data model.

I am not saying the current data model sucks, I think its great.  But we can make it greater if we streamline it with the thinking of relational not object oriented.  Of course a lot of the community members has to agree on this.  This just my opinion.

About moving more code from PL/SQL to tcl that would be good if it makes sense.  Because less PL/SQL means less effort in porting.  CCM already tries to do this.  There are less than 10 PL/SQL on the core data model.  But PL/SQL makes sense too since its faster and more simple and I don't want to fatten Tcl.  Since is mean to be glue.  So I guess I don't have much opinion this.

What I mean about more Tcl api on db stuff it that move the sql calls from www/tcl to tcl.  That should make www more simple for beginner developers.  Plus the fact advance developers will logically think of making their sql reusable.

<p>
My vote for long term:
<ol>
<li>Streamline further now involving the tcl, packages, etc.
<li>Remove layout on packages and move the layout to portals or subsite template.  An idea is to have packages output very simple yet standard on all packages HTML and have the portals and subsite template put in the layout stuff or even transform it.
<li>Add more service contracs.  Like have permission service contract where LDAP will store the user, group and perms.  Semi core modules are in service contract like CMS, where you can choose the CMS of choice.
<li>.net? web services... I don't think we have to wait for that the next major release may not yet have those things running real things.  Unless there is a project that made use of web services.  I think in the future new features will come from projects.
<li>Official support for multi data store.  Say bboards is stored on another db.
</ol>
<p>
I believe Malte is correct.  We should move more PL/SQL to Tcl.  Now that he have said it my opinion is to move PL/SQL to Tcl for scalability reason.  So lets streamline further the data model and move PL/SQL to Tcl as a general direction.

Although this are not well though of this are just examples.  As I have stated above lets take for example cr_revisions.

Currently for each cr_items we put it in the acs_objects because its suppose to inherit it.  Of course a cr_item can't survive so you have to have a cr_revision for it.  This cr_revision is again registered to acs_objects because of this so called inheritance.  Why not just relate the tables.  a cr_revision is related to cr_item which in turn is related to acs_objects.  That should reduce acs_objects to real objects that needs to be related.

As for moving PL/SQL to Tcl.  Malte has pointed out because of scalability.  I have never though of this see by older posts above Malte's post wherein I said "I have no opinion about this".  After Malte's post I had an opinion same to him, because his opinion makes sense to me.  Maybe an example is looking for the title of a content item.  See etp__get_title PLSQL.  Its simple enough with only 1 to 2 queries but lots of if and thens.  Now it would not be bad to move those in Tcl.

Again the examples above is not well though of.  They just serve some possible concrete examples.

• A cr_item should be an object so we can assign permissions to it. ACS 3 had a funky relational table containing pairs as the permission key. This was awkward to work with and having a single object space that permissions works on is a real advantage, IMO
• acs_objects contains the basic audit information you want to keep about an item (creation date, creation user, etc). If revisions aren't an object you lose that. It could be replicated in the revision itself but we don't want to start duplicating such stuff again ala ACS 3. After all duplication of effort to continuously replicate basic stuff like this in ACS 3 was one of the motivations for creating a common table to hold it all in ACS 4.

PL/SQL ... nearly all of the selects in the system are done from Tcl, so we're not getting much benefit from the wrappers that are there now. insert/update operations are always comparatively slow so caching plans for them isn't a big win plus the plan for a basic "insert" is so simple that Oracle won't take much time generating it. As Stephen points out most of the various PL/SQL functions just wrap simple DML statements - they don't really provide a significant abstraction layer, they just make porting harder.

Dave and I have been working on moving openacs.org to OpenACS 4.5. We moved all the users already and were working on moving the bboards and bboard posts, but we bumped into the driver installed on openacs.org being an old one that didn't escape things properly (oops).

But we'll get the new driver up there and get it rolling. We've talked about using forums instead of bboard (what do folks thing about this?), but I don't know how is the port to PG going, if it depends on anything on the devel branch, and we'd have to come up with new migration scripts (someone I can't remember wrote the migration scripts for bboards).

We'll document how we did this move so others can benefit.

Isn't OO's strength mainly for developing systems that evolve gradually and incidentally, whereas RDBMS strength is in a stable structure (if well planned prior to development)?

Currently there already this system in place.  And you don't need OO to make it work.  The basic way to get content in is to create an acs_object then a cr_item.  After that create an acs_object then a cr_revision.  The are also content repository api for this.  You may also want to start reading the doc of content repository in particular the data object model page.

http://youopenacssite/doc/acs-content-repository/object-model.html

Then look at how file-storage or etp and study how it puts in content to the database.

As for getting it out its just a matter of selecting the cr_item and cr_revision.  Or better yet use some Tcl API what DonB has wrote in particular cr_write_content.  Its a matter of getting used to the data model and api, after a while everything will make sense somehow.

I hope we can move the code easier for new developers coming in.

Both databases OpenACS supports are relational, and the language used to talk about the datamodel doesn't change that fact. The OpenACS datamodel is relational. What that means to me is that it is a very fluid object model. Any row from a select query is an object. Any row from a table is an object. Objects in languages like Java are fixed and require programming effort to create. In an RDBMS, the super object is the entire database and the SQL language. You can use the methods of the SQL language to extract parts of the data, to change the data and to create new data (objects).

What PL/SQL does is to extend the methods that operate on the data, maybe simplifying the interface to the database. In the case of creating an acs_object, the __new procedure handles quite a bit of effort. If this effort were moved to the tcl layer, the porting effort would extend to every page that called the __new procedure. Instead of one change, potentially many would be required.

For me the most time consuming porting effort seems to be the select queries. The join syntax for Oracle and PostgreSQL are significantly different.

If standard PL/SQL and plpgsql packages were used for create, update and delete operations, most of the pl could be auto generated. All that would be left to port would be special operations. These special operations could be moved to tcl procedures, which could perform the operation in tcl or simply wrap a pl function.

Defining (and redefining...) acs_objects as a view over many other tables just wont work, PG's native OO features are broken (and both would just move any performance problem to a different set of queries),  and if we're to go back to the ACS3.X way of doing things we should just start developing the old code base again.

I like the direction of Malte's second suggestion, and think the toolkit should be developed with this in mind:  our object system is important but not free;  acs_objects should be used extensively, but inheritence hierarchies should be kept short;  the core acs should aquire more features;  when the core lets us down it should be fixed in the core, not worked around in umpteen packages.

How about a web-based spell checker tied into the forms API so that arbitrary text fields could be tagged as "spell-checkable"?  I remember reading once on the old AD site that someone had written a cgi script to use the aspell library, but I never saw any code.

With all the work being done on dotLRN and CMS, it seems only natural that folks submitting home-work and/or content would appreciate this feature.

What PL/SQL does is to extend the methods that operate on the data, maybe simplifying the interface to the database. In the case of creating an acs_object, the __new procedure handles quite a bit of effort. If this effort were moved to the tcl layer, the porting effort would extend to every page that called the __new procedure.

Many, many PL/SQL functions in the toolkit are simple wrappers around simple DML. They really don't provide an abstraction layer for the most part. Some do of course and those that do aren't the ones I'm talking about.

Regardless ... I want to move to a higher-level description of OpenACS object types anyway. I get tired of writing code for both Oracle and PG that could be automatically generated for me.