Forum OpenACS Development: How OpenACS coding could get its groove back

What's made Open/ACS coding a bit of a love-hate thing is that we have
a rough balance between things that make coding a groove thing - quick
and easy success and feedback - and things that make it good long-term
engineering - reusability, encapsulation, and the other buzzwords.
It's always been a rough balance, and perhaps the early success was
sheer luck.  I don't know if there's a consistent long-term trend, but
I'm concerned that development by and for the high-end developers -
the very people who do the heavy lifting in the community - may be
making the groove harder to find for new entrants.  Maybe we can dig
up the aD forums and try to do some statistical analysis over the last
five or six years of the frequency and intensity of newbie
frustration vs joyful posts.

I'm not sure I have any useful
prescriptions for this beyond what we're already doing with
documentation and the tutorial - trying to make a clearer trail to the
groove.  A few people have made some great contributions to the
tutorial, such as Simon's automated testing intro and Jade's many
comments.  I'd love to see more pieces of tutorial get contributed.
Maybe people could pick a few of their own packages, that are in
production in the real world, and choose a few pages around which to
write tutorials for interesting techniques.  Or people could pick some
core modules and write tutorials explaining bits of them.  In terms of
API improvements, I'd like to see it even easier to build the basic
unit of a database-backed website: a sortable, filterable list of
records combined with one page per record add/edit screen.  I would
like to back away from the recommendation of xql for everything, and
yank it from the tutorial in favor of xql only for pages that need
it.  Making things ACS objects is more useful than even before, since
now you get not only permissions but also comments and notifications
almost for free; someday soon maybe categories; can we eliminate all of
that ugly, hard-to-debug, lots-of-work-with-little-positive-feedback,
groove-killing PLSQL code required to Objectify records?  Maybe code
generation?

Anyway.  food for thought, I hope.  Which parts of coding OpenACS
packages still provide groove?  Which really don't?

Joel, I agree 100% code needs to be easy to write. Actually why write it at all? For simple applications there is no need. Several efforts have been made toward this. There is some code in the core associated with groups. The CR has a different take on the same problem. Both auto generate pl code and data model.

As usual I disagree with both of these approaches for one major reason: you never write down the data model, so it can't be very useful. You never write down your pl code, so it can't be very easily debugged, and you can't add anything special.

So when I started using ACS4, I wrote a package to help me write Oracle packages. When I started using OpenACS and writing in pg, I needed to port code from Oracle. So I wrote a few procedures to make queries in pg as simple as in Oracle, that is the object.new type queries. The procedures are actually used in the cronjob package, with examples in the /www pages xql files.

Then I combined the two ideas into query-writer. It was a difficult to use package with no documentation, but it did exactly what I wanted it to do: write pl code and handle inserting and updating data.

Recently I upgraded the package to be more user friendly. Now all that is required to create a pg package is to write the data model. There is a wizard where you supply the name of the table, the primary key and the name of the 'object' you want this to be. The wizard rummages through the database and loads the query-writer tables with all the necessary data.

Another improvement is the almost complete elimination of pl code. From the information already provided, the pl code which creates the object type is written, plus an object_p function is created, and a drop script is written as well. All object create,update,delete pl is eliminated. I achieved this by just including tcl apis for acs_object__new, etc. This is very nice as when you realize you have forgotten a table attribute, you just add it without having to drop and resource the pl functions. None of this precludes you adding pl functions as before. But if additional code needs to run that can be based on the attributes inserted into the object, you can use a table trigger to get the same effect.

The tcl api for the new object still isn't written down, and I need to address this issue so that query-writer could be used just to generate apis.

On the downside I have never ported to Oracle. It does write the pl code for an oracle package, but the datamodel is for pg. Also there are 8 little tcl procs that need to be written to handle the generation of insert/update/delete in Oracle. This is probably a 3-5 day effort overall, but the package isn't feature complete, why port it?

Query-writer is undergoing some major changes at the moment. qw_groups are being tied to rel_segments. A qw_group is more like a role, an idea that is best supported by a rel_segment. Currently there are two groups: admin and the default group. Qw_groups encompass a set of objects+attributes+values+operations. If your application need more than two roles, you can now create a rel_segment and add the user to the rel_segment to give them the role. Another major change is support for file uploads and sub-attributes. If you object has an associated file, you can write a handler for the upload (examples exist). The handler could write the file to disk, or stuff it into an attribute field. Sub attributes were needed for dates. But sub attributes also required multi-attribute filters, to both combine and remove attributes. Until now query-write didn't support filters on more than a single attribute. Now there is a place to run filters or other code after all attributes have passed validation. Additional call back locations are being created as well.

Overall the query-writer form processing page acts like a generalized ad_page_contract. It enforces the same filters on objects and attributes whenever these are created or updated. (But note that query-writer also enforces a 'role contract')

Jade, if query-writer becomes feature complete and I port it to Oracle, that might be a good time to consider adding it to /contrib. Until then I think it mainly serves as an example that achieving a rapid development environment is possible, at least technically.

I've spent a lot of time recently looking at other development frameworks, mostly related to CMS products. Technically OpenACS is in the upper tier of available products. Philosophically OpenACS is nowhere. There is no coherient vision on the OpenACS way of doing things. For instance a lot of Java frameworks stress Model2 or MVC ( Model-View-Controller). OpenACS does a better job of this, I think, than what Java applications are achieving with JSPs or even taglibs. But we are not talking about this, or how it is being achieved, or not. It simply isn't on our language list.

Additionally, although I'm not an xml lover by any means. This is still an important technology. It would be a nice, useful feature if package ui created xml which could be transformed via xslt into the various formats needed. This is something else that doesn't seem to be on the radar screen.

Related to this, is the lack of discussion on how groups of developers, designers, and managers can work togeather to build and maintain a site. It is actually pretty easy to throw togeather an app on OpenACS, however maintaining it, or adapting it is not well covered. Since packages will sit around and be reused and updated for years, you would think this would be a huge concern to the OpenACS community. It shouldn't be a mystery why folks constantly create packages that do almost the same thing as other existing packages. You simply cannot maintain code when it is so hard to figure out why it was written the way it was. Again, this is mostly due to a lack of good, well understood philosophies for development with OpenACS.

OpenACS does have some very "groove"y things going for it. Administration is much easier than in ACS 3 days. Permissions is much slicker. Etc.

But coding is a little more difficult. In ACS 3 days, all you had to know was HTML, and SQL, and then learn some Tcl and maybe 10 API calls, and you were ready to go.

I understand why the decision was made to make things more complicated. But I'd like to think we can improve our groove a little.

- First of all, I think we need an OpenACS Wiki (now that Dave has provided this package) that lets us document in details how to use things. My docs aren't going to address every situation. Neither are Joel's. Or the official docs.

I think we then should move the tutorial section of the docs to there. That way, we can develop an easy reference for people who are getting up to speed.

I think a large part of the reason ACS was so easy was that Phil and Co. were providing so much documentation on how to do things. They wrote a book for anything that you needed to refer to, it seems like :)

I think we should also focus on finishing up the portions of the toolkit that are unfinished, like the form builder. And the documentation.

Unfortunately, most of us have to pay the bills, so this happens slower than package development.

But I'm optimistic about OpenACS' future. We're moving in a way that we weren't before. I think the groove is coming back 😊

At least the warning isn't as severe as ad_raise's

To a certain extent, an individual developer could be pulled along like a small car in the draft of a semi truck.  The closest thing to a central entity that has its lifeforce tied to the toolkit now is dotLRN which, in comparison, is a much more distributed project.

aD didn't just suck ass at the end.  It had problems from the get go, but the bigger it got, the more it sucked.

Most packages don't even have one maintainer 😟

Here's a unordered list of wishes I have:

Remove unnecessary layers of abstraction. We are still suffering from the Second System Effect of moving from ACS 3 to ACS 4.

So let's move code back to SQL statements. Especially DotLrn is filled to the brim with such constructs:

Retrieve a list of objects
For each item of this list do
  Get the object's owner's name
  Get the viewer permission record on this object
End for

all this nicely put into procs. Inhibits scalability, more code, more intricate, less readable.

I also wish we would put more intelligence back into the SQL layer. After all that is where we differ from similar toolkits and the fact that we are free to write "arbitrary" SQL statements ensures a reasonably high level of scalability.

No wrapper procs around SQL statements. I was told that I can access an arbitrary SQL statement in an arbitrary XQL file, so why the heck do I need a wrapper proc that does nothing but return a bitstream if I can run the SQL statement directly.

Best practices pages. That could be an ETP page for which we handle write permissions very liberally.

Have peer review by establishing a commit diff mailing list (Raised a bunch of time by myself). That's a light-weight way to enable eletronic programmer collaboration. (I'm willing and available to throw work at this)

Here are some problems ACS 4/OpenACS are supposed to solve:

From the kernel requirements documentation https://openacs.org/doc/openacs-4/object-system-requirements.html :

"A major goal in OpenACS 4 is to unify and normalize many of the core services of the system into a coherent common data model and API. In the past, these services were provided to applications in an ad-hoc and irregular fashion."

"A major goal in OpenACS 4 is to unify and normalize many of the core services of the system into a coherent common data model and API. In the past, these services were provided to applications in an ad-hoc and irregular fashion.""

So one goal is to increase code reuse and reduce ad-hom implementation of general services in every package.

And this:

"The OpenACS 4 Object Model must address five high-level requirements that repeatedly exhibit themselves in the context of existing services in OpenACS 3.x, as described below.

Object Identifiers for General Services

Support for Unified Access Control

Extensible Data Models

Modifiable Data Models

Generic Relations

"

I didn't want to copy the entire document in here. Go ahead and read that, and then we can decide how we are meeting these goals and if we agree they are still important.

I think they all still are important. Most of the 5 features are not well used besides object identified and unified access control.

---

One the ability to use a SQL query anywhere you want, you can still do that if you must. One of the criticisms of OpenACS 4.x is that it is too hard to get started. It has been suggested that having to learn pl/sql, in addition to tcl is one barrier to entry. By keeping most of the queries in tcl procs, we can simply the learning curve. This has to be balanced against performance of course. A tcl proc can still return mutiple rows from a query, so we don't need to do a seperate query for each row. The query inside a tcl proc should do whatever joins are necessary to return all the information just as efficiently as if you had embedded the query inside your tcl page.

return [db_string foo_query]

isn't adding anything besides making mantainance harder.

IMO the correct solution to "people don't know where to find the queries" is to have an xql browser, not to make everything a proc.

You seem to be suggesting that we should make OpenACS customizable by programmers that do not understand SQL and refuse to learn it. I strongly disagree. Direct access and manipulation of the database is in my opinion a fundamental design principle of OpenACS and a major reason for its strength.

There is a crucial difference between central services like the CR and frontend packages like forums. For the former it makes sense to simplify access by providing a tcl api in my opinion, since the same operations like item creation are used by many different packages, and the interface to these operations should thus not change. Still it should be possible to access the tables of the CR directly from a package and join against them in package specific queries for example.

Using tcl procs in front end packages like forums is a bad idea because it adds another level of abstraction that makes it harder to debug, develop and optimize.

One example from forums: there is a proc called forum::get which executes this query: "select forums_forums.* from forums_forums where forums_forums.forum_id = :forum_id" and returns the results in an array. Besides the fact that it is bad practice to use * instead of naming the fields to return - even if the query would list all fields of the forums_forums table that need to be returned, this way of doing things shows that this is inefficient. Look at message-post.tcl for example - it never needs any other value from the returned array than forums(name), so the query "select name from ..." would have been sufficient.

The wast of db resources becomes propably worse when the queries outsourced to tcl procs involve expensive joins etc.

And I don't see why code that requires me to look into yet another file and keep the whole stack of what calls what in mind, would be easier to understand or maintain.

So when the logic is in the DB, I think it's way easier to make sure that duplicate code doesn't end up being written.  It has always seemed to me that OpenACS was supposed to be a "thin" layer on top of a rich database schema.

And *huge* reason for that is in case you ever have to do some work with the data outside of OpenACS--say you need to have some other system get at and possibly change your data.  It's fairly easy to get an ODBC client or some other software (say a PHP front end, for whatever reason) to connect with the schema, be able to do permissions queries, create new objects, etc. You know you can get to what you need because all the logic isn't in Tcl.

I know, I know, the right answer would be "well, don't use X technology, make sure its a toolkit application", but we don't always have that luxury. 😊

Here's a code snippet ...

        # Form creation hack!

        ad_form -name new-user -export {
            {parent_id $some_parent_id}
            {name $some_name}
            {other_stuff $some_other_stuff}
            {content_type phb_person}
            {content_method no_content}
        }

        # create the photobook user record.
        set id [content::new_item new-user]
        cr::item::set_latest_revision_live -item_id $id

Boom! A new content item with its latest revision set live ... now from this hack to an extension to ad_form that properly allows one to integrate properly defined CR types is not a large step IMO.

As far as xql files are concerned...I'm very, very loath to give them up. In fact I'd love to (in the far future) only pass SQL directly in the db_* API if one uses an explicit -sql switch - only for those rare cases where we need to generate dynamic SQL inline. Of course it's fine for one-off packages or packages that will only support a single RDBMS to avoid .xql packages - but this doesn't define packages in the toolkit itself other than some of the probe-the-db-schema type package work that by its nature is 100% db dependent.

But creating objects shouldn't require writing PL/[pg]SQL and already doesn't, actually - there's a primitive object creator for properly defined objects and as I mention above CR items have much more capability (if they're defined properly).

Now currently you have to "properly define" such things in PL/[pl]SQL, notably the tedious defining of attributes for CR items, but this is yet one more thing that can be done via a TCL API in a declarative way simplifying things ...

i still get excited by the possibilities of the openacs, and there are a lot of damn smart people involved.  is there a groove?  do things flow easily?  not for me.  i agree it's partly due to circumstance, partly different visions and ideas.

as a new user to oacs, there are a million ways to do things and no way to tell which one is correct or to know if it will change as soon as you take the time to learn it.  i'm new to oacs but not web development -- the complication is learning another person's idea of the way things should be done.

one of my perpetual roadblocks is wondering why things are done the way they are and why they seem so complicated.  i obviously understand the necessity of certain complexities...  oacs installation and management is better than ever, the permissions system is wonderful.  however, some of the modularity and abstraction and objectification and all brings back bad memories of old employers and clients justifying futile efforts to rewrite sites in java.  yes, it is my responsibility to figure out what is going on -- but that is my point.  it can be difficult to do that.

i think the original acs was easier to absorb because development seemed to lean more towards providing acceptably-balanced standards and tools to solve human problems.  maybe the oacs concentrates more on providing the perfect solution to technical problems.  i don't know if considering whether developers should know sql or learn another database abstraction methodology is the right focus.

as far as my experience with the code or direction, my big bastard enemy is xml.  i understand the theory, i've found cool things to do with it, but i have yet to understand why it's worth the complexity.

i like the idea of bringing back bootcamps in some form.  why not deliver them through dotlrn?  use the tools being developed to teach the community.  what if the people voted on to the board of directors were encouraged (or required) to teach or contribute to a bootcamp?  ...just throwing some ideas around.

What's wrong with using object metadata to drive things?

As far as this comment goes:

"Differences from the cr include the fact that you write down your data model: a real table with real constraints."

I must say I'm at a loss to understand it.  When one customizes a content datatype one writes a real table or allows the attribute creation code to create the table for you, but in both cases your custom type has a "real table with real constraints" resulting.

As far as comments above regarding the fact that we have too many ways to do things with little documentation as to which are best practices and which are deprecated/frowned upon, this is absolutely true and something we should strive to fix.  All forms in the system should be built with the form builder and preferably ad_form, for instance, which in almost all cases greatly simplifies their creation and processing and makes easy the task of providing forms within a site with a uniform look and feel.  On and on with item after item we should standardize our approach.

But of course lots of people don't like the form builder, are uninterested in using the content repository, are uninterested in exploring the use of object metadata to automate the boring stuff, etc etc.

We have people who want to build square, pentagonal and hexagonal wheels as well as reinvent the traditional round wheels to be found in the toolkit.

So I'm not optimistic about our ability to settle on a handful of best practices used throughout the toolkit.

Query-writer can be used with the form builder. You just fill an array with the attributes/values and call the query writer proc.

The query-writer url api has built in per attribute validation. I'm currently adding multi-attribute validation. It uses the same filters as ad_page_contract. It also automatically does a length check on char/varchar attributes, although now I think there is an ad_page_contract filter that does the same thing.

Using the query-writer url api ensures that the same filters are applied every time an object is created or updated, I think with ad_form you can only ensure this is true for one form.

The query-writer url api is designed to make it easy to handle insert/update/delete of any number of objects in one pass. I'm sure you can do the same with ad_form, but you don't need to write any tcl code with query-writer.

Query writer has flags for each attribute, arranged by group. By default there are Admin and 'registered user' groups. These flags could be used to build forms which only show the attributes (and even only allowed attribute values) on a per group basis. So one form could serve both admin and regular users.

There is nothing to prevent someone from extending query writer with a generic form. Given the object name, operation, and group, you can relatively easily do this. It might be nice if you could pass in a subset of possible fields you want displayed. Query writer already has help fields on a per attribute basis that could be included as well when errors occur.

It is just my opinion that a written down data model is the best documentation of what is possible with a package. Trying to discern/write the data model by reading/writing a bunch of tcl or pl code which creates a table attribute by attribute is impossible for my small brain. I've written and used more generic methods of representing a data model, but they quickly become useless when it is time to figure out how to use it. I also like being able to drop a package and recreate it to see if it installs after I have made changes. Probably you can do this with the cr too.

Most of the stuff I work on isn't what I would consider a content application. But if I needed revisions, and whatever else is on the laundry list of features provided by the cr, I would still look at using it. Query writer is not a content manager.

Great work!  I better update mine to use /lib instead of /resources.  Also make a /test :)

Great work!

-Vamsee.

How this applies to a toolkit of multiple intergrated modules, I don't know ...

I'll have to go dig that code up.