Forum OpenACS Development: How OpenACS coding could get its groove back

It seems to me that the top incentive for volunteer coders to pick an
environment/toolkit/language is its groove.  I suspect I'm speaking
for a lot of other Open/ACSers when I say that I started using the ACS
toolkit because I got into the groove very easily.  I could code
working database-backed pages, and they worked, and when I thought of
an improvement, very often I could do it and see it in a few minutes
or less.  And backing this up, I could see that the groove wasn't a
dead end - that this effortless code had a fair amount of stability,
reusability, and other good stuff.  That I wasn't just making cheap
progress that would come back to haunt me, but that I was using a
well-engineered system.  (Or at least, a system with a lot of
promise.)  It was a long groove.  And that was a rare combination in
1998, and it's still rare or there wouldn't still be people committed
to this toolkit.

What's made Open/ACS coding a bit of a love-hate thing is that we have
a rough balance between things that make coding a groove thing - quick
and easy success and feedback - and things that make it good long-term
engineering - reusability, encapsulation, and the other buzzwords.
It's always been a rough balance, and perhaps the early success was
sheer luck.  I don't know if there's a consistent long-term trend, but
I'm concerned that development by and for the high-end developers -
the very people who do the heavy lifting in the community - may be
making the groove harder to find for new entrants.  Maybe we can dig
up the aD forums and try to do some statistical analysis over the last
five or six years of the frequency and intensity of newbie
frustration vs joyful posts.

I'm not sure I have any useful
prescriptions for this beyond what we're already doing with
documentation and the tutorial - trying to make a clearer trail to the
groove.  A few people have made some great contributions to the
tutorial, such as Simon's automated testing intro and Jade's many
comments.  I'd love to see more pieces of tutorial get contributed.
Maybe people could pick a few of their own packages, that are in
production in the real world, and choose a few pages around which to
write tutorials for interesting techniques.  Or people could pick some
core modules and write tutorials explaining bits of them.  In terms of
API improvements, I'd like to see it even easier to build the basic
unit of a database-backed website: a sortable, filterable list of
records combined with one page per record add/edit screen.  I would
like to back away from the recommendation of xql for everything, and
yank it from the tutorial in favor of xql only for pages that need
it.  Making things ACS objects is more useful than even before, since
now you get not only permissions but also comments and notifications
almost for free; someday soon maybe categories; can we eliminate all of
that ugly, hard-to-debug, lots-of-work-with-little-positive-feedback,
groove-killing PLSQL code required to Objectify records?  Maybe code
generation?

Anyway.  food for thought, I hope.  Which parts of coding OpenACS
packages still provide groove?  Which really don't?

Collapse
Posted by Andrei Popov on
Joel,

You almost read my mind.  This is exactly the feeling I used to have when I started my acquiantence with ACS.  It was not in 1998, but it was in late 2000/early 2001 -- in the 3.x era.

It was easy to join in -- and it was a snap to put together a few useful aps guided largely by aD forums and Phil's books.

I am not a professional developer -- I am more of a tinkerer, who normally skratches only his own itch.  Since the day job has very little to do with programming -- these itches rarely applicable to others.

I'll give you a sample of my nightmares -- I wanted to make a few changes to Bugtracker.  It's a great piece of work -- I use it a lot as an issue tracker (with a few additional summary pages that are used to publish PDF reports for PHBs).  But when I looked inside -- I got simply lost.  There's so much *under* the hood.  And deeper you go -- the more there is.

This is not meant to critisize Lars' work -- like I said -- it is great, work out of the box, etc.  But is it easy to understand all calls in/out?  What is stored in bt_bugs vs. what is in cr_revisions, cr_keywords or God knows where else?  No.

I'm sure there are other examples like this -- and I agree that there are so much fewer "joyful newbie posts" on this forum (though there are also quite a few angry ones for which an "itch" is to reply "RTFM" in 1337 way).  The question is whether this is a sign of toolkit maturity?  Or just the fact that it is not coherent/finished/stil growing?

Collapse
Posted by Tom Jackson on

Joel, I agree 100% code needs to be easy to write. Actually why write it at all? For simple applications there is no need. Several efforts have been made toward this. There is some code in the core associated with groups. The CR has a different take on the same problem. Both auto generate pl code and data model.

As usual I disagree with both of these approaches for one major reason: you never write down the data model, so it can't be very useful. You never write down your pl code, so it can't be very easily debugged, and you can't add anything special.

So when I started using ACS4, I wrote a package to help me write Oracle packages. When I started using OpenACS and writing in pg, I needed to port code from Oracle. So I wrote a few procedures to make queries in pg as simple as in Oracle, that is the object.new type queries. The procedures are actually used in the cronjob package, with examples in the /www pages xql files.

Then I combined the two ideas into query-writer. It was a difficult to use package with no documentation, but it did exactly what I wanted it to do: write pl code and handle inserting and updating data.

Recently I upgraded the package to be more user friendly. Now all that is required to create a pg package is to write the data model. There is a wizard where you supply the name of the table, the primary key and the name of the 'object' you want this to be. The wizard rummages through the database and loads the query-writer tables with all the necessary data.

Another improvement is the almost complete elimination of pl code. From the information already provided, the pl code which creates the object type is written, plus an object_p function is created, and a drop script is written as well. All object create,update,delete pl is eliminated. I achieved this by just including tcl apis for acs_object__new, etc. This is very nice as when you realize you have forgotten a table attribute, you just add it without having to drop and resource the pl functions. None of this precludes you adding pl functions as before. But if additional code needs to run that can be based on the attributes inserted into the object, you can use a table trigger to get the same effect.

The tcl api for the new object still isn't written down, and I need to address this issue so that query-writer could be used just to generate apis.

On the downside I have never ported to Oracle. It does write the pl code for an oracle package, but the datamodel is for pg. Also there are 8 little tcl procs that need to be written to handle the generation of insert/update/delete in Oracle. This is probably a 3-5 day effort overall, but the package isn't feature complete, why port it?

Query-writer is undergoing some major changes at the moment. qw_groups are being tied to rel_segments. A qw_group is more like a role, an idea that is best supported by a rel_segment. Currently there are two groups: admin and the default group. Qw_groups encompass a set of objects+attributes+values+operations. If your application need more than two roles, you can now create a rel_segment and add the user to the rel_segment to give them the role. Another major change is support for file uploads and sub-attributes. If you object has an associated file, you can write a handler for the upload (examples exist). The handler could write the file to disk, or stuff it into an attribute field. Sub attributes were needed for dates. But sub attributes also required multi-attribute filters, to both combine and remove attributes. Until now query-write didn't support filters on more than a single attribute. Now there is a place to run filters or other code after all attributes have passed validation. Additional call back locations are being created as well.

Overall the query-writer form processing page acts like a generalized ad_page_contract. It enforces the same filters on objects and attributes whenever these are created or updated. (But note that query-writer also enforces a 'role contract')

Collapse
Posted by Jade Rubick on
Tom, I think your query-writer is a very interesting project. I don't have time to look at it right now, but when it is in a state that you think even beginners could use, please put it in /contrib.

I'd like to take a look at it at some point.

I think a tutorial showing what it can do would convince a lot more people to use it.

Maybe we can put it on a Wiki once we get that set up. :)

Collapse
Posted by Tom Jackson on

Jade, if query-writer becomes feature complete and I port it to Oracle, that might be a good time to consider adding it to /contrib. Until then I think it mainly serves as an example that achieving a rapid development environment is possible, at least technically.

I've spent a lot of time recently looking at other development frameworks, mostly related to CMS products. Technically OpenACS is in the upper tier of available products. Philosophically OpenACS is nowhere. There is no coherient vision on the OpenACS way of doing things. For instance a lot of Java frameworks stress Model2 or MVC ( Model-View-Controller). OpenACS does a better job of this, I think, than what Java applications are achieving with JSPs or even taglibs. But we are not talking about this, or how it is being achieved, or not. It simply isn't on our language list.

Additionally, although I'm not an xml lover by any means. This is still an important technology. It would be a nice, useful feature if package ui created xml which could be transformed via xslt into the various formats needed. This is something else that doesn't seem to be on the radar screen.

Related to this, is the lack of discussion on how groups of developers, designers, and managers can work togeather to build and maintain a site. It is actually pretty easy to throw togeather an app on OpenACS, however maintaining it, or adapting it is not well covered. Since packages will sit around and be reused and updated for years, you would think this would be a huge concern to the OpenACS community. It shouldn't be a mystery why folks constantly create packages that do almost the same thing as other existing packages. You simply cannot maintain code when it is so hard to figure out why it was written the way it was. Again, this is mostly due to a lack of good, well understood philosophies for development with OpenACS.

Collapse
Posted by Tilmann Singer on
I still find coding on OpenACS relatively groovy. It's easy to create a package, write a datamodel and do relatively sophisticated things quickly.

One feature that makes that a lot more efficient and still flexible is ad_form, once you got used to it.

Of course I don't use xql files or i18n-ize my code from the beginning, and to be honest I'm also reluctant to create acs_objects unless I'm 100% sure I'll need them.

And I don't like to create tcl api's for stuff that accesses the database. If some data from the db is only needed on one page, and slightly different on another page, what's the point in generalising this into an exra abstraction? One groovy factor of OpenACS coding is to write an sql query or dml directly into a tcl file, reload page and voila - changes are in effect. Instead of changing tcl file here, change call of tcl proc there, select file to be watched etc. etc.

I'm not saying there is no point in tcl apis (and I'm well aware DaveB will come over now and beat me for this), but they should only be used for operations that really are used in many places. Otherwise they decrease the groovyness of OpenACS which comes a lot from the fact that it directly deals with the database.

Lack of philosophy: I think that OpenACS has still a good foundation in some very sound principles, which I found to be well described in Phil's book, such as the closeness to the database, two-tier architecture etc. And a bunch of best-practices that have developed over time and in sum make out the quality of OpenACS sites such as abstract urls, inline display of form errors, redirect to display page after an update with POST, etc. etc. These are not as famous as big paradigms like MVC, but they have proven to work very well in the world of web apps.

Collapse
Posted by Jade Rubick on
I agree with you Joel. It was pretty easy to get up and running in ACS 3 days. It was very easy to get going.

OpenACS does have some very "groove"y things going for it. Administration is much easier than in ACS 3 days. Permissions is much slicker. Etc.

But coding is a little more difficult. In ACS 3 days, all you had to know was HTML, and SQL, and then learn some Tcl and maybe 10 API calls, and you were ready to go.

I understand why the decision was made to make things more complicated. But I'd like to think we can improve our groove a little.

- First of all, I think we need an OpenACS Wiki (now that Dave has provided this package) that lets us document in details how to use things. My docs aren't going to address every situation. Neither are Joel's. Or the official docs.

I think we then should move the tutorial section of the docs to there. That way, we can develop an easy reference for people who are getting up to speed.

I think a large part of the reason ACS was so easy was that Phil and Co. were providing so much documentation on how to do things. They wrote a book for anything that you needed to refer to, it seems like :)

I think we should also focus on finishing up the portions of the toolkit that are unfinished, like the form builder. And the documentation.

Unfortunately, most of us have to pay the bills, so this happens slower than package development.

But I'm optimistic about OpenACS' future. We're moving in a way that we weren't before. I think the groove is coming back 😊

Collapse
Posted by Alfred Werner on
Everything went downhill when util_PrettySexManWoman got deprecated. These days it just generates a warning. Sheesh.
Collapse
Posted by Vinod Kurup on
Everything went downhill when util_PrettySexManWoman got deprecated. These days it just generates a warning. Sheesh.

At least the warning isn't as severe as ad_raise's

Collapse
Posted by Alfred Werner on
LMAO !!
Collapse
Posted by Walter McGinnis on
Don't forget the momentum of having a big number of aD programmers working on client projects which were then rolled back into the toolkit, for better or worse.  How many fulltime programmers worked for aD at its peak?

To a certain extent, an individual developer could be pulled along like a small car in the draft of a semi truck.  The closest thing to a central entity that has its lifeforce tied to the toolkit now is dotLRN which, in comparison, is a much more distributed project.

Collapse
Posted by Malte Sussdorff on
I just read the latest column of "I, Cringely", which talks about the different view of M$ and Linus concerning the coding abilities of a company vs. many distributed developers. At the peak of it's time AD had I'd say around 100 coders working on ACS TCL in client projects. They did not care that much anymore about what the outside world was doing, but drove their truck at a speed defined by marketing and project management (aka. business guys). Though a momentum like this is worthwhile it bears the risk of releasing code when it is not ready to be released. Which, in effect, alienates other developers because of the crappy maturity of the code, regardless how easy it is to get going with it. So I do not think that the existance of aD contributed to the groove of ACS. The existance of Philip, the marketing machinery and the fun environment of people with a huge ego, but willingness to help (as part of their ego) did attract some people to ACS, more than the existance of AD.

Now, looking at this community, I see we are releasing well tested code that has put a lot of thinking behind it. What we lack though, is to release the knowledge, how to use the code, as the methods got considerably more sophisticated. Furthermore, what happened to the concept of Bootcamps. I know, there is no AD anymore, that could run a bootcamp on it's own (well, okay, we do it occasionally, but that does not really count, as we did not open them up to the public so far), but why did this stop. Furthermore, how many universities are using dotLRN? How many are teaching how to use it?

Getting fresh people into the toolkit through a university course or a bootcamp will give you a lot of feedback, how to make it easier to develop with OpenACS. Having a teaching forum for exchange between bootcampers and university students across bootcamps or courses, would be a good place to collect all this feedback, which in return could make it back into the documentation effort and maybe even into improvements for the toolkit.

And would it be too much to ask for developers in the community, to commit one week of their time to teach a bootcamp? Or give a class at a university next to them.

One of the strengthes as Walter pointed out, is dotLRN. But not necessarily only because of the (monetary) lifeforce (though it looks like that most developement nowadays is dotLRN driven, at least if you look at hours spend on development), but because of it's potential to attract new developers and become a standard in the world. I found it very intersting that the language with the most translations in the latest CVS checkout is Arabic. I doubt Blackboard or WebCT would have an easy sell in the Arabic markets nowadays.

Collapse
Posted by Venkatesh Goteti on
For those of you interested in Cringely's article, you can find it at http://www.pbs.org/cringely/pulpit/pulpit20031023.html . I found it on a blog posting here.
Collapse
Posted by Walter McGinnis on
I'm just making a point for historical context, not disparaging OpenACS or anyone else for that matter.

<blockquote>> So I do not think that the existance of aD contributed to the groove of ACS.
</blockquote>

This is a big overstatement.  Just because aD sucked ass at the end, doesn't mean that its existance didn't contribute to the groove of ACS.  As you point out, aD put on Bootcamps, sometimes weekly in three to four locations around the world.  On top of that, there was aD University, the bboards, CVS browsing, etc.  In other words a lot of resources that independent developers could take advantage of.  I think that definitely contributed to the ACS groove.

I count myself as one of the more bitter former employees of aD, but I must admit that we can thank aD for making a lot of those resources open source.  An important difference between aD and M$.

Walter

Collapse
Posted by Walter McGinnis on
Oh yeah, for historical context...

aD didn't just suck ass at the end.  It had problems from the get go, but the bigger it got, the more it sucked.

Collapse
Posted by Torben Brosten on
I learned much from aD's way and still think that something like it could work now, if only founding principles would include rejecting external investments to the company --investers could almost as easily invest in a project that uses the company resources instead.

Part of the aD way, included having two sw developers per package, which helped increase documenting in code etc.  Are there two package keepers for each Openacs package now? If not, maybe efforts should be made to increase participation?

Collapse
Posted by Dirk Gomez on

Most packages don't even have one maintainer 😟

Here's a unordered list of wishes I have:

Remove unnecessary layers of abstraction. We are still suffering from the Second System Effect of moving from ACS 3 to ACS 4.

So let's move code back to SQL statements. Especially DotLrn is filled to the brim with such constructs:

Retrieve a list of objects
For each item of this list do
  Get the object's owner's name
  Get the viewer permission record on this object
End for

all this nicely put into procs. Inhibits scalability, more code, more intricate, less readable.

I also wish we would put more intelligence back into the SQL layer. After all that is where we differ from similar toolkits and the fact that we are free to write "arbitrary" SQL statements ensures a reasonably high level of scalability.

No wrapper procs around SQL statements. I was told that I can access an arbitrary SQL statement in an arbitrary XQL file, so why the heck do I need a wrapper proc that does nothing but return a bitstream if I can run the SQL statement directly.

Best practices pages. That could be an ETP page for which we handle write permissions very liberally.

Have peer review by establishing a commit diff mailing list (Raised a bunch of time by myself). That's a light-weight way to enable eletronic programmer collaboration. (I'm willing and available to throw work at this)

Collapse
Posted by Dave Bauer on
Dirk,

I am not sure I am against a Tcl proc that returns the results of a query. The main reason is that I know where to go to find the query.

Besides that let's discuss the reasons for OpenACS 4. What problems was it meant to solve. After that, let's see if those are still the goals of OpenACS. Then we can decide where it is sucessful and where we can make improvements.

Collapse
Posted by Dave Bauer on
Replying to my own message:

Here are some problems ACS 4/OpenACS are supposed to solve:

From the kernel requirements documentation https://openacs.org/doc/openacs-4/object-system-requirements.html :

"A major goal in OpenACS 4 is to unify and normalize many of the core services of the system into a coherent common data model and API. In the past, these services were provided to applications in an ad-hoc and irregular fashion."

"A major goal in OpenACS 4 is to unify and normalize many of the core services of the system into a coherent common data model and API. In the past, these services were provided to applications in an ad-hoc and irregular fashion.""

So one goal is to increase code reuse and reduce ad-hom implementation of general services in every package.

And this:

"The OpenACS 4 Object Model must address five high-level requirements that repeatedly exhibit themselves in the context of existing services in OpenACS 3.x, as described below.

Object Identifiers for General Services

Support for Unified Access Control

Extensible Data Models

Modifiable Data Models

Generic Relations

"

I didn't want to copy the entire document in here. Go ahead and read that, and then we can decide how we are meeting these goals and if we agree they are still important.

I think they all still are important. Most of the 5 features are not well used besides object identified and unified access control.

---

One the ability to use a SQL query anywhere you want, you can still do that if you must. One of the criticisms of OpenACS 4.x is that it is too hard to get started. It has been suggested that having to learn pl/sql, in addition to tcl is one barrier to entry. By keeping most of the queries in tcl procs, we can simply the learning curve. This has to be balanced against performance of course. A tcl proc can still return mutiple rows from a query, so we don't need to do a seperate query for each row. The query inside a tcl proc should do whatever joins are necessary to return all the information just as efficiently as if you had embedded the query inside your tcl page.

Collapse
Posted by Jun Yamog on
Hi,

I am not totally against Dirk suggestion to on not to use tcl proc to do database work.  But I took the other route of using tcl proc to do database work.  Upon getting some suggestion and indication that is the path to go.  I decided to go that way.  Here are the pros and cons I have encountered:

- makes it life harder for the one making the tcl proc.  You have to think about what to make and how to present it.
+ makes life a lot easier for those who will use.  they don't have to know the lower layers.
+ make one object manipulation a lot easier. that is add, edit and delete.
- makes listing of objects harder.  I have actually decided that the listing of objects be only used for preliminary development or basic behaviour.  the end developer should just use the query as reference and create his own select based from the reference query.
+ optimizing or bug fixing becomes easier, as once the tcl proc is fixed.  code that uses it get fixed too.
- as more people/code use it.  you will have to more and more worry about compatibility.
- added burden for you to review your code.  which tcl proc to merge, split, bad presentation, etc.

Anyway I hope the following info may help both group people looking at moving db work to tcl AND moving db work away from tcl.

I think given that there are many of us, its likely we will have different opinions.  The OCT should be able to give direction what is best of OpenACS in general.

I am not yet completely sold to the idea of moving db work to tcl proc.  But so far I think for my current use despite the negatives, I value the positives more from what I got from it.

Collapse
Posted by Jonathan Ellis on
I think Dirk's point is well-taken, though, that a proc that simply does

return [db_string foo_query]

isn't adding anything besides making mantainance harder.

IMO the correct solution to "people don't know where to find the queries" is to have an xql browser, not to make everything a proc.

Collapse
Posted by Dave Bauer on
There are conflicting goals. Allowing easier access for experienced programmers and a shallower learning curve.

I definitely want to be able to resuse queries. The reason I have moved them to a tcl proc is I find I am using the same query in every page of a package.

I don't see how having a query in a tcl proc makes maintenance harder at all. Just different. The concept is that someone who wants to modify a package could make changes in the tcl/adp pages without diving too deeply into the database.

Are there examples of code that is difficult to maintain because of queries in tcl procedures?

Collapse
Posted by Tilmann Singer on
___By keeping most of the queries in tcl procs, we can simply the learning curve. This has to be balanced against performance of course. A tcl proc can still return mutiple rows from a query, so we don't need to do a seperate query for each row. The query inside a tcl proc should do whatever joins are necessary to return all the information just as efficiently as if you had embedded the query inside your tcl page.___

You seem to be suggesting that we should make OpenACS customizable by programmers that do not understand SQL and refuse to learn it. I strongly disagree. Direct access and manipulation of the database is in my opinion a fundamental design principle of OpenACS and a major reason for its strength.

There is a crucial difference between central services like the CR and frontend packages like forums. For the former it makes sense to simplify access by providing a tcl api in my opinion, since the same operations like item creation are used by many different packages, and the interface to these operations should thus not change. Still it should be possible to access the tables of the CR directly from a package and join against them in package specific queries for example.

Using tcl procs in front end packages like forums is a bad idea because it adds another level of abstraction that makes it harder to debug, develop and optimize.

One example from forums: there is a proc called forum::get which executes this query: "select forums_forums.* from forums_forums where forums_forums.forum_id = :forum_id" and returns the results in an array. Besides the fact that it is bad practice to use * instead of naming the fields to return - even if the query would list all fields of the forums_forums table that need to be returned, this way of doing things shows that this is inefficient. Look at message-post.tcl for example - it never needs any other value from the returned array than forums(name), so the query "select name from ..." would have been sufficient.

The wast of db resources becomes propably worse when the queries outsourced to tcl procs involve expensive joins etc.

And I don't see why code that requires me to look into yet another file and keep the whole stack of what calls what in mind, would be easier to understand or maintain.

Collapse
Posted by Jeff Davis on
I think most dml should have a tcl wrapper function (permission::grant for example) and for display queries the reuse should be via includable .tcl/.adp pairs. Of course, no rigid rules like that work in all circumstances...

A big advantage with having dml in tcl procs is that it is much simpler to write unit tests and in most cases the dml is for a single object anyway. I would agree that for things where you are operating on a collection of objects you probably will want to do it directly in sql rather than create a tcl wrapper function (although even there I think an api that takes a sql fragment or list would be useful and more easily tested).

Collapse
Posted by C. R. Oldham on
However, I like the fact that so much logic is in the database--this was part of Philip's original design.  He would always say "make the database do the work if you can".

So when the logic is in the DB, I think it's way easier to make sure that duplicate code doesn't end up being written.  It has always seemed to me that OpenACS was supposed to be a "thin" layer on top of a rich database schema.

And *huge* reason for that is in case you ever have to do some work with the data outside of OpenACS--say you need to have some other system get at and possibly change your data.  It's fairly easy to get an ODBC client or some other software (say a PHP front end, for whatever reason) to connect with the schema, be able to do permissions queries, create new objects, etc. You know you can get to what you need because all the logic isn't in Tcl.

I know, I know, the right answer would be "well, don't use X technology, make sure its a toolkit application", but we don't always have that luxury. 😊

Collapse
Posted by Dave Bauer on
Tilmann,

Excellent points.

Application coding should have different priorities than a service package which function is to provide an API.

We seem to be having this same discussion among the same folks :)

So I think we all agree that efficiency is of top priority as well as maintainability of code.

Right  now there is no easy way to resue queries except by encapsulating them in a Tcl procedure.

We are in agreement that often you just need the name of one value out of the database, and an additonal layer of abstraction gets in the way.

Does that sound right?

This might change if more packages used more central services to create and manipulate acs_objects. One example is the content repository. Getting a list of items is not the best in a tcl procedure, but for common manipulation of one object at a time it seems to work.

Collapse
Posted by Dave Bauer on
Ack, I posted before I saw Jeff's reply. I agree with Jeff, and will slightly modify my answer because unit testing is very helpful. We definitely need to do more of that.
Collapse
Posted by Ola Hansson on
The reasons why I personally like the idea of Tcl procs for simple queries is the same as why I like reusability in general, i.e., less coding and increased maintainability. I fail to see how maintaining one Tcl proc and its associated common .xql file (or separate PG and Oracle *.xql files) can be more work than maintaining any number of clones of the same *.xql file(s) in all places they occur within a package (say) ...

Having said that, I am certainly not promoting the use of a Tcl API which calls something like "forum::get" in a foreach loop - those fields should, of course, be retreived from a separate query. OTOH, just because the previous is true doesn't mean "forum::get" has no use ... it might be useful in other places.

Also, we have a mechanism called the query dispatcher for a reason; the ability to support multiple dbs. Let's not go out and tell people that it's fine to skip this mechanism just because it's possible to do without it, or at least bypass it (by writing the queries inline). Let's instead encourage people to uniformly adhere to the standards we have once set out to follow!

As has been said elsewhere; we need fewer ways to achieve common tasks in this toolkit, not more ... What we need more of, though, is coding *standards* - not something vague like "best practices"!! This is how we lower the learning curve, by telling people explicitly how things are done.

What's most convenient/easy for an experienced OACS developer ought to also be easiest for a newbie, no?

/Ola

Collapse
Posted by Ola Hansson on
I was wrong when I said that inline queries bypass or override .xql files - they don't. However, I still think it's bad practice to leave them there after the .xql files have been written, since they can be missleading if not maintained and they only reflect the query of one db.
Collapse
Posted by Dirk Gomez on

Neither Til nor I want to copy the same (or similar) SQL query allover the place. As Til notes: OpenACS need to know SQL. They need to know it better than TCL in fact.

If all SQL queries were moved out into the XQL files we would have something like a database "persistence layer". Having those queries in their own respective files would make us look at similarities and probably reduce the number of different queries and make the toolkit and the programmer more scalable.

So can the query dispatcher pick a query from a randomly named file or not?

(Coding SQL is more groovy than coding TCL btw :)

I agree that DML should have a wrapper to support unit testing. But we don't need a wrapper proc on both TCL and SQL...and there is a unit testing package for Oracle SQL (by Steven Feuerstein - utlplsql or sth similar). TCL procs are easier to debug, PL/SQL procs are easier for migrations. Oh whoever thought that PL/SQL is easier to port to other databases - I guess he/they were wrong ...

Collapse
Posted by Don Baccus on
We are much closer to being able to support rapid development than many believe. While Tom's query writer is one way to go, we can already generate content almost automatically from PROPERLY DECLARED content types (not general object types).

Here's a code snippet ...


        # Form creation hack!

        ad_form -name new-user -export {
            {parent_id $some_parent_id}
            {name $some_name}
            {other_stuff $some_other_stuff}
            {content_type phb_person}
            {content_method no_content}
        }

        # create the photobook user record.
        set id [content::new_item new-user]
        cr::item::set_latest_revision_live -item_id $id

Boom! A new content item with its latest revision set live ... now from this hack to an extension to ad_form that properly allows one to integrate properly defined CR types is not a large step IMO.

As far as xql files are concerned...I'm very, very loath to give them up. In fact I'd love to (in the far future) only pass SQL directly in the db_* API if one uses an explicit -sql switch - only for those rare cases where we need to generate dynamic SQL inline. Of course it's fine for one-off packages or packages that will only support a single RDBMS to avoid .xql packages - but this doesn't define packages in the toolkit itself other than some of the probe-the-db-schema type package work that by its nature is 100% db dependent.

But creating objects shouldn't require writing PL/[pg]SQL and already doesn't, actually - there's a primitive object creator for properly defined objects and as I mention above CR items have much more capability (if they're defined properly).

Now currently you have to "properly define" such things in PL/[pl]SQL, notably the tedious defining of attributes for CR items, but this is yet one more thing that can be done via a TCL API in a declarative way simplifying things ...

Collapse
Posted by Tom Jackson on

Just a note about query-writer: it applies to any database table. If the primary key refers to the acs_object table, you can use it to create new object types. Differences from the cr include the fact that you write down your data model: a real table with real constraints. A wizard is used to create the information necessary for query-writer to work. You just supply the table name and the primary key. You don't have to write any tcl or pl code in order to define your object type, that is one reason it is called query-writer! Once you have an object defined in query-writer you can use a tcl api to create new objects:


set object_array(title) $title
set object_array(content) $content

# create the object:

qw_new object_name object_array

Of course you wouldn't even need to do that if you use the supplied url api. This page, called qw.tcl takes form variables and processes them to create a series of qw_* statements. Processing includes running filters you specify on each attribute, and the newest version uploads files, runs multi-attribute filters and allows other callbacks to be executed. So for instance, the above title and content would probably come from a form. The form (this is all you would need to write) could be:


<form action=/qw/qw method=post>
<input type=hidden name=return_url value=@some_url@>

<input type=text name=new.object_name.title.1>
<br \>
<textarea name=new.object_name.content.1 ></textarea>

</form>

So once you have setup the object in query-writer, you can turn over the form entry design to someone else without worrying about them messing around with tcl and database stuff.

Granted, I still believe query-writer is difficult to get used to, and if you have a small data model, one or two tables, it probably isn't worth the hassle, this is the way with most tools.

Collapse
Posted by bill kellerman on
i was initially attracted to the acs through greenspun, the book, and the charisma of arsdigita.  i worked in a crappy, stereotypical dot-com that did everything wrong, and what appeared to be a clearly-defined vision of "do the most with the least and keep your perspective" was exciting.

i still get excited by the possibilities of the openacs, and there are a lot of damn smart people involved.  is there a groove?  do things flow easily?  not for me.  i agree it's partly due to circumstance, partly different visions and ideas.

as a new user to oacs, there are a million ways to do things and no way to tell which one is correct or to know if it will change as soon as you take the time to learn it.  i'm new to oacs but not web development -- the complication is learning another person's idea of the way things should be done.

one of my perpetual roadblocks is wondering why things are done the way they are and why they seem so complicated.  i obviously understand the necessity of certain complexities...  oacs installation and management is better than ever, the permissions system is wonderful.  however, some of the modularity and abstraction and objectification and all brings back bad memories of old employers and clients justifying futile efforts to rewrite sites in java.  yes, it is my responsibility to figure out what is going on -- but that is my point.  it can be difficult to do that.

i think the original acs was easier to absorb because development seemed to lean more towards providing acceptably-balanced standards and tools to solve human problems.  maybe the oacs concentrates more on providing the perfect solution to technical problems.  i don't know if considering whether developers should know sql or learn another database abstraction methodology is the right focus.

as far as my experience with the code or direction, my big bastard enemy is xml.  i understand the theory, i've found cool things to do with it, but i have yet to understand why it's worth the complexity.

i like the idea of bringing back bootcamps in some form.  why not deliver them through dotlrn?  use the tools being developed to teach the community.  what if the people voted on to the board of directors were encouraged (or required) to teach or contribute to a bootcamp?  ...just throwing some ideas around.

Collapse
Posted by Tilmann Singer on
Regarding xql files I meant to say that I don't use them in the beginning when prototyping a package or when I'm working on a specific package for a client that will always use one db only and is not intended to be redistributed, that's all. The ability to write inline queries makes developing a little quicker but of course for general packages xql files should be used. Btw, making inline queries explicit with an -sql switch sounds great and would eliminate lots of confusion for newcomers too.

Ola, forum::get has the other drawback of being slightly inefficient by selecting more fields than needed, propably it doesn't matter in this case at all, but it's symptomatic for the dangers of redirection via tcl apis for queries. I would prefer using something like "set forum_name [db_string get_forum_name {}]" instead, possibly with a reused xql file if that query is used more often.

Collapse
Posted by Don Baccus on
From your example it looks like the use of the query writer as you suggest also removes the usage of the templated form builder.  This is, I do believe, a very bad idea.  Having worked with a large site built with custom built-by-hand forms which saw the .adp files shrink by a factor of (I kid you not) roughly 500 times when moved to the use of a single form builder template shared among all forms, I will never go back to writing my own input tags again.  Ditto for giving up the kind of validation made easy by the form builder/ad_form.

What's wrong with using object metadata to drive things?

As far as this comment goes:

"Differences from the cr include the fact that you write down your data model: a real table with real constraints."

I must say I'm at a loss to understand it.  When one customizes a content datatype one writes a real table or allows the attribute creation code to create the table for you, but in both cases your custom type has a "real table with real constraints" resulting.

As far as comments above regarding the fact that we have too many ways to do things with little documentation as to which are best practices and which are deprecated/frowned upon, this is absolutely true and something we should strive to fix.  All forms in the system should be built with the form builder and preferably ad_form, for instance, which in almost all cases greatly simplifies their creation and processing and makes easy the task of providing forms within a site with a uniform look and feel.  On and on with item after item we should standardize our approach.

But of course lots of people don't like the form builder, are uninterested in using the content repository, are uninterested in exploring the use of object metadata to automate the boring stuff, etc etc.

We have people who want to build square, pentagonal and hexagonal wheels as well as reinvent the traditional round wheels to be found in the toolkit.

So I'm not optimistic about our ability to settle on a handful of best practices used throughout the toolkit.

Collapse
Posted by bill kellerman on
don,

it's probably too hard to get everyone to agree on a single iron-handed rule deciding the way things should be.  everyone has their particular best way of doing things.  get the community to at least agree on what is best, but always allow for and accept reasonable variations.

the problem is actually motivating people to want to follow the agreed standard and not stray.  i made the following suggestion in the top ten oacs priorities thread:

"require packages maintainers to document the scope, requirements, and status of their packages.  the community should "certify" packages (ie. a community-recommended package) by reviewing quality, completeness, qa-testing and value.  require package maintainers to document and communicate the status of their packages to the community or risk losing certification.  clearly document packages that are not certified, and what the ramifications are for using them.  encourage users to get their packages certified."

with a system somewhere along these lines (maybe applied more generally), you can pretty much tell everyone that if they want to receive the community stamp of approval and a gif of a big gold star next to links to their code, then follow the guidelines.  if they don't get the stamp, they risk their packages and code not being used or supported.  keeping a stamp of approval will also motivate people to keep up with their code and make sure it meets evolving standards.

Collapse
Posted by Tom Jackson on

Query-writer can be used with the form builder. You just fill an array with the attributes/values and call the query writer proc.

The query-writer url api has built in per attribute validation. I'm currently adding multi-attribute validation. It uses the same filters as ad_page_contract. It also automatically does a length check on char/varchar attributes, although now I think there is an ad_page_contract filter that does the same thing.

Using the query-writer url api ensures that the same filters are applied every time an object is created or updated, I think with ad_form you can only ensure this is true for one form.

The query-writer url api is designed to make it easy to handle insert/update/delete of any number of objects in one pass. I'm sure you can do the same with ad_form, but you don't need to write any tcl code with query-writer.

Query writer has flags for each attribute, arranged by group. By default there are Admin and 'registered user' groups. These flags could be used to build forms which only show the attributes (and even only allowed attribute values) on a per group basis. So one form could serve both admin and regular users.

There is nothing to prevent someone from extending query writer with a generic form. Given the object name, operation, and group, you can relatively easily do this. It might be nice if you could pass in a subset of possible fields you want displayed. Query writer already has help fields on a per attribute basis that could be included as well when errors occur.

It is just my opinion that a written down data model is the best documentation of what is possible with a package. Trying to discern/write the data model by reading/writing a bunch of tcl or pl code which creates a table attribute by attribute is impossible for my small brain. I've written and used more generic methods of representing a data model, but they quickly become useless when it is time to figure out how to use it. I also like being able to drop a package and recreate it to see if it installs after I have made changes. Probably you can do this with the cr too.

Most of the stuff I work on isn't what I would consider a content application. But if I needed revisions, and whatever else is on the laundry list of features provided by the cr, I would still look at using it. Query writer is not a content manager.

Collapse
Posted by Joel Aufrecht on
I've updated the tutorial (partially) and diagrammed how ACS templating, template::list, the content_repository, ad_form, and automated testing interact. OpenACS coding at a (long) glance.
Collapse
Posted by Jun Yamog on
Hi Joel,

Great work!  I better update mine to use /lib instead of /resources.  Also make a /test :)

Great work!

Collapse
Posted by Joel Aufrecht on
Branimir and I came up with the tongue-in-cheek assertion that ACS has many fewer contributions and shared libraries than Java because, in Java it's so complicated to accomplish anything at all that, once you've done it, you've already done it in a global, reusable fashion and you might as well share it; whereas in OpenACS, it's just soo easy to make little tweaks that solve your problem, but really hard to contribute them back. In the terms of this thread, the groove can be the enemy of the community, because it can lead to lots of customized OpenACS work not being shared or sharable.

Here's a concrete example. I want the separator character in my breadcrumb trail to be a <, not a :. Since I've made this same change five times on five different sites, I already know that I just find ad_context_bar (it's in acs-tcl). ad_context_bar takes an argument to override the default :, but since it's called from so many places, it's not practical to use that. The Right Way at this point would be to create a new parameter in acs-subsite that overrides the default, but that would involve changes to two packages and upgrade scripts and testing etc. Instead, I do it the Wrong Way: I just change a single character in one file and I've solved my problem. And forked my code. And I'll have to do again in the future, and newcomers who want to change their own sites will each have to waste an hour or two tracking down the code to ad_context_bar and figuring out how to change it.

How can we make it easier to do the Right Way?

Collapse
Posted by Vamsee Kanakala on
I totally agree with Joel. I recently spent a whole day scratching my head over how ad_context_bar really worked, and why there seems to be almost half a dozen ways to do the context bar. In the end I did resort to the 'Wrong Way' :)

-Vamsee.

Collapse
Posted by Malte Sussdorff on
I don't think there are many possibilities in general to make things the right way. The task you mentioned (create a new parameter, test it) is something which we could put up in a task tracker and maybe someone will find the time and actually implement it. But for the experienced developer it is (at this stage) easier to fork, keep a note on the fork and resolve any conflicts that might arise from a CVS update.
Collapse
Posted by Nagita Karunaratne on
Somewhere in the aD literature (not that it means that its correct) is the argument that trying to make a piece of software serve every possible situation it's likely to encounter will exponentially increase the size and in the end will not 100% solve the problem.

How this applies to a toolkit of multiple intergrated modules, I don't know ...

Collapse
Posted by Andrew Piskorski on
Joel, on your specific little ad_context_bar example, there's at least one much easier "right way" to do it:

You're saying that the default separator character is hard-coded into the proc. Ok, simply change that to instead read the default from the AOLserver config file, defaulting to the current hard-coded character if the parameter is not defined at all in the config file. Then define whatever character you want in your own config file. There, you're done, it was really easy, and your change should also be suitable for contributing back to the toolkit.

Collapse
Posted by Mark Aufflick on
actually I thought I submitted a context_bar wrapper for the family of context procs to take an option specifying the separator character which was then specified by a parameter.

I'll have to go dig that code up.

Collapse
Posted by Tilmann Singer on
I would not call hacking the tcl code of OpenACS to customize your site the 'wrong way'. After all this is a 'toolkit', not an out-of-the-box-do-everything product - in fact if every tiny little thing that one can imagine someone wants to customize would be a parameter setting it would become quite unusable.

We need a good out-of-the-box look, that's true. But for something like context bar separators I don't have a problem of telling people that they have to modify some tcl code. If the api-doc would be more often written with that customizability  in mind it would be better.

And it would be great if there was an easier way to maintain local customizations and OpenACS code than with CVS vendor branches. Maybe arch will fulfill that need.

Nagita: I think you are referring to this: http://philip.greenspun.com/panda/community.html scroll down to 'Try to solve 100% of the problem or 40%?'. I agree with the views in there and wish OpenACS would stay on the toolkit40 side instead of tending towards the toolkit90 approach.