Forum OpenACS Development: Proposal for hooking forms to metadata

I've put together a quick prototype for interfacing the form templating package and the acs metadata. It also includes code for making most of the metadata acs objects. For a first look see

Demo and Docs

Posted by Don Baccus on
Have you looked at add_form_elements in acs-subsite/tcl/attribute-procs.tcl?

There's a bunch of other attribute metadata Tcl wrappers in there, too.  Not all of it works.

Anyway ... as I mentioned earlier there's duplicated (but more sophisticated) code in the content repository that does similar stuff.

What we really need to do is to plan out our strategy to do the automagical form generation, data insertion, data editing stuff so that we can look forward to the day when we have one API to do so that's comprehensive and works.

Then we'll be able to lash together packages with a lot less effort and if they can be done through a decent API of this sort, can have them support both RDBMSs out of the box, to a large extent.

Your demo's slick but it represents a third code branch which works towards using metadata in this way.

We may be stuck with two due to there being so much redundancy in the CR of features found in the object model and these features in somecases are grafted onto rather than organically part of the rest of the toolkit.  Merging is probably more a medium-term than short-term proposition.

But ... let's work towards having one fully-functional API for standard objects rather than work in parallel!

There are several of us currently interested in, and to a varying extent looking into, solutions.  You, Lars Pind, me, Jeff Davis, Dan Wickstrom at the moment (there are others, I'm sure)

How can we best coordinate our efforts so we don't double up?

Posted by Jeff Davis on
Tom Jackson's query writer also covers some of the same ground. It's worth having a look at it as well.

The problem with the stuff driven from the acs_attributes alone is that the attributes don't really tell you enough about what widget should be used or populate select widgets etc. cms has that additional data for driving widgets (cm_attribute_widgets, cm_attribute_widget_params are the two tables, content::new_item_form is the form creation code) and it lets you do quite fancy things but the datamodel presumes that there will only ever be one form for editing a given content item and also the code forces you to edit one object at a time. For real world problems you end up with several interrelated objects you want to be able to edit simlulaneously (for example, an article you might want to be able to edit the list of associated photos, the author data, title, blurb etc. and it's probably the case that that is at least 3 content_items (probably even more than that).

Anyway, I think driving the UI from metadata is a great idea and am a big fan of doing it that way but I also think we need a better datamodel and better supporting code so that our saving programmer time won't come at the expense of creating horrifyingly bad UI experiences.

I think one place where we fall down now is in not having nicer higher order input widgets (like a standard address widget or phone entry widgets) and doing multifield validation (oh and not signing enough variables but thats not a UI thing).

Posted by Jon Griffin on
Add my MDev as another, although not public yet fork. (Actually a few people are using it, but I don't really envision it going as far as this discussion)

Also, I am almost done with Contacts, several people have reviewed it and when that is done, it shouldn't be hard to add a standard phone or address widget. All this address/phone/contact stuff follows the HR-XML standard and is fully international.

Changes to person are required for this to work but the PG upgrade script is written and most of the Oracle upgrade script is also.

Posted by Don Baccus on
There's no way that automagical generation of forms etc will solve all problems, that's for sure, Jeff's right about the potential for UI hell.

This is one reason I'm still interested in extending ad_form to be able to generate blocks of elements from content (or object) attributes - this would give one a way to blend custom form needs with automatically generate elements from attributes.

I think there's room for both approaches - simplifying the coding of more complex cases while providing essentially automated handling of simpler cases.

I think Tom's work also includes code to generate package/proc stuff (new?) and that's also duplication of effort of code that already exists in acs-subsite.

Maybe we should start up a forum for discussion of how to best coordinate efforts and reduce the redundancy of effort we seem to be seeing today?

I really hate to see so many people working in parallel on the same essential problem!

Posted by Tom Jackson on
I think Tom's work also includes code to generate package/proc stuff (new?) and that's also duplication of effort of code that already exists in acs-subsite.

Don, please. This writing of the 'new' function for a package is a tiny tiny part of what query-writer does. This code isn't 'duplication', because the functions are generated from data that is necessary to write the queries for insert/update/delete. Query-writer already can write regular dml for these operations, if you don't supply functions (in case it isn't an acs object).

If you use the url based api, you can easily put any combination of attributes on a form for insert/update/delete and have all submitted attributes checked for whatever filters you specify. In addition to the regular filters available, you get automatic checks that the length of each attribute will fit (for varchar). Permissions are checked for any number of attributes on an insert/update (for delete there is usually only one).

For instance you could have an invoice which is generated from data from a number of tables, say invoices and invoice_details. In one form you could update attributes from the both tables, including all the rows of the details. You could delete rows, or add completely new rows all with one click. None of this requires anything more than writing the form and directing the form to the qw processing page.

However, since it is a url based api, you don't even need a form. A program could issue a GET or POST of the proper form to do the insert/update/delete.

Although the examples available in acs-subsite, group types and cr/cms do a few things that resemble this, these efforts are narrowly focused and incomplete.

Also, my code has actually been used to create a useable package of many many pages.

I would not recommend using my package for acs-core, or for specialized cr/cms applications. It wasn't designed for that. I wound not recommend using it for a one or two table datamodel either. I would recommend using it if you are writing an application with a lot of object tables.

This is just what the current query-writer does. The new and improved version will be easier to use and more general in application. As it stands the current query-writer is annoying to setup.

Anyway as far as I can tell most apps work by sticking their data into tables (or updating and deleting it), data gleened from forms submitted by users. This is what query-writer handles. It doesn't write forms, it doesn't select data from the database. It also controls access to attribute values, not just attributes. And it caches all the info needed to work, without hitting the database.

Writing the pl code is really a side effect. Someone or some thing has to do it. My tack was to generate a first stab at some pl and allow the developer to copy/paste it to the sql directory and edit, if necessary. The new version will be a little more developer friendly in this respect. So neither version is dynamic like what you find in other code examples. On the other hand, query-writer can be more general and handles overloaded functions. None of this could have been achieved by starting from the other code.

I believe that your intent, which I agree with, is to get everyone to join forces and come up with a general solution that would satisfy everyone. I would of course not mind doing that, but query-writer is necessary for my own development work. I doubt that anyone else is even using it.

Posted by Don Baccus on
Yes, this is exactly my intent, except I doubt we'll end up with anything that makes *everyone* happy.  Also, I don't mean to imply that folks shouldn't experiment with different approaches.

What I'm asking for, I guess, is ongoing discussion and some effort at communication.  As this thread makes obvious, I (and others) can't necessarily even judge what is and what is not duplicate effort!

And I know all of us are working hard on projects with deadlines, and that servicing our clients comes before coordinating with the community due to the need to put food on the table.

The CR/CMS API has been used to build one complex package, the CMS itself.  From a functionality POV it's very usable, it's only from a UI POV that it's not.  And Jeff's using it (and improving the API) to write another.

Your query writing package sounds interesting, especially in its new form (the first version seemed really clunky, which I don't mind saying because you're saying the same thing and rewriting it in part because of this.)  If it turns out to be really good, Jeff may decide to put less effort into improving the CR/CMS API, for instance.  Maybe we could stick him with rewriting the admin UI that everyone hates so much instead!

Posted by Jeff Davis on
Maybe we could stick him with rewriting the admin UI that everyone hates so much instead!

Oh no, that's definitely Lars...

Posted by Barry Books on
I've been out of town but here are my comments on the comments.
  • I put Jaguar on my Mac and I've been messing with Objective-C so I've got objects are objects on the brain.
  • I had seen the acs-subsite code (odd place for it). The code I wrote serves two purposes. Make object_types etc objects and generate forms from object_types. I was insprired in part by add_form_elements.
  • I'm going to mess with query-writer this week. I noticed it only runs on Postgres and I run Oracle.
  • I really like the form templating stuff because it will do almost all the work for free but you can still customize form layout if you like. In fact you can template the same code differently by instance, something that's difficult with the rest of the templating system.

I've spent that last month looking at the java/tomcat/struts/turbine. Some parts are interesting but the thing that struck me was how much code you have to write to do anything. My wife ( a graphic designer ) can write tcl pages to process forms, send email and insert into a database and she does not think it's hard. I think java etc is hard, too much complexity to solve the 'scalablity problem'. It solves it by ensuring you'll never have one.

That said I think there is much room for improvement in openacs. The metadata model is a good start but not complete. This causes poeple to not use it or extend it in package specific ways. There are two ways to implement form validation. ad_page_contract and the forms package. This causes a UI problem. There are things I like about both and perhaps ad_page_contract could be extented to work better with the forms package.

One thing I did like about struts is the MVC model. It seems like this could be implemented in the request proceesor but that would be a whole different discusstion.

Where to go from here? I think the first steps would be to make the object_types etc objects and extend the attributes to support form templates. I think this would help the permission system and make the form system more usable.

By making object types objects you can give groups of people access to groups of objects. For example you could make an abstract type called developer_forum have supertype of forum, give the developer group access to developer_forum and write queries like

select object_id 
from acs_objects 
where object_type = 'developer_forum'
and get a list of forum objects the developer group can access. You can also control what attributes users have access to as well as what enums they could choose. This could be done by granting permissions instead of writting code.

Added data to the attributes so forms can be generated from the meta data would allow packages to be more dynamic and configurable. It would also allow the development of a RAD package. The are pieces of that now but I would be nice to add to the core because even if a package was not completed by RAD it could be started that way. That would help insure that sql runs on all supported databases and the sql object interfaces and metadata are created in a consistant manor.

I think all of this can be done with a second set of tables so there will no impact on existing apps. Should we start a discussion about what should be in those tables?

Posted by Dave Bauer on

You plan sounds good, I will leave it to Jeff and Don, to comment on the plan.

I am curous about your statement that you can't use more than one template with the same code. You can use a tcl datasource script with more than one template. The default is to use the template with the same name as the tcl datasource script, but you can specify any template name with the ad_return_template command.

Is this what you were referring to? Or am I misunderstanding?

Posted by Barry Books on
By default the templating system pairs by name so index.tcl is paired with index.adp. You can override this but I've never had much luck with that. You can place files in the filesystem with paths like that those created with site map and override index.tcl and index.adp, but it's both or none. It would be nice if the request processor would allow you to override just index.adp so you could change the look without changing the code.

The form package is slightly better because you can specify a form template in the form tag. You still have to write code but it just seams cleaner to me.

Now that I say this I guess fixing the first would also allow forms to have different templates since the form template is in the adp file.

Posted by Dave Bauer on

Correct. If you do not specify the ADP template when you call ad_return_template, it looks for an adp with the saem filename as the tcl file.

You can specify an alternate template, or decide in the tcl code which template to call.

But it sounds like you are talking about replacing the built-in templates with locally customized versions. Is this what you are referring to? There isn't currently any built in way to tell the request processor to use a local version of a file instead of the default one supplied with a package. I have a couple of different ideas on how to fix this to make it easier to make local changes to packages.

Posted by Don Baccus on
We implemented customized .adp templating for scripts for Greenpeace, but not in a general way.  The only barrier to doing something general is deciding on just how we want it to work.  For GP we searched first in a custom location then used the existing paired .adp file (index.adp for index.tcl for instance) as a fallback, that way subsites only need to customize those templates they actually want to change.

But in the general case I'm not sure how it should work.

Regarding object types as objects ... Barry, could you e-mail me a tarball of your changes and (hopefully) update scripts to take a look at?  As I said above I do agree that this would be useful, and yes, permissions are why.  We were looking at this in the design of a "general ratings" package (that hasn't been implemented as the volunteer got a real job, a never-ending one because the company's using Java.)  The idea was that rating dimensions were to be ACS object types, and if these types were objects you could control who could rate an object using a particular dimension (rather than just "user can rate all aspects of a particular object")  The volunteer had a lot of experiences with knowledge management systems and had found that level of granularity important in the past.

Anyway ... I'd love to look at your object-types-as-objects changes.  4.7 is the perfect place to slide in fundamental changes of this sort.

form templating vs. ad_page_contract.  When I wrote ad_form (have you played with it?) I considered trying to do a tight integration with ad_page_contract but decided against it for the time being.  There are several reasons for that.  Among other things pages handling multiple forms aren't uncommon, so you want more than one ad_form call for the separate forms. And individual forms frequently need to be built piecemeal, this the "ad_form -extend" construct to add to an existing form.  On the other hand ... ad_page_contract sets a contract for the page and should be a one-call device.  Also errors are returned in an entirely different way by the form builder (ad_form) and ad_page_contract.  And ad_form handles the flow of processing self-submit forms for you in a way that's straightforward but doesn't blend in well with the ad_page_contract paradigm (IMO)

These aren't simple design issues to solve, so I decided to keep ad_form and ad_page_contract separate.  At the moment I think it would be a lot easier to document how the two interact.

Posted by Alfred Werner on
For history's sake, I wanted to link this thread to this announcement for AMS - the Attribute Management System .