Forum OpenACS Q&A: Thoughts on CCM

I wanted to respond to Jun's post
(https://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0005pP&topic_id=11)
listing some thoughts on redhat CCM, but thought it more appropriate
to start another thread to not clutter the nsjava discussion.

Thank you Jun - your thoughts on your experiences with CCM are
excellent. I would like to share a few of my own experiences working
with CCM over the past year. I too have built sites of various
complexities using many different flavors of ACS (1.x classic to
openACS 4.5).

Most recently, I worked with a group of 4 other developers putting
together the website for www.futuredex.com. Underneath, the site is
fairly complex in terms of data capture, administrative UI, and search
functionality. We store literally 1000's of unique pieces of data
related to our core object types and the relationships between them.
We built a pretty advanced search engine which is capable of
efficiently searching through the data using many different criteria
simultaneously.

Throughout the development process, we've constantly compared our
experiences to what we would expect had we used a product like
openACS. As Jun points out, each has its advantages. Some thoughts:

  * The user interface for search and the underlying logic to generate
the correct, efficient queries is complex. With CCM, we are able to
provide a framework for interacting with the search engine, providing
for relatively easy reuse and extension of the forms and queries we
use. This allows us to provide basic search, advanced search, browsing
along many different dimensions, customized UI, and unique attributes
for administrative interface using the exact same core code. Could we
build the the same type of functionality in openACS? Absolutely. Would
it be easier, faster to implement, or more robust? I'm not sure. CCM
and an object oriented language like Java certainly were good tools
for what we needed to do.

  * The CCM has a complex dispatching system that lets each individual
package dispatch in its own unique way. This flexibility proved to be
completely unnecessary for us, and the code itself was very hard to
debug. When I built a subsite-based online alumni database using ACS
3.x, the request processor provided an elegant solution that worked
exactly as expected and was relatively straight-forward to debug.

  * Once you understand how the CCM persistence system works, it
really has a few wonderful advantages. One example is that data model
changes are very easy to make using CCM -- changing a column name is
often as simple as updating one line of code. The other major
advantage of CCM is that it lets you override all aspects of itself -
if you don't like the way an object is retrieved from the database,
you can provide your own custom query to do it the way you like. One
disadvantage of the persistence system is a loss of some efficiency -
for most cases, you will retrieve a bit more information than you
absolutely need for your page. You can always override this, but
chances are it won't matter enough to do so.

  * Some of the greater abstraction provided by CCM can really kill
you if you're not careful. For example, it's easy to write code which
hits the database once per row for a hundred rows. In the ACS world,
when the database api was first introduced, we had a similar problem -
all of a sudden it became very easy to nest db queries. With CCM,
there are more ways to do this which are not obvious at first, but
once you know what to look at for and *how* to tune your pages, the
problem can be handled.

  * The current community surrounding CCM, simply put, sucks. Unless
I'm looking in the wrong places, I can't find any place remotely
similar to the openACS community. Redhat told me they were planning on
opening up their development process once the dust settles, but it's
been a few months and nothing is happening. I submitted a simple patch
to redhat to see how well user contributions would be accepted, and it
has been ignored as far as I can tell.

  * The learning curve on CCM is much longer compared to openACS. Part
of this is due to the greater level of abstractions, and part of it is
due to the greater number of large problems the CCM tries to address
(db abstraction thru a persistence layer, presentation through
xml/xslt, reusuable web UI thru bebop).

  * The amount of interest in Java itself is great - the JDK's from
Sun are rich in functionality as is the open-source world (open
symphony is a good example).

I've written an update that I'll try to maintain: http://people.redhat.com/richardl/ until we launch our ccm.redhat.com website.

Although its unfortunate to confirm my thoughts that pretty much the core development team of CCM is not with Red Hat anymore.

Jun, virtually the entire engineering team that designed and developed ACS 4 Tcl, ACS 4 Java, and CCM Core/ACS 5 is still working hard on improving CCM. I'm not sure who you think of when you refer to the "core development team", but I many (if not most) of the key contributors are still working in Red Hat's CCM group.

All of us at Red Hat view community as absolutely critical to our growth and ultimate success. I'm glad that OpenACS has flourished and I hope that we can find a way to work together as we go forward. There's tremendous opportunity for synergy between our groups, and I'd like to see what we can do to take advantage of the fact that we're all trying to tackle similar types of problems.

For scripting with Java, I'm not familiar with any of them, but would BeanShell be more or less usefull than Jacl or TclBlend?

As far as CCM goes, the @author tags are only a partial measure of who contributed to CCM because:

1. Typically, the @author tag is only used by the initial creator of the file, even if the file is only just a stub. Most of us don't bother to update or add to the @author tag as we add code.
2. There are a lot of steps in software development other than implementation: requirements, design, testing, and so forth. Contributors to those steps are generally not reflected in the @author tag. Virtually all of the systems have been designed by teams of people and implementation handed off to a member of the design team.

So, don't worry if you see just a few names in the @author tags and you don't recognize a lot of the current names. Rafael, for instance, conceived and designed the entire ACS 4 object and permissions systems, the CCM persistence layer, among other things. Jim is our resident XSLT guru and was a key member of the initial presentation team. Archit and I wrote the ACS 4 security subsystem, tuned the request processor with Rafael, and helped design the presentation system, login system, and other parts of CCM. I won't bore you by giving the biographies of every individual in engineering, but all of us have been working on the web development problems for several years now.

Peter and Jun bring up an interesting point about complexity. There is general agreement (and I agree) that CCM is substantially more complex than its predecessors. This complexity is an issue we struggle with every day: how to build a powerful system without presenting a lot of that complexity to the user/developer. IIRC, when we released ACS 4, with a unified object/permissions system, a lot of people had issues with the complexity of ACS 4 as well as the lack of feature breadth (where are all the ACS 3 modules was a common question). We haven't been very good about hiding complexity in the past, but we're working on trying to make things easier to use, and hopefully we will be able to work together with the community.

Regarding community, I hear you. We're going to work as hard as we can to launch ccm.redhat.com this week in beta for everyone. If it looks like we're not going to be able to launch a Beta this week, then I'll set up a mailing list.

On an unrelated note, PG hackers might be interested that we've uncovered a number of unorthodox behaviors in PG that have been fixed on the tip of the PG CVS tree (mostly related to warning messages and inconsistencies between the published API and implementation). We've also been talking to the RHDB team about CONNECT BY (we're encouraging them to implement something like DB2's recursive queries, but the end result is that we're hoping that by 7.3 PG will have a solution for tree queries).

We've also been talking to the RHDB team about CONNECT BY (we're encouraging them to implement something like DB2's recursive queries, but the end result is that we're hoping that by 7.3 PG will have a solution for tree queries)

The tree sortkey approach we've taken is a lot more flexible than Oracle's CONNECT BY and appears to be roughly comparable in performance. There is a cost in storage, but our BIT VARYING implementation is quite economical.

In fact Open Force likes this approach so much that they've begun using it rather than CONNECT BY in Oracle, using the RAW datatype to hold the hierarchical keys.

While I don't forsee us abandoning CONNECT BY queries in the near future due to the large number of queries involved and the tediousness of generating upgrade scripts, if we were to do so we could get rid of parent ids altogether and just use the tree sortkeys to generate parents (one nice thing is that you can generate the full set of parent keys rather than just the immediate parent without selecting against any db table).

If I were designing from scratch, I'm about 90% convinced that I'd use the tree sortkey approach in both PG and Oracle.

PG 7.3 will be in beta in five days so I doubt CONNECT BY will be implemented. There is a contrib extension that implements tree operators which was developed after we did our tree sortkey implementation, but frankly I don't see that it gives us anything that we don't already have in our implementation using the standard PG BIT VARYING type.

Hi Richard,

Thanks very much for the informative post.  We really need some bits and pieces of information.  I am glad CCM is still in good hands.  I am sorry if I am really pushing hard for the mailing list thanks for hearing our concerns.

Thanks for posting and participating, I very much appreciate it and I think others too.

All,

Yeap I think tree_sortkey is much much better than connect by.  It does takes some getting used to, even right now I can't still do a tree_sortkey query on the first try.  But I sure love it and use it as much as I can.

* One major problem I have which limits my productivity is the need for frequent server restarts while developing. I believe this can be alleviated by using Resin rather than Tomcat.

Yes, it can. You do this by setting the <class-update-interval> element in resin.conf.

I don't think I have a 100% work around on the no server restart issue.  I used jdk 1.4 ability to do hotswap.  Sometimes works... sometimes not.  I think it depends on how dramatically you changed the code.  If that does not work I use resin class reload.  Still not 100% perfect.

The good solution is to have a scripting languange for CCM.  Like tcl or beanshell.  Nope JSP won't do since some stuff are not really jsp besides.  Jsp is still uses java which is not good at iterative coding.

Our DBAs are hosed right now, so ccm.redhat.com won't be up this week. In the interim, I've set up a mailing list for CCM discussion. You can subscribe by going to: https://listman.redhat.com/mailman/listinfo/redhat-ccm-list or sending email to redhat-ccm-list-request@redhat.com (with "help" in the subject or body). I'm disappointed by that we won't have ccm.redhat.com up this week and we're not eating our own dogfood, but I'm optimistic that we can do this soon.

Re: Mike's comments, good points. As part of the revamping of dispatcher, we're looking at deploying each application as its own webapp, which would enable dynamic class reloading without rerunning all the initializers. We are working on a data model initializer that versions the data model and will generate the necessary upgrade scripts for your DBA to run. The enterprise.init as an XML file is a bit further out, just because changing the formats doesn't seem to be a huge win in terms of functionality (although it is the right thing to do). There's a lot of other stuff we want to clean up first.