I have been watching the effort to release 5.2 from afar, and I cannot help but feel the frustrations that must exist. OpenACS is a very complex system despite the seeming simplicity of its components, codebase and otherwise. I was considering a TIP, but Dave's invite to discuss here makes it as good a spot as any to present my idea to ease these things.
The current system of piling all things into /packages is no longer appropriate for a system of this complexity, and greatly inhibits integration, release, and regression testing. As Dave noted, 5.2 took a long time, and a smaller set of features might be a better next bite. As well, as becomes clear when searching through past forum entries, the needs and desires for some applications have conflicted with the need and desire for a stable, releasable, progressive codebase. Further, duplication of function in one package to add just a few features has led to abandoned hulks cluttering up the landscape. Just like any good piece of software, eh?
I don't know the size of the original ArsDigita team, though I imagine it was small. The organization of the ACS code certainly looks like it was managed by a tight knit team with high-bandwidth internal communications. Today, however, the team appears to be diverse; in geography, need, and coding style. The current codebase does not serve the product well in that environment.
The problem lies primarily in regression issues. This is an age old problem. Every successful large software project is riven by the need to produce quality code and the need to add features. Stable code is mostly mature code. New features are necessarily less mature than old code. And so it goes.
Fortunately, the solutions are nearly as old and are quite proven: separate compilation and functional segregation. Though a scripted system cannot benefit directly from separate compilation (notwithstanding the inestimable shoulders of the giants of the aolserver, oracle, and postgresql), it can benefit, and greatly, I submit, from functional separation. To that end, I propose a separation of the packages into core, base, system, contrib, test, and site. The migration of packages/* would look like this:
packages/acs* -> packages/core/*
packages/calendar -> packages/base/calendar e.g.
packages/dotlrn-* -> packages/system/dotlrn/*
Putting the packages currently called packages/acs-* into packages/core means that they can be tested as a group and released as a group. If features are necessary for the market acceptance of the system, then features can be migrated to core as they mature. New packages, as they mature and gain acceptance, can be migrated to the feature area.
Now, I'm expressly not suggesting a wholesale change in internal namespaces. While there may be some advantages to that, namespace changes are often hijacked to become architechtural overhauls, and I don't see any advantage worth the risk of that. What I am suggesting is modification to the package manager, so that it looks for packages in an ordered set of subdirectories, perhaps with a control list that an administrator can use to control the order.
The benefit from this will flow from a subsequent decision o update the core and system directories separately from various features, perhaps even from each other. Thus, the ore can be refined and released on a regular basis. Regression testing of feature packages won't hold up the ore. Fixes that address critical issues will not have to be tested against every possible combination of packages, only against those in the system collection. Over time, this will allow the system-core interface to stabilize such that poblems can be readily localized. The result will be that the fix to a problem with a feature will lie within the feature. These benefits, of course, are what are made of changes as I propose. Merely re-arranging the files will do nothing.
