Forum OpenACS Q&A: Reuse in the large is an unsolved problem !?

If I understand Lars article, the main problem is granularity.

[High-level organization] If you are a developer-user of the platform then granularity stops at the package level and the things you can do with the package manager and the site (map) admin interface.

[Medium-level organization] OTOH, if you are a developer-developer the "granules/constructs" are not as finely grained as they need to be. You would either have to go with the existing constructs, i.e. form, list, page, portlet or you would have to build something from scratch.

This won't cut it. What I believe is missing, is a lower-level of organization which is currently (wrongly) substituted by copy/modify/write practices of tcl procs manipulating db tables and plsql code.

What is needed, is a larger granule that can be customized without copy/pasting existing code. This is mainly subdivided in three items:

* [reusable behaviour] a class granule is a step forward since it provides added flexibility (for example, via subclassing)

* [reusable structure] move data model into a db neutral granule, for example a db class [I do NOT advocate the use of a persistence layer here, just some way of exploiting reusable behaviour for structure]

* [reusable interface] expose UI granules both at the tcl level and also at the highest level via customization services... for example, there must be a way to represent a page as a tree of UI granules/portlets that depend on a collection of data sets and their decoration and placement (layout). This IMO would enable a developer-developer to prepare a page quickly and only at the end improve the graphic design part. [This is not as simple as it sounds but it can be done]

The alternative is to use OAK http://www.weg.ee.usyd.edu.au/projects/oak/.

I asked Timo to take a look at OAK and enhance it with his current knowledge of package builder and best practises in OpenACS.

My feeling about OpenACS is that it has become relevant only to .LRN. It is becoming more and more a marginal development platform. That doesn't take away from its technical strengths. Its primary weakness is the lack of innovation. SOAP, AJAX, Web services, all are used marginally.

and

The OpenACS community is dying or dead.

and of course, Lars' conclusion:

All the problems I’d seen weren’t because of poor execution. It was a dumb idea to begin with! Reuse-in-the-large is exactly the problem that OpenACS is trying to solve, and it “remains a mostly unsolved problem, even though everyone agrees it is important and desirable”. Can it be said any clearer than that?

Problem is, that when you take away that piece, you’re left with an little known, hard-to-install, overly complex, partially debugged, aging toolkit built on a stack that nobody else uses. It started to look very unattractive. I knew I had to get away.

I find this kind of stuff distressing, but I really can't figure out what to conclude from it -- particularly since I also have had other far more optimistic discussions lately. Some random thoughts:

• What is the difference between a "mature" toolkit and an "aging" one? Now that OpenACS has finally settled a lot of the core stuff that was in rapid evolution earlier, it is now suddenly passe? Have the requirements for Internet-based database-backed applications suddenly changed and I missed some memo? How is it smarter to start over with some new first generation toolkit that has to re-solve all the old problems?

• Why is the "stack" issue such a problem for so many people? I guess that if you want to build/install systems that you walk away from and leave for maintenance to others who don't/won't know/use AOLserver/PG, then it is an issue. But if maintenance/support is part of your biz model or you're running the system/service on a box you control in some cage, why isn't the stack a black-box issue of no concern to clients? (Same point re installation.)

  However, if the fate of AOL (eg a M$ acquisition) kills off AOLserver, then this is indeed a Big Problem for the OpenACS stack.

• The complexity and bugginess of the toolkit are indeed important issues, but those are always important issues. Is there any reason to think that a ROR system that does what a typical OpenACS system does is going to be any simpler? Here's what one of my post-OpenACS correspondents admits about that:

  RoR is at the forefront of new technologies. Its real strength is Ruby, a marvelous OO scripting language. RoR's framework isn't suited so well for component reuse in the way of OpenACS's packages. Try combining a blog (Typo) w/ bug tracking (Collaboa) and you'll find that you either have to merge two code bases by hand or that you have to run separate web servers.

  Yikes; this is less complex or prone to bugs? Maybe if you're simply running a blog, but an ecommerce site or clinical trials system? Sounds almost like a joke: "How many http servers does it take to run a ROR intranet?" And note the comment (near the bottom) in this ROR wiki that ROR should derive lots of stuff from OpenACS. So once the ROR community has built its own OpenACS, will developers then flee to the next kool new toolkit because ROR is "too complex"?

• I've presumed that the relative paucity of posts from core members here (like Don and Jeff, eg) is mostly because much of the core toolkit has stabilized and there's nothing much to say -- not that these key figures have gone away or aren't involved or interested. (Of course, if they would pop up and say this is the case every now and then, they could clearly dispel any such thoughts.)

At this point, OpenACS still appears to me to be a powerful platform to build complex solutions. I'm always looking for a better starting point (isn't everyone?), but I don't think I've seen it yet for the kinds of applications I find interesting.

My take as a completely non-active developer, is many fold.

I was tempted to write a "Why I stayed with OACS and dumped my thoughts to move to ROR", I don't have time. Instead I will try to explain some of the reasons I stayed as well as some perceived problems.

1. ROR is still at pre-release, not a big deal but nonetheless, we will see how big a behmoth it becomes. I remember the old AD days when ACS was much, much smaller. I can't think of any project that doesn't bloat.

2. I don't care that TCL/AOLserver is not mainstream. I am not marketing to anyone and to be honest with you, my short look at ruby didn't impress me. Nice language, but why change, been there done that. In fact I used Perl 1.0 when it was still Practical Extraction and Reporting Languge. I used zortech C++ 1.0, MS C 3.0, Borland Prolog and Turbo Pascal, need I go on. I find the promise of yet another language that will change my life not only boring but counterproductive.

3. Contrary to the MySQL brainwashed, you can't convice me that ACID is a bad thing. http://www.loudthinking.com/arc/000516.html is a bunch of hogwash written by a good software engineer that seems to not have a grasp on writing for databases. Yes, I know that is flamebait, but that isn't how I mean it. Mainly that after 20 years of software development, I too know a thing or two about reality.

4. OACS already has many of the apps I need and is improving all the time. Are all the apps category killers... of course not. Are they good enough, certainly and I would still bet there are more ACS/OACS sites going than there are ROR.

5. AOLServer kicks ass, that is the end of the story. Screw Apache, read the history of A Patchy Server and you will gain a much better appreciation of AOLserver. Also, do a little research on security issues and you will definitly be impressed.

Now for the bad..

1. Yes it is true that the complexity of OACS is at an insane level. I can't even figure out the best practices for writing a package anymore. And for the newbie users forget it.

2. .LRN, while a great commercial vehicle, has really hijacked the project. I discussed this with CR and some others 2 years ago and while I understand the impetus for adding this complexity, putting switches in the code always struck a nerve with me.

3. The moving of logic out of the DB into TCL has left me cold. The advantages of DB enforced logic are many as CR pointed out in his reply to Lars.

Now for what I would like to see.

1. Simplify the system. I can't stand the current CSS fiasco. Why is it so hard to make a page look the way you want. Why is it so hard to figure out how to CORRECTLY use the core procs and write an app, etc, etc...

2. Make .lrn a complete layer on top of OACS, not an almost requirement to run the base system. If you have to modify the core with if blocks, something is wrong. By the same token, most likely if the .lrn app needs to modify core packages, most likely that isn't a .lrn feature, put it in the core, but make sure it is really needed.

3. There are way too many ways to do the same thing. Packages should not reinvent the wheel. One of the first things I learned when OO first came out was that if your proc was more than one screen long (and remember screens were 25 lines at the time), it was more than one proc.

4. Learn from ROR and apply it to OACS. Why is there no AJAX? I am not saying this as a critisism of OACS, I completely understand the project and the volunteer nature. Also, we have Gustaf and XOTcl, why not refactor the toolkit and use some of the excellent advantages of OO. We don't need Ruby to do OO. Again, I am fully aware of the constraints of the project, this is my ideal world.

I guess I did write a "Why I stayed with OACS and dumped my thoughts to move to ROR",

Anyway, the real reason I decided to post is that I do not agree with any statements about "reuse at large being an unsolved problem". Why? well, if it were true, we would be writing new operating systems, graphical user interface, and apps EVERY time we installed a new computer. And a new set of networking protocols everytime we put in a network. We don't. It clearly works. But, it must be done right.

I think OpenACS is busy failing to do that. And there's a particular line of thinking that seems to exemplify the problem. Caroline says "The value OpenACS delivers through reuse is larger the closer the next use is to the previous use." Well, yes, if you design all your apps to work this way, then it will work out that reuse at large doesn't work. Note that the old electronic word processors, which were designed as having a close use to the previous use, have all disappeared in favor of the far more general personal computer. On the other hand, if you keep actual reuse in mind while designing and coding your apps, it should be fairly simple to end up with a system that can be widely deployed with very little work.

I intend on outlining my ideas and suggestions for OpenACS improvement and growth in a series of (bi)weekly articles on the above listed blog over the next month or so.

** Martin

I don't speak for everyone, obviously, but from my perspective what's wrong with Tcl is that when you search for it on monster.com there are very few hits. I've lost track of the number of potential clients who have told me that. They think they are going to end up with an unmaintainable site because "no-one out there knows Tcl". Combine that with the me-centric perspective of their existing developers, who tell them that Tcl sucks because they don't want to be bothered with learning a language that won't make their resume look better, and you have a pretty serious sales hurdle to overcome.

Adding that to the other hurdles of AOLserver and OpenACS being unfamiliar and things can get quite challenging. You can talk all you want about how client should not care about the underlying architecture and you'd be right, but the fact remains that when the IT dept is doing the evaluating, as it often is in the larger companies we deal with, they care about these things.

I personally have no complaints about working in Tcl but in the long view it's not about me or my preferences, at least not very much. It's about what our clients can feel good about putting their reputations on the line for. And unfortunately, too often the OpenACS stack ends up looking radioactive from their perspective.

I suspect an unstated problem here may be just that smart hackers get tired of working on the same stuff month after month. You burn out, and want to try something differnt.

Now, of Lars' bullet points (and some of the implicit ideas from his text), IMNSHO these are the only contentions that are actually important:

1. OpenACS community not growing enough.
2. OpenACS code quality uneven.
3. Ruby on Rails is a better toolkit than OpenACS.

I'm also pretty disapointed with what seems to be the, 'Ruby on Rails rocks, but I'm not going to bother to tell you exactly how or why, it's just better.' subtext of Lars' article. I'm not familiar with Rails, so I'd definitely have liked to learn something about it vis-a-vis OpenACS besides that Lars was "blown away", and that "all the problems that OpenACS had were solved."

Also, like Simon, I think Lars is completely off base with his hand-wavy "reuse in the large is an unsolved problem" argument anyway. What's critical for reuse? Clean, well factored, well maintained code. OpenACS is large, has grown organically over a long time, and has not had nearly as much dedicated house-cleaning and re-factoring as it could benefit from.

Think about it: Don has periodically complained about the ugly beast that ad_form has become (thanks in part to Lars' hacking on it...), and there are probably plenty of other similar (if less obvious) spots in the codebase. (Hell, I volunteered to refactor the acs-tcl code into Tcl packages suitable for use either within or without OpenACS, "sometime", but I've still never got around to even starting on that.)

When it comes to adapting code to new, unanticipated circumstances, simplicity and cleanliness of the codebase are key. IMNSHO OpenACS isn't all that bad in this regard, but there's definitely lots of room for improvement. For OpenACS, a useful yardstick is AOLserver: We should aspire to have OpenACS code at least as clean as that in AOLserver.

Btw, my uses for and interest in both OpenACS and AOLserver are similar to (if a bit less broad than) Simon's. They are both toolkits, and rather full-featured, ready to go ones. Kind of like an industrial sized Dremel tool that comes with 67 bits - but keeping the bits organized and sharpened and replacing the ones lost or broken over time takes real work.

And yes I know, the sticking factor is always time, especially time of the most expert developers. I don't have any answers there, but certainly welcome suggestions from others...

one of the problems I see within the OpenACS community is a tendency for free riding among some of the members. This is in itself okay, though with the advent of 5.3 we at least hope to figure out who is using the system.

On the other hand we have people who write wonderful applications on top of OpenACS and talk about them in detail, but sadly neither to commit their changes (like: "Now Form Builder allows us to automatically integrate metadata changes in our edit screens.") or give a pointer on where to download the sourcecode.

There are usually a ton of reasons why the latter do not release the code or make it easily available, but one of the ideas of an OpenSource Project is to write code and give it back. And if the steering comittee (in our case the OCT) is not happy with the changes, they can prevent them from happening or suggest improvals.

So I'd highly encourage any member of the community to give back code that they have written so others can take a look at it, reuse it in their solutions and generally work with it. It is in your own interest, as a fork generally makes it harder to maintain your code and you will not be able to profit from the experience of the OpenACS community as a whole. Well, that is unless your ego is high enough to believe that the experience of the community as a whole is lower than your own 😊.

Malte wrote:

Out of curiosity, is there a policy on
having a standard footer in the forum postings?

The footer URL doesn't seem to work... 😊 I just add the link for the people reading this thread. The OpenACS.org pages aren't really indexed by Google. I even don't get the references when I do "ego surfing"... But you can try it in other forums.

However, I very much wonder about the purpose and intent of your previous comment. I really don't think that the "free riding" ethical misbehaviour applies in our case. I know that you have very detailed knowledge about our GPLed modules and our announcements of their availability. It is a different question why the community hasn't included them in OpenACS. You have had a detailed look yourself at the "Flexbase/DynField" SQL Metatadata system. It was you who decided in our meeting in Hamburg not to include it in OpenACS because of incompatibilities with AMS. That is perfectly reasonable and OK, but please don't blame it on us.

I also know that Quest Computing is sharing the experience of being "ignored" by the community in terms of code being incorporated in the platform. I asume that this is due to the different focus ("ERP" like applications vs. Online Community applications).

Here is the list of our GPLed modules and how to get them (please use: cvs -d :pserver:mailto:anonymous@berlin.dnsalias.com:/home/cvsroot checkout module-name for CVS access):

- OpenACS Windows Installer (GPL, source code included in the installer itself. There is a posting on OpenACS about how to compile the installer)
- Flexbase/DynField SQL Metadata system (GPL, module: intranet-dynfield. DynField is a modification of Flexbase, that has been developed at Quest Computing. Also, you can ask Ciaran De Buitlear from Quest for the original code)
- Automatic Software Updates CVS Frontend (code says proprietary, but we can GPL it if you want, intranet-update-client & intranet-update-server)
- PostgreSQL Efficient Intranet search with permissions (GPL, there is a posting on OpenACS explaining the underlying thoughts and architecture.)
- Hierarchical Menu System (GPL, part of the intranet-core module)
- "Plugin Components" GUI System (GPL, part of the intranet-core module)
- Category System with multiple inheritance (GPL, part of intranet-core)
- We were the first to organize a Spanish ACS bootcamp

Also, there are many new maintenance screens (permissions, parameters, menus, ...) as part of the GPLed intranet-core module that you could include.

On the other hand I admit that we have developed several innovative modules for OpenACS that we haven't GPLed for several reasons. These modules include our financial system, a reporting system and the sector specific code for translation and consulting companies. However, I have announced and explained this decision before we actually started developing the modules and I have got positive feedback from the community in general. Maybe this perception is changing?

However, I asume this thread is about why OpenACS is loosing important developers and I am not sure whether the "free riding" problem is at the core of this issue.

Where I do see important need for action to "save" the OpenACS community is the Windows installer and the right branding of OpenACS.

Concerning the installer, we have been contacted by Nima and Rocael. So I asume that this area is progressing. However, it seems as if the community is not very keen to collaborate with us. Instead, e-Lane seems to try to build a separate/ branched installer. That is OK for us, but a bit surprising. I think I have stressed several times the technical and organizational complexity of creating and maintaining a Windows installer, but maybe I haven't expressed myself sufficiently dramatically about the hassle and the suffering that the Windows environment can inflicts on you. The problem is that the job is extremely unpleaseant and that any (sane) developer will run away from it if he or she can. For this reason the Windows installer development is very difficult to manage and only viable with a fixed employee whom you can "force" to work on it...

Concerning branding: We have made experiences with PR and have had some success with it, particularly in our core sector of translation/ localization project management systems. We are perfectly willing to share this experience with the community.

Let's get out of the "niche" existence!
Frank

mailto:frank.bergmann@project-open.com
http://www.project-open.org/

Simon Write:

The problem OpenACS is having is that Forums, Chat, File -storage and so on are simply not very challenging. They've been done to death, and largely for a customer its all about tweaks and cosmetics. OpenACS is a sophisticated, re-usable architecture. Unfortunately website are not challenging enough for it. Rather then trying to be an also ran imitation of the PHP applications which are infact imitations of us (ACS being one of the early movers in the forums, file storage community sites) we should be looking for the things that make us different and highlight them.

The conventional wisdom is that open source communities are communities of individual hackers and you have to be very careful if you are an organization approaching them to use their code to make money.

http://producingoss.com/producingoss.html (Read the Money chapter, note there area actually a lot of very good ideas in this book.)

I am putting out for thought and discussion the concept that we are a community of organizations and that the new users we wanted to attract are more organizations. If we felt that way we could go even further then a footer and support automatically putting a linked logo on each post similar to the way other forums put in portrait images. Possible Advantages:

• Look organization friendly. I think we already are an organization friendly open source community. Its an advantage. Lets flaunt it.
• Help readers understand the context of the post. I often find myself wondering when I read a post what they are doing with OpenACS..is that an e-lane person? Is that the company in Italy doing the ERP work? Although we need to explicitly address realy making it easier for people to understand how OpenACS is used, this would provide a way for a motivated person to follow the links and begin to learn what OpenACS is actually being used for.
• Provide a realistic context for new people. New people sometimes wonder why they don’t get more free support on the boards. Highlighting that it is a community of paid developers might help them understand.
• Make it easier for developers in organization to justify posting helpful comments and writing up their work for the boards. The vast majority of people here use OpenACS as their day job and may well have to justify to a manager why they are spending time writing answers to other people’s questions. Giving back some good exposure to their organization might help us promote good citizenship.
• Look Different. We need to stand out for something other then tcl and AOLServer. We should be looking for ways we really are different in a way that is positive for our target new users and do PR around it. If we consciously shifted our orientation from a community of individual developers to a community of organizations developing in OpenACS we might attract attention from the anthropologists/open source scholars. Simply put, it might be an interesting publicity stunt.
• Show that OpenACS can do forums with little pictures. Forums with little pictures are in right now, and some people assume that since they don’t see them we don’t do it.

I hear several other probably important points in this recent discussion.

1. We have companies who want to give back but are not doing so.
2. Google isn’t indexing our forums well.

The conventional wisdom is that open source communities are communities of individual hackers and you have to be very careful if you are an organization approaching them to use their code to make money.

I've always assumed that the OpenACS community wasn't resistant to commercial activity. After all it started as a commercial, single-company enterprise. I hope thats still true?

What might be an idea is some discussion on how we can make the toolkit more 'commerce-friendly'. I was tempted to raise the 'Is the GPL the right license issue' but perhaps thats too emotive. :) There are other things though that include marketing, organisation, presence and so forth. One criticism of OpenACS is that its a high barrier to entry for new developers/volunteers. Is this such a bad thing? No enterprise I work with has that high issue on its list. Thats what they employ us for! What they want to know is

I'm more exicted about OpenACS than I have ever been.... amazing things and project cropping out all over the place (some years old, some new). The momentum is building.

The side bar would contain information about the poster, such as a portrait, "member since...", name (link to pvt/home), company/institution (link to homepage). Most of this information can be taken from pvt/home except for the company/institution information.

Out of curiosity, is there a policy on
having a standard footer in the forum postings?
>
>Fank B Wrote
>for the people reading this thread. The OpenACS.org pages
>aren't really indexed by Google. I even don't get the
>references when I do "ego surfing"... But you can try it in
>other forums.

For the record, Google has almost 1.8 million openacs.org pages in the index:
http://www.google.com/search?num=100&hl=en&lr=&q=site%3Aopenacs.org+-asdf&btnG=Search

That's a LOT of documentation 😊

If you want ego surfing, 466 of those pages have Frank Bergmann on them:
http://www.google.com/search?num=100&hl=en&lr=&q=site%3Aopenacs.org+-asdf+Frank+Bergmann&btnG=Search

Only 15 or so have the url project-open.org -- not bad.
http://www.google.com/search?num=100&hl=en&lr=&q=site%3Aopenacs.org+-asdf+http%3A%2F%2Fwww.project-open.org%2F&btnG=Search

Does it help your google score? It can't hurt. Does it help your site appear higher in search engine result pages? For certain keyphrases, sometimes, yes. Does it hurt openacs.org? Probably not. Does it bug me? Not at all, I like the option of seeing other openacs people's sites.

As for Lars and the reuse in the large apology: Maybe rewriting applications works for him and the other great programmers here, I do see the point, the added complexity of openacs is often unnecessary, but plebeian programmers like me who rely on the shoulders of giants unfortunately don't have the option of writing whichever applications we dream up, as needed, as if we were Philip Greenspun on a mission. We, thankfully, like the "community in a box" idea and properly discourage clients from their (often un-needed) customizing. 😊

Rails is a bunch of procs. Maybe one day in the future someone will aspire to use Rail to to make a united multi-user multi-groups "community in a box" application suite that approaches what openacs is today -- but that day is not today and it might never happen.

And at some level, doesn't reuse in the large have to occur? I mean, even Rails is on linux or some database that IS reused programming?

Which timezone are you in? At about 12am UTC most days there are about 3-4 OCT members on #openacs.

We are well aware of the documentation issue. It is something that has been brought up a numerous times. It is an issue that is difficult to resolve. Currently there are two mindsets, one is to persist with docbook, the other is to start using the wiki. Until the community decides which approach to take, we won't be making much progress with improving documentation for the time being.

1) Our company does lots of non OpenACS based work and so working from the OpenACS CVS wouldn't fit with our company's quality system.
2) Only a sub group of our company use OpenACS so the open source mindset is not as prevalent here as in other places. This means it can be difficult to explain time spent on this kind of thing (but I do my best  ).
3) We started out by taking the OpenACS code and changing packages in a fairly business area / client specific manner. This happened in a quite “organic” or chaotic manner initially. These changes would be more of a hindrance than a help to others who are not in the business area we occupy.
4) We provide a business area packaging of the OpenACS (in a similar way to .Lrn) which centers around the Petri net workflow and ticket tracker and as the Petri net workflow seemed dead in the water it seemed that our stuff would be of no use.
5) In competitive environment getting changes specified, cleared with our clients, coded, tested and released is pretty much all we have time for. I’m not sure how we could fit in a consultation process with the community. Part of this problem of course could lie with me as I’ve always found it hard to converse usefully via a bulletin board.
6) We also have some packages that we would prefer not to GPL for similar reasons to Frank’s.
7) We operate in the same type of space as Frank and have the same issues that things we’re doing are not interesting to the community (or so I’ve found).

Ok so on what we have done and would like to do:
1) We have contributed two packages to OpenACS. Templated letters https://openacs.org/forums/message-view?message_id=71372 and
2) ePDQ credit card payment integration https://openacs.org/forums/message-view?message_id=89032
3) We have hired contractors from the community.
4) I encourage our developers to contribute to the forums as much as possible.
5) I believe we have features in our code base that would be useful to the OpenACS and others that would not but would like to figure out which ones would be of benefit to the community so that we release them. Here’s my post about this: https://openacs.org/forums/message-view?message%5fid=329333#331678 . I’ve asked Christian Knopfel Eva to take this on and he should be in the office this week so expect to see more.

If The Goal is to create the best possible documentation for the all toolkit's users; end users, administrators, graphic designers and developers should be facilitated to contribute to that joint effort.

I haven't got a clue about Docbook, but when I googled for a conversion tool it came up with http://deplate.sourceforge.net/index.php Maybe that way we can create a platform that keeps all hapy, and saves many moons of discussion.

Just a thought.

"I think some day it would be nice to have colleges have programs, very strong programs, in this sort of user interface but with a very humanist attitude point of view, and start playing around with new approaches. Some year, some decade, it might just come into the popular conscious to start making things more beautiful for people."

Maybe one of the first things we have to to in our effort to revamp oacs.org design & presentation, is close the existing bugs. Many still refer to old releases, or relate to problems on individual installations.

Chances are users have found work arounds, moved on and left OACS for what it is, or have upgraded to the latest release where the reported bugs have been closed.

Maybe an idea to appoint a human editor, I mean, someone that keeps track of reported problems, and coordinates if issues are solved or not. I have noted sometimes newbies report problems on the bboard as well that remain unanswered. I feel we need to coordinate these kind of issues better.

Just a thought.

Ben