Forum OpenACS Q&A: Request for Comments and Discussion: Building a Leaner, Meaner OpenACS with MIST
Many others have noted this, too. I've spent a few weeks collecting ideas and putting together a proposal for how we might fix this with a bit of clever technology. I'm calling it OpenACS MIST (Magical Installation and Setup Tools), and I'm anxious to see what people think.
This post is meant to foster discussions: nothing on this project has been done yet except for putting together the proposal.
Check it out at http://openforce.biz/openacs/mist and please post your feedback!
And that folks have already agree that we should strive to do this for OpenACS 4.7?
Comments like this:
Aren't really based on fact. It seems to me that you're just trying to rehash the "where should new-portals live?" argument.However, we are currently pursuing a development route very much in conflict with this goal. We are adding more packages to the core, adding more dependencies to the release of a monolithic distribution, and generally slowing down development.
What we've faced with is a resource limitation. I ended up being stuck with the task of packaging OpenACS 4.5 all by myself, and I made it clear that this meant that the release would be in the form of a monolithic tarball.
For resource reasons only. I have made this clear more than once.
And do keep in mind that our current 3.x-based openacs.org website doesn't really give any support for a modularilized distribution. This, too, is a resource limitation.
I'm sick and tired of explicit comments and implicit hints that I favor a monolothic distribution and synchronized release schedules as our preferred distribution. I've been following this paradigm out of necessity, not preference.
So, my first reaction? I wish I could quit the project entirely and walk away. Unfortunately I've made commitments that don't make this possible. Those commitments are the only thing keeping me here.
Since I can't follow through on my first reaction, I'll read through your note when I have time - perhaps this weekend - and comment.
But let's set the record straight. Solving distribution issues has been an action item for a year now, with the original thinking that OpenACS 4.5 would be the one and only release made in which one was forced to download a monolithic tarball. Unfortunately OpenACS 4.6 will be distributed in the same way (the fact that we're still running our 3.x openacs.org website being one factor in that decision on my part).
Now ... one thing I will say. There will always be many people who want to download a single tarball of all packages officially supported by the OpenACS project. Huge numbers of people download full ISO images of every RedHat, Mandrake and other releases (including Debian), typically well over a gigabyte download as compared to our puny < 10 megabyte download. Many others buy official CDs of Linux releases.
So I *will* personally oppose any suggestion that we not provide a single tarball for those who want it.
as a new developer and end user. One of the "problems" i see with
it is that these out of the box "distributions" i.e. dotlrn, etc.
are not very hard to set up - sure CVS isn't that fun, but once
you have APM's of dotlrn, etc. its pretty easy to do - thanks in large part to the package manager and *.apm files.
Given that currently people need to know how to compile
postgresql (or worse install oracle), aolserver, make them
automatically start/stop and respawn themselves, etc. change user
profile settings (i.e. bash_profile, etc) before being able to
install openacs, I don't think the ease of install MIST would
provide would be necessary given that user base already needs to
be reasonably computer savy to get to the point where they get
the core working and are able to use MIST... If all one needed to
do to get the core running was to apt-get openacs, or rpm -ivh
openacs, and if that took care of postgres, oracle, aolserver,
etc, setup, as well as the service monitor, (possibly even Image
Magik) and all of the that, then openacs would be catering to a
different group of people that might find the install scripts
helpful, but I don't think the current user base would find it to be very important. And i'm not sure if openacs should even cater to the user base that needs rpms and deb files... websites inherently need to be customized, and you need to be reasonably savy to do that.
I do like the idea of clearly defining a core and separating
packages from that core as much as possible, but the automatic
install aspect of MIST doesn't seem to important to me (as a new
developer/end user). It would be nice to have
packages.openacs.org, as a shared ftp type server (though most
likely with file-storage on top of openacs) so anybody can
upload a module they create with a description of its application - but it sounds like this is already in the works.
I agree with Don that getting the whole tarball is a nice thing. This is what i do with Linux Distro's just like he says... hard drive space is cheap (especically when were are talking about less than 100MB no matter what), and it is nice to get everything... that doens't mean it all needs to be activated, but there is no harm in having it sit and take up space on your server... it isn't that much space after all. Same things goes for linux distros - i always install everything from the beginning, and just deactivate it until it is needed... much like openacs distros.
This is a total misrepresentation of current thinking.We know the answers because we just changed our name to Open Architecture Community System. OpenACS is an architecture for online communities. Thus, as an architecture, it defines how various pieces of code are going to interact with one another. It does not define all the features that will ever be part of an OpenACS installation. It defines the object system, the permission system, the package system, the request processor. It does not define the UI or specific end-user (forum, calendar) functionality. "Being part of OpenACS" should simply mean that you: 1. adhere to the OpenACS base APIs, 2. build a compliant, good-citizen package, and 3. generally follow the OpenACS development style guidelines. But does your package have to be distributed with the OpenACS distribution to be "part of OpenACS?" No. In fact, going down this road is dangerous: it creates bloat in the core distribution of OpenACS, complicates the learning curve, and generally makes OpenACS far too monolithic. Our current thinking is leading OpenACS 4.x down the monolithic, un-modular 3.x path.
So perhaps there has been some miscommunication there, so let's look at this objectively.
Regarding the especific issue of new-portals being merged into the OpenACS core and used throughout administration (instead of just being "distributed" in the OpenACS tarball), I don't know all of the issues there, and I think it is the matter for a different thread.
MIST however adds something really interesting to OpenACS, which I had toyed with in my mind while trying to figure out how to go about creating Debian packages for OpenACS and the different vertical applications based on it.
With MIST, my Debian (or other types of binary packaging) would just include automated MIST scripts to install everything the user wants.
I am merely trying to contribute to the pot. There is nothing revolutionary about what I'm saying, only a combination of existing ideas that others (including yourself) have brought up.
I am not questioning any intentions of yours or anyone else's. In fact I mention clearly that there's been talk of doing this, just not much action because we're all (justifiably) worried that splitting things up will make life vastly more complicated for users and to some extent developers. The amount of work you've put into OpenACS - especially v4.x - is enormous, and I wouldn't dare question that. I'm not criticizing anyone for the current situation, only stating how it is and trying to help things out a bit.
I'm also not trying to rehash any old discussion, although certainly I hope that a solution like this one will simplify future discussions and maybe even help prevent confusions.
This is a proposal for discussion. It takes empirical evidence into account: to date, we've only added to the core, never taken away. Packages like acs-workflow which get little use are still installed by default. Packages like content repository which may not be necessary in certain installations are still installed by default. Packages like new-portal are being considered for addition to the core as required elements when this may not be in everybody's interest.
This stuff needs discussion! Forget intentions, forget hopes, let's talk action. Is this useful? How so? How can it be tweaked to be more useful?
First of all, I'm in total agreement with everyone that a monolithic openACS is not the direction the toolkit should be going. I'm glad that most involved in development see this need and are talking about it and making plans to make it more modular and easier to handle.
In addition, I think MIST is a great idea that should be considered and discussed on its technical merits. It is true that most users of openACS will need to be pretty savvy to get everything installed correctly, but for the most part this only requires some attention to the docs and some unix/linux experience. With the current way of setting up a site, you have to have a pretty good familiarity with the toolkit in addition to being comfortable in unix. This is something that not everyone will have right off the bat. Being able to choose an "distribution" that contains only needed packages and is set up close to the way the user wants will allow him to gain experience with openACS more gradually, while giving him a good, functioning website with a minimum of previous openACS knowledge.
Personally, I could see myself making my own MIST scripts to help speed up the process of installing my own websites to get a base of common packages I normally like to use.
Thanks for your feedback and your very valid points. The most important response I can provide is that I think MIST is an idea that will become vastly more important as time goes on.
Today, there's a bunch of packages in OpenACS and in dotLRN. That's just two big chunks. But as other projects take off (dotWRK, dotORG, etc...), there's going to be more and more conflicts as to what code should be downloaded to make something work. As Don and Dan have pointed out, it's everyone's goal to split things up into individual packages soon. That will make the development job more flexible, but the installation job more difficult.
MIST is meant as a patch to make the installation part easier and more flexible to different user needs.
And if you want to still see a monolithic distribution, that's fine, too. I'm only trying to add options here, not take any away.
Finally, since you mention Linux distros, remember that there are many. RedHat, Debian, SuSe, Mandrake, etc... That's *exactly* the idea I'm pushing here. I'm saying that the OpenACS architecture is like the Linux kernel, the packages are like the many pieces of software on top of the kernel, and we, OpenACS.org, are not RedHat or Debian or Mandrake. We are Linus, and we want to *enable* the presence of RedHats, Debians, and Mandrakes.
(and yes, as someone who took high school German, I should have remembered about the name... ah well, certainly a better name is probably a good thing :).
If someone has time/resource to make OpenACS easier to intall then I hope we can start on the installation of OpenACS core itself. Of course Ben and Openforce can do whatever they want. I think MIST will greatly improve existing OpenACS users... which is us. But I still think getting new people is something I would like to happen more even at the expense of still doing CVS, etc.
I think Red Hat had the market share now since way back then it was easy to install compared to other distros. Debian had superior package management, but RH back then can be installed easier. Mandrake is here, I think due to its easy install. Windows is here due to the lack of installation, its comes preinstalled. Apache is dominant because its also preinstalled.
I apologize Ben for voicing out this opinion. Your concern and my concern I guess is a little different. You also have the time and resources to address your concern. Just trying to see if I can push my luck and hopefully my concern of getting OpenACS easy to install to attract more new developers may become your concern too. :)
I'm aware that for a long time everyone has agreed we need to address this a move to a more modular approach, and its been time and resources held that back.
Having skimmed your document Ben, I generally agree with the approach (but I need to consider it in more detail).
But... your tone, and the unnecessary ramblings which *do* seem to be taking side swipes and highlighting positions that simply don't exist is not all that constructive.
Let me take just one choice example
'Our current thinking is leading OpenACS 4.x down the monolithic, un-modular 3.x path.'
This kind of comment is unfair, untrue and unhealthy... So why's it in there Ben?
I can see you may have an evangelical or political persuasion on this, but mixing it up under the premise of a proposal is a little, tiny bit underhand.
Would it not have been better to put up a *neutral* proposal for what you'd like to see, and then post separately if you want to soapbox, making it clear thats what you're doing?
And of course, checking that something is *actually the case* before you throw it at us!
How? How is dividing up effort useful? Aren't we going to end up answering ten times as many newbie queries along the lines of 'so which one the X number of bboards is the one I should use for....'
I am certainly very puzzled about what you actually want this community to become. Its beginning to sound to me (and I'm only going on what I have in front of me) that you want some sort of hacker club for constant experimentation and diversity. I have no objection to diversity per-se... but this 'chaos theory of software development' that seems to be the undercurrent of many of you posting is asking for disaster.
I'll use my favourite little Japanese proverb again as its telling of what seems to be emerging.
'Anyone can paint a Demon, but painting Dogs is far harder'
It seems the discussion is not about configuraton tool (MIST), but about architecture. If OpenACS project adopted layering pattern, it could be easier to maintain and extend. The question is it OpenACS objective?
The essence of the pattern is that lower layer provides services only to upper layer. Lower layer never depends on upper layer (i.e. core OpenACS never depends on dotLRN). So architecture looks like this:
- System services (Linux, Postgresql, AOLserver)
- Foundation layer (authentication, objects.)
- Core application layer( forum, news, portal, calendar etc.) aka "business object layer" -a set of basic applications to solve problems in a particular domain (in OACS' instance - task of building online communities; but somebody may build a set for building stock market trading system, for example)
- Standard Applicaton layer (ready solutions for more specific task, such as community aimed at learning: dotLRN; or a "typical" blogger community for news discussion, or a web-store with attached community of customers (does it ever work?) etc.
- Custom applications done by customers to their own needs.
As you already noticed, it looks aufully like Enterprise Java. Nothing wrong with that, but with all those layers you may expect slow performance. Besides, it's inefficient trying to do object oriented programming in scripting languages like tcl. Why not direct in Java then? The language was designed for this stuff, after all. Example of ezPublish shows - you can build decent object-oriented framework with PHP, but it will be sloooow... And you never achieve stability of Java anyway.
But hear this. On the other hand, if you "compress" above core layers 2 and 3 into a monolitic OpenACS, you will still be able to provide basic services needed to build online communities. This application will be very difficult to upgrade and maintain, but who cares since people will be constructing new "types" of online communities on top of that (learning community, shoping community, political party, religious group etc). As far as core service set is stable and doesn't change (forum is forum after all and news/comments are news/comments) - nobody cares if there is a "expandable" object framework under it. Eveybody cares for speed and stability, though...
You can go even further and stuff your code into database (triggers and what you got), simplify your table structure (and break your neat data object framework) - and you can achieve a blazing speed for your application. This will lock OpenACS in PostgreSQL and Linux, though. Or you have to write another OACS for Oracle, more or less.
If anyone remembers, Greenspun had argument against using n-tier architecture in ACS. Why create layers when you know exactly what web services you will need in the end? Do most online communities need stock quotes? Do most communities use Windows? What is obvious enterprise-class database? So, ACS was made monolitic, married to Oracle and Solaris. Point. The system could scale up like a beast on a simple machine. The same machine that would choke on its first EJB...
So, I would argue 3.x system was made monolythic on purpose as one big "business object" that is stable, fast and provides only basic services to create online communities. After that all creativity and talent of the aD went into creating business applications with the toolkit.
Which path OpenACS community will chose, depends on its members vision. I see danger in trying to build Enterprise Java-type of busness object framework in tcl, however challenging it may be. On the other hand, if layers 2-3 (above) are made stable and optimised for performance - you have one great core business object for community building, including built-inIDE to create application like dotLRN - and many new ones. That is what I see as differentiating and unique feature of OpenACS under the sun.
I like the idea of MIST, and it certainly sounds useful if the only onus it puts on developers is accurately describing their dependencies. How much more work beyond what's included in current apm files will this entail? Would package maintainers have to also keep track of 'core' patches to keep everything independent?
Also Jun, I should mention that I'm working on reimplementing all of the ns_* commands in pure tcl to enable FastCGI deployment of OpenACS (link). Although not my direct project goal, one of the early deliverables is a tclsh auto-installer for openacs. I've got that part working so that the required modules in openacs core are installed in Postgres using just a 10 line tcl script.
Between this work and tclwebtest for automating or scripting installers, I think we have a good basis for making current and future installation processes more straightforward. Having a framework for walking dependency graphs and managing patches seems like a logical next step.
In a world of limited resources, however, I agree with Jun. My money would be on making OACS easier to install.
Regarding new-portal. Up to this point I haven't participated in the discussion. Clearly this is something of a pandora's box. Politics aside, it seems like there is room here for good-natured technical debate. On the one hand, there is a big advantage to using new-portal for admin pages. Namely, you get nifty auto-generated admin pages that expand as new packages are added to your system. dotLRN showed us that this is possible not to mention very cool. But the downside, as Ben points out, is that it adds yet another required package to OACS and generally raises the bar for writing new packages. To me this seems like a reasonable concern.
Perhaps the right solution is to have "fancy" and "lite" admin pages. The "fancy" pages would work like dotLRN and require new-portal. The "lite" admin pages would query the site map for all instances mounted under the current subsite, and link to package-key/admin for each instance. This way we could take advantage of new-portal when it is available, but not require it.
Ok I've had more time to look through the document in more detail.
I'll start by picking up a few points in the text, and slip in my comments and conclusions there.
I'll start with the summary section and work down
'However, we are currently pursuing a development route very much in conflict with this goal.'
There is no reason to suspect this. We are pursuing a development goal of a suitably stable release (4.7) from which the issues raised in your document can be properly addressed. The ability to do this has been hampered by limited resources.
'We are adding more packages to the core, adding more dependencies to the release of a monolithic distribution, and generally slowing down development.'
Its hard to define what 'adding to the core' means. If you mean its all downloaded as one, well yes thats true (as no suitable system for separable downloads is available at present other than CVS). Nmost packages have dependencies, but it is not conversely true that they all have to be installed. As for references to slowing down development, this appears to be baseless as development in recent months appears to have increased if its changed at all. It also extremely difficult to find examples of how exactly the issues of a single release has slowed development. As a product matures and stabalises, it is only natural that new additions have a greater effort in terms of meeting the current quality levels. I agree that just hacking out new stuff without consideration for the state of the rest of the system is quicker.... but not necessarily better, as this statement seems to imply.
'If we do this right, we retain all the current advantages while gaining faster development, a leaner core, and generally a technology environment that makes it easier for anyone to innovate.'
Yes, no disagreement there but, your premise here is that innovation is the ultimate, if not only, goal of the project/community. It is not! That may be your primary goal, but this cannot be applied 'blanket' style to everyone.
'Certain specific packages (e.g. calendar) have plenty of bugs left, but the code freeze date for OpenACS 4.6 is past due: calendar bug fixes will have to wait for OpenACS 4.7. Progress of an end-user package still depends on a central release date'
Well of course. If its not to trite to say so, thats software development bud! The reason for this is that we are trying to get some focussed testing done on this (not something this community has traditionally been all that good at, OpenForce included). For that to have any meaning we need some sort of cut off to work against. That doesn;t prevent bugs being fixed etc.. what it does is prevent chaos-hackers whacking away at code while people are trying to assess it. No-one likes working to a constantly moving set of goalposts.
'People interested in a subset of dotLRN have to download all of dotLRN'
If thats so, well, that is largely down to OF in the first place, secondly this is not the OpenACS and thirdly this is now an issue for the Governance Board to address, from which you have opted out.
'People interested in a subset of OpenACS have to download all of OpenACS.'
Yes, everyone accepts we'd like a smaller download. But this really only about download time. This doesn't imply people have to use/install everything.
'"Rolling package X back into OpenACS" is generally considered a good thing, but it has at least 3 different meanings. Plenty of miscommunication occurs based on this confusion.'
You make a very salient point here, and I agree a definition or common understanding of what that means is required.
'no one should ever be forced to download or install new-portal if they don't need it.'
Yes, but my understanding is that in general new-portal is very popular as a concept and a lot of people have said they'd want to see it as an inherent part of OpenACS. Therefore, your personal preference aside this is a moot point. The decision seems to have been taken, and so thats the way it is. There's no point rehashing this debate again.
'Unfortunately, these properties are too often ignored in our discussions and development process. We've lost touch with the true nature of OpenACS modularity.'
Who has? I for one don't like being spoken for and this kind of comment is personal conjecture only. You think we've lost touch. I disagree most strongly.
' Under this proposal, lars-bloggerbecame "part of OpenACS" only a few weeks ago, when the source code was added to the OpenACS central repository. Did something technically important change when Lars moved the CVS from his site to the OpenACS repository? Not really. Should the location of the code really matter? Not if we can help it. '
Location does matter in as much as we want to make it easy for people to find what they're looking for. As you say, if the location doesn't matter, then it doesn't matter if its all stored in the same place does it!
We must find a more sensible and balanced approach.'
Yes, overall I agree with you, I just think you're making the point based on perceived problems that don't necessarily exist.
The 'Two Questions, Two Answers' Section. No one has said that your package has to be in the core to be part of the OpenACS... I just wonder where you;re getting all this from? You also refer to 'Our' current thinking a lot... Who the hell is this group of people? My thinking ain't along those lines, and I don't know anyone else in the community whose is either. I'm sorry but this sounds like creating an artificial sense of something is wrong to justify your own argument.
A Leaner, Meaner OpenACS'. Firstly let me say I think we're all working towards that. And secondly given the bloated nature of dotLRN, I think there's an element of 'people in glass houses'. If this is a highly motivating issue for you how do you reconcile that with the nature of the dotLRN distribution?
Although I agree with the premise, again I have to express concern over the '
MIST does away with the ever-growing and now bloated OpenACS core'
Warming Warning Warning... shameless negative marketing. This kind of tish really gets in the way of a serious discussion.
Ok, now finally I can comment on the scheme your proposing.
Apart from your obsession with Debian as an great exemplar, which I think is open to argument, the scheme your proposing seems pretty workable, and I think I probably would get behind an effort of this kind.
I think the only sticking point is not ending up creating a diverse, but incredibly thinly spread community. As you point out it may lead to umpteen Content Repositories. This is dilution of resource, plain and simple, and to be honest it smacks of hacker mentality. If there's one thing guranteed to piss me off its computer nerd banging on about how much better their version of something is compared to someone else. All you end up with is lots of shit, half finished code. If we can think of a way that might avoid this senseless dissolution, then I think the scheme's got a lot of potential.
Just think how much more positively you suggestion might have been received has you not indulged in something of an egotistical rant about the current state of things. I think Don echoed similar thoughts earlier in that stuff like this does make you feel like saying 'sod this, I'll spend my time and energy elsewhere'.
I for one would love to pontificate about what might come next, and how shit everything is, but I like Don and many other are also actually having to do the hard, and less glamorous tasks that will actually get a 4.6 release out.
Maybe you didn't intend it this way, maybe you just have slightly poor written communication skills, but either way the net effect is a good idea, somewhat spoiled by a lack of respect for the hard work people are currently putting in, the difficulties that that entails and a lack of sensitivity in your approach.
I'm sorry I can't be more conciliatory about this, but to be honest your article is insulting.
'Its easier to paint Demons than Dogs'. Unfortunately I've got testing to do, and thats a dog if ever there was one.
Thanks I am aware of your work. I hope it does work out fine. That way we can have an OpenACS that other people can try out that only depends on tcl and postgres, or other people not willing/unable to install Aolserver.
Good points thanks for reminding us about it.
As you probably know I *rarely* post flames of any kind. But I reserve my right as a member of this community to express my opinion and respond to posts.
As you will see I have commented on the technical aspects and the architectural ones, but given that Ben's original document was a proposition based on a number of his observations/arguments, I am also going to comment on that.
I certainly resent your apparent 'chastisement' of my posting. I put a lot of time (and company money/resources) into helping out here and I for one am not going to sit quietly whilst Ben using derogatory, dimissive and frankly bloody innacurate descriptions of the current state of OpenACS. *That* is certianly no help to anyone, and very damaging if it gives off the wrong picture of the current state. These BBoards are not mere soapboxes for Ben (or anyone else) and I'm sure he understands that.
But hopefully now I'm sure Ben will have taken the comments on board and we can now stick to discussing the actual substance of his proposition. In which I must add I am very much interested.
My post was not a "chastisement" of your post, and I meant nothing personal with it, so don't assume I did. I also never mentioned anything about your (or anyone's) freedom to post whatever you want (including tags in the subject line --with the exception of the h2 tag--, which continue happenning), so I don't know what that was for.
Kudos for your investing of resources on OpenACS. You're not the only one. Many others do it with and without profit reasons.
My point is that disagreements/critics about the document at hand can be brought up (and evidenced by other posts in this thread) without any personal attacks, and that is a good thing. That's all.
Disclaimer: No personal attack was intended on this post.
(I've only done the <h2> thang once.... honest)
Well, doh! That's why I find Ben's propping up of strawmen to knock down, strawmen based on a misrepresentation of truth, strawmen that misrepresent my personal position, so offensive. What is technical about this approach?I think this discussion would be more productive if we focused on the technical issues at hand, behaving objectively and stopped with personal comments/attacks. There's simply no need for that and it doesn't do any good to anyone.
Sorry, when my position is misrepresented to this extent, I'm not going to sit by quietly. Ben and I had this discussion months ago, after which Ben stated that he understood my position regarding monolithic releases and synchronous schedules, and that he would agree that new-portal should be part of the OpenACS, rather than the dotLRN, project.
Why is the new-portal issue being raised once again? I'm tired of it. Done deal, folks, right or wrong the discussion's been had and the discussion is over. OpenForce originally proposed a new portal package to be part of our standard OpenACS toolkit. Ben has made no compelling argument that convinces me, at least, that his original decision that it be part of the OpenACS toolkit is wrong. We're sticking with OpenForce's original proposal.
As far as something like MIST, sure, it's a great idea.
I personally think we shouldn't make users depend on it, though. The linux world has apt-get and up2date, the BSD world the ports system.
But they also have ISO images, individual packages available for ftp download. "centralized"? Well, many packages have both their own home *and* a slot on, say, RedHat's centralized ftp server. Why? How about "doing so makes it easier for users"?
Anyway ... before we worry overmuch about how to distribute a modularized distribution, don't folks think that we should first try to figure out whether or not we have the resources to manage such a distribution and, if so, how we'll divide up the work and so manage such a distribution? It is my hope that folks will step up for 4.7 and that we'll have the resources available to discard the monolithic tarball approach as our sole means of delivering code.
But I see no guarantee that this will happen. Personally I'd like to see Open Force step forward and help with this aspect of the problem (not to mention to step forward and help with the nitty-gritty work needed to get 4.6 out the door, i.e. testing, bug fixing, and the writing of missing upgrade scripts). There's no point of writing MIST if we can't gather the resources to properly manage modular releases.
The actual means by which we shove the bits down the user's pipe is not, IMHO, on the critical path. Once we have a more modular distribution we'll probably want to deliver it in several ways.
If we were pushing 600MB down the user's pipe then splitting that delivery into manageable pieces would be absolutely critical, but let's face it, a 10MB payload isn't large enough to bother many people.
How many users have actually complained about the download taking too long? How many users have complained about the initial install of the core taking too long? (the *core*, not all the packages one can load optionally after the initial OpenACS install).
I've received exactly zero such complaints, myself, and as Project Manager I'd expect to hear about it if many were complaining.
So I look at arguments based on size of the existing tarball and install time of the core as being weak, technically, since I see no evidence that any real users are bothered by either issue.
However, we all know that there are other reasons for wanting to move from a monolithic release scheme. It's going to happen, the only question is when, and that's entirely a matter of available resources.
On a point you raised, I have had one or two people tell me they are bothered by the size of the distribution, but they don't mean the download time, they actually mean that for a newbie its 'looks' like a lot of stuff and thats a bit daunting. I guess, now I think about it, if I hadn't come across it before, it does seem quick a bit of code there.
But I think it not a question of splitting up bits, but could be immediately and effectively releived by improved introductory documentation explaining whats there, whats important, what you don;t need to worry about initially.
another threads been raised about documentation (with which I agree) so I'll leave further discussion on the to that thread
A few responses:
- Don, Simon, Dan: I've had 4 people re-read this document with me. I've reread it myself 10 times at least. I don't mention anyone by name, I don't ever imply that anyone has done something wrong, I say that we are going down the wrong path. we includes me. This is not a criticism of anyone.
- Dan: the current OpenACS distribution is only getting bigger, with nothing ever being taken out. Both the core and the tarball. This is not opinion, this is fact. All you have to do is check the CVS. And since OF has contributed to this, I'm once again not accusing anyone, simply pointing out a fact and a potential solution.
- John: ideally this puts no more stress on developers than putting together a MIST script, as you said. But your worry is a good one, and we should keep that in mind if we move to actually planning development of MIST. If MIST ends up making developer life too complicated and user life only moderately better, then it's not worth it. Thanks for pointing this out!
- Andrew: right on! fancy-ui and simple-ui is exactly the kind of idea that I want to see with MIST. If people want portals-based UI, they should get it, but if it doesn't need to be imposed on everyone else, then why not make it modular?
- Simon: I think something like MIST would *help* on the testing front, meaning you could stagger testing of different packages and begin doing more unit testing instead of the monstrous integration testing you have to do every time. You're putting in an incredible amount of work to do this (and thank you!), which I think could be eased a bit.
- Simon: yes, currently, dotLRN has to be downloaded as one big tarball. Not because of dependencies (we've done the best we can to be as modular as possible), but because in a world without a MIST-like system, how else can we do it? That's why we're proposing MIST, so that people can use parts of dotLRN without downloading the whole tarball.
- Simon, again: context is important. From a technical standpoint, location of code shouldn't matter. From a management standpoint, it matters very much. MIST tries to make location unimportant so that management of code can be performed in whatever way the code's author prefers.
- Don: you say "there's no point in writing MIST if we can't gather the proper resources to properly manage module releases." I say there's no point in modularizing releases if we don't have an end-user tool like MIST to keep it simple (and even make it simpler) to install. This is what's worth discussing! Let's hash it out.
I'll try to summarize again the points here:
- yes, many people, including Don, Dan, myself, Simon, etc... have expressed the desire to move to decentralized packages. I don't question that.
- at the same time, we haven't really done anything about it to date. Discussion doesn't count as action. That's what I claimed in my proposal and what I continue to claim today.
- if we do something about it without something like MIST to make installation easier on the user end, then we will find ourselves with an even less usable system, or force users to resort to whole tarball downloading... in which case, what's the point?
Once again, I am not cricitizing anyone in particular. Check out the proposal again. I say "we". "We" are a community. "We" includes me and OpenForce. We're all in this together.
However I do want to point out that my understanding has been for some time now, that a more serious effort on modularisatio etc had always been anticipated to occur around or just after the 4.7 release. So far from not making any progress, as far as I'm aware everyhitng moving forwarg against a timescale thought practical.
(PS, I still feel the tone of you original document was unnecessarily critical,and I still feel you are over stating somewhat the current situation and the priority/seriousness of the current deficiencies)
As far as the size of openacs, the number of packages has increased, but when you load the core, you're still loading basically the same packages that were loaded when aD released acs. We need to do a better job of documenting what we have, as Roberto suggests, but having more choice for modules can generally be considered a good thing. I sympathize with the plight of newbies when they're trying to decide what is useful in openacs, but having Mist won't rectify this problem.
I think something like Mist could be a good addition to openacs, but the wording of your proposal is inflammatory, and it would have been much better received, if you had made it a purely technical proposal.
Query files aren't cached unless the package is installed, nor are Tcl init files sourced. A core-only install, then, doesn't eat memory based on the number of uninstalled packages which happen to have been dumped in your install directory. Changing the distribution methodology wouldn't change the size of the minimal footprint, then.
Removing some things from the core install, say the content repository, would shrink the footprint but that's true independent of the physical means by which the toolkit is distributed ...
In the spirit of a true meritocracy I'd prefer to allow people to assess the arguements for themselves and consider the merits of any such arguments rather than reduce it to a hand waving exercise. After all I don't think its fair to ask people to 'choose you side' like that.
I suspect you didn't mean it that way, it just came across so (well to me anyway ;)
At the moment, ok it comes in a bigb bang and then nothign for a while, so I can see the benefit in spreading that about a bit, but the scheme seems to suggest (looking forward a year or two) a large collection of suff, scattered about quite widely and I mean that in the sense relationship to the community, and various developers working away each on an entirely different schedule, with there being greater difficulty therefore in every acheiving meaninful points of synchronicity.
And due to this arrangement, the it does make the business of organisign and managing a testing effort to get a fairly even quality level becomes much harder (and more time consuming ;).
I'm not saying that this is necessarily something we can't put a different kind of scheme around, but I wanted to highlight that the MIST scheme, if adopted in this full form, will also have quite an impact on other activities, and therefore is actally a pretty radical overhaul of the way things are done now.
I think its may well be a goal we'd like to reach, but perhaps this (and the things its related to ) might be broken up into baby steps at least, or perhaps change slightly to accomodate the other activities
I think Ben courageously takes the initiative and tries to put focus on architecture and strategy. He probably doesn't need my defence, but his approach will take OpenACS further. Seriously, further. To enterprise-class application level where it deservingly belongs.
But the attitude I see in this forum makes me feel this is PHPNuke. So many modules, all the hackers happy, plenty of politics... For God's sake - what is this: "Since Don has been leading the project, you're really criticizing him..."? This is the level of discussion at OpenACS today? Are new ideas welcome at all? Are you developing political correctness or software?
Thanks for the telling me off though... consider me suitably chastened...
- Move away from monolithic releases
Everyone agrees that we should not have single monolithic releases of all packages. It's unwieldy and makes the release interval too long.
- Provide a Configuration Management tool
Important to have and hopefully someone will step forward to write one. It's not clear that it has a lot to do either with having a central package repository or whether we do monolithic code releases or not.
- Provide a Central Package Repository
Again a good thing, someone should write one and fortunately APM is almost there. The metadata you would want to keep is in the info files already and all we would have to do is have a place to upload/post .info files and little bit of code to manage it. Unfortunately the current metadata in said .info files is complete crap. Hopefully if it meant something people would spend a few minutes cleaning them up.
On monolithic releases; I consider it a waste that we do not leverage CVS. If we used CVS properly and maintained a stable and dev branch we could do relatively frequent bugfix releases from stable and rapid cutting edge development on dev.
It is really little additional work and we would derive great immediate benefits, the biggest being that on stable we could limit data model changes to those with well tested upgrade scripts ensuring that production sites would be able to get bug fixes and be more confident about installing new packages.
Maintaining a central CVS repository makes merges dev<->stable much easier. and provides a great deal of transparency.
On configuration; ultimately, configuration is an awful lot more than just which packages you install and where you get them, and it's something we definitely need to think about. Although when I consider the state of the documentation I am not sure the effort is not better spent there rather than on a complex feature complete tool.
Also, From a purely technical standpoint I think I am leery of using a config tool which has Tcl code in it or depends on a lot of other tools to be installed. My preference would be for it to be data driven much like the current APM, although I guess I could be persuaded that the flexibility afforded by such a scheme might outweigh the dangers (the temptation to hack things in instead of using a clean Mist:: API, the risks of wiping out existing configuration data, obscure hard to reproduce bugs etc).
As for a package directory/repository, I think a central place to record where different modules live is great, and a download repository is great as well, but I think an awful lot of projects get by on CVS and tarballs and I think that's good enough for now.
All this is a long winded way to say that all the ideas are good ones and should figure in the plans for OpenACS but the problem is now (as it has been) who will write the code and is it what we want to spend those resources on. I don't believe either of the two code things are the most important things to be focused on, and I think we can avoid monolithic releases by simply overhauling some of our practices and educating developers about CVS. My list of major issues are:
- Documentation -- it is currently "not so good"
- Looks -- also generously described as "not so good"
- Bugs -- too many
After all, given the new users complain the OpenACS is a lot to learn/understand, we don't want them to have to learn the tool they need to get it in the first place...
I think you've hihglighted the fact that MIST seems more of a developer aid than a user/downloader and someone who is developing client work, but not work directly on the OpenACS. And perhaps therefore needs to broaden to include all developers/user?
- Dan: you're implying that if I say anything negative about OpenACS, I'm insulting Don. Should I just shut up unless I have good things to say, then? I see certain issues, and I try to propose solutions that we can all discuss. There is nothing disingenuous in what I'm saying. I am not blaming Don. If anyone thinks I'm blaming Don, they are wrong. I didn't blame him in my paper and I'm not blaming him now. I simply think we all have made a mistake in delaying something like MIST. I'm looking for rational discussion around that idea.
- Dan, again: I think the core was already too big when aD released it. And in fact, I seem to remember that we all agreed on that. We recommend using acs-mail-lite instead of acs_mail whenever possible. We agree that acs-messaging should probably be dumped. We agree that acs-workflow should not be installed as part of the core. Some don't want content repository installed by default, either. So let's do something about it. It wasn't right 2 years ago, and our lack of action doesn't make it any more right today.
- Don: agreed that we can shrink the toolkit size without MIST. That's an excellent point. Now, I'm not so much pitching a new way of breaking things up, I'm pitching a new way of putting them back together for users. If it's easy to put things back together, it's an easier decision to make to break things up.
- Jeff: great comments and great analysis, thanks. I initially thought about an XML-driven MIST, but then I realized that the installation itself might be a bit more functionally-oriented than data-driven. But certainly nothing is set in stone, this is just a proposal. Understood about the documentation complaint. We're trying to fix this first by doing the new-portal cleanup with documentation in parallel.
- To clarify, the source code for new-portal is available to Janine, who manages both dotLRN and OpenACS, and she can make any decision she wants on what tarballs to include it in: the code is owned by MIT Sloan and managed by the dotLRN boards. OpenForce is only wrapping up cleanup and documentation as a means to help future work in that direction, and will provide that code ASAP to Janine for inclusion in whatever version of OpenACS and/or dotLRN she sees fit.
- Jeff, again: OF is definitely willing to contribute resources to build MIST, sorry I didn't make that clearer. That's why we're making the proposal, to see who would find this useful before we put time into it. No client is paying for this. This is definitely not a request that current volunteers shift their focus to MIST. This is what I'm interested in and I thought it might be useful to more than just me, that's all.
- Simon: on the contrary, MIST is meant as an end-user tool to enable end-user configurations. Developers would use MIST only when it comes time to package a distribution of the stuff they've written.
However, by way of a bit of a side step, could we agree a time when we should consider this more a priority. I think if nothing else with a 4.6 release in the middle of beta, and lots oif requests to get the 4.7 release out, can we at least agree its not worth jumping into this until the 4.7 is near completion?
I for one would have more time to contibute and consider this more thoroughly?
Perhaps a certain amount of effort pegged for this, should be set to at least getting what we have as organised as well as it can before starting on a MIST initiative, I guess thats sensible in any case
rectify the perceived problems, but the tone of your proposal suggests
that you have another hidden agenda. In addition, you're rehashing a
problem that has been discussed at length before. If you trimmed your
document to a strictly technical proposal, it would be less than half
its current length, and we would currently be arguing the technical
merits of it instead of having this fruitless discussion. Even
better, if you had offered to implement it, your proposal would have been warmly received.
Yes, acs-messaging should go, and acs-workflow should go, but I'm not
so sure that the CR should go. In past discussions, we agreed that packages
should move towards using the CR, so that common services like search,
categorization, etc. can be enabled easily for supporting
cross-package aggregation of content management. I think it was a mistake to not base the new forums packages on the CR, since we now have to add additional code to support searching and categorization of forums that otherwise wouldn't be needed, if it had used the CR as a base.
We don't need to use mist to rectify this problem. I could go into cvs
and easily fix that problem right now, if we had a consensus to do so.
If you wanted to change this, you could have done it yourself (years ago) after a
short discussion on the bboards.
What would make OACS better and more useable, in my opinion, would be un-installation/upgrade help. Installation is rarely an issue. The current APM seems to do a very poor job at removing package instances. The design of the ACS kernel/core makes deleting data nearly impossible, leaving behind trash objects.
The core should be revised so that data belongs to the core, not to the application that creates the pageflow/ui. A package would load by creating any tables found in the sql directory and telling the core what kinds of 'objects' it needs to run. The core would create the basic insert/update/delete functions (if needed), and add any specialized functions found in the sql directory. An application/package would interact with the core for create/update/delete operations via a simple api.
Deleting an application would not delete the data associated with it. This would be done by a site admin as a separate step, after considering the consequences. Multiple applications could access the same data. In this way, a portal, for instance, would simply be a separate application.
In simple terms, the core would be responsible for maintaining and protecting objects, everything else would be an application, hopefully with no additional layering.
Anyway, I'm sure this idea is pretty dumb, but I'm stickin' with it. Once these problems are solved, installation will be a breeze.
I'm very sorry I didn't make that clearer. I thought it was implied when I posted the proposal, but apparently it was not: OF is definitely proposing to put development power behind this. We'd *like* to work with the rest of the community who's interested and who's already working on packaging, but certainly we meant this as something we would take the lead on.
Stating that "this will happen in 4.7" counts as action. There's been work done on the new incarnation of openacs.org to support modularized releases. You claim this won't be adequate and perhaps you're right, but the fact is that work has been and is being done to support a move away from monolithic releases.at the same time, we haven't really done anything about it to date. Discussion doesn't count as action.
At the risk of sounding rude, this reads like it was written in 1972 by Richard Nixon's speechwriter.There is nothing disingenuous in what I'm saying. I am not blaming Don. If anyone thinks I'm blaming Don, they are wrong. I didn't blame him in my paper and I'm not blaming him now.
However ... I'd much prefer that for 4.7 we coherently try to prioritize shortcomings and problems in the toolkit, encourage people to make proposals for work on the toolkit they're interested in pursuing, to try to set a rough schedule for code freeze and testing, and to then - as a team - finalize a set of things we want to accomplish in this release.
And I'd then prefer that OpenForce work as part of the team to make sure that work items that make the cut for 4.7 actually get done. A distribution tool like MIST may or may not survive a prioritization process for 4.7 (or 4.8 or 4.9 or ...). Actually I'd prefer we set forth problems and prioritize them first ("bug fixes", as Jeff Davis mentions, is high on my list; so is "scalability", still an issue) and explore proposals to solve them afterwards.
Software projects need to be carried forward in a disciplined manner.
As the resident Release Mother, err I mean Manager, I have a request:
Could this discussion please be tabled until the 4.6 release is out the door? I think it has become a huge distraction and I need the attention of many of the folks who are posting to this thread.
Perhaps we could have a general rule that no non-emergency distractions should be brought up between code freeze and release? At least for these smaller releases, where the time frame is pretty short. Considering how thinly stretched our resources are already, I think this would be a big help in getting the things done and out the door.
I could go on at length about my opinions on the subject at hand, but I'll be the first to set a good example and keep them to myself for the moment. :)
From this, no one should draw the implication that OpenForce failed to deliver its contractual obligation to Sloan. OpenForce was contracted by Sloan to build SloanSpace v2. It has done so. We are live with SloanSpace v2 and we are very pleased with the product.
I was referring to dotLRN, which is an open source effort in which many of us participate. I apologize if the distinction was not clear.
Your comment that OpenForce has "pulled out of dotLRN" seems to imply that OpenForce will never do anything else with dorlrn on help in its development as an open source project. Is that the case or has OpenForce only refused to participate take a seat in the TAB (technical advisory board)?
My understanding was that it was the latter, and it's an important distinction.
You are correct. The distinction is important. I have spoken with Ben and he will be posting a reply soon.
Roberto: OpenForce has turned down participation in the dotLRN governance structure because we feel the difference of opinion is such that our participation wouldn't help.
Our participation in dotLRN code is a somewhat linked, but mostly separate issue. Yon and Arjun have begun the massive undertaking of seriously cleaning up (and simultaneously documenting) new-portal (even if it's soon to be part of OpenACS, it's code that emanated from the dotLRN effort). We will continue to try to answer questions about dotLRN and provide as much documentation as our time allows.
In general, we're going to continue with what we've always done: as a company, we have to worry first about client-driven work. That said, with any client-driven work we undertake, we will attempt our very best to make the product available to the community as quickly as possible. Where possible, we will also undertake our client development in an open-source fashion, where we involve the community in the development process. If these contributions fit within the dotLRN world, then surely we will work with the dotLRN governing bodies to contribute within the approved mechanisms of the dotLRN community.
I hope this helps to clarify things.