Forum OpenACS Development: RFC: automated installer
In looking at his script, which is an excellent start, it seems clear that a truly comprehensive script is going to have to be flexible in several ways:
1. It needs to support all the major *NIX variants, which will mean that paths and commands will be somewhat OS-specific
2. It should support customized installs, so that someone who already has Postgres installed, for example, can skip that piece
3. It should be easy to run it multiple times on an already set-up system without worrying about clobbering anything
4. Although it should use defaults that mirror our installation documentation, it should allow the user to use different locations and names if s/he chooses to.
David's script does not actually install OpenACS, just everything you need to prepare for the install. I know that an automated installer for OpenACS/dotLRN exists, because the Collaboraid test servers get reinstalled automagically every night. I would like it to be possible to invoke that process from this script, when it's ready for prime time.
My thought at this point is to split the install into at least two parts. The top-level script would allow the user to specify what pieces they want to install, and the second one would ask the user for desired names, locations etc, always offering reasonable defaults which mirror the recommendations in the install docs, and then do the install. If it's too much to do in one script it could easily be split up into one script per package to be installed.
I'm not sure what to do about OS differences in paths, commands, etc because I'm not sure yet how extensive they will be. I would prefer to keep it all in one set of scripts, using variables to work around the differences, but it remains to be seen if that approach will work or if it will end up being unreadable.
The consensus when David wrote his TIP for this (http://openacs.org/forums/message-view?message_id=117914) was that the scripts should be written in Bourne shell (sh) so I'll stick with that.
Any comments or suggestions on how I should (or should not) proceed?
That is a tough problem. One I have been working on for a while. I actually have a tcl based installation script, currently tagged 'app-install', which installs any software that can be installed via a command line.
Tcl is good for a number of reasons, not the least of which is that it _stops_ when it reaches an error. Shell scripts tend to keep chugging along. Currently my script runs through a loop for each application:
- Query the user to set variables, using a default if available, i.e. install locations, usernames, etc.
- Run a series of commands. If a command fails, prompt the user for advice: abort, skip, substitute command.
- Write/append/permission files.
- Download/install tcl
- Download/install daemontools
- Download/install aolserver
- Download/install aolserver-modules (including tDOM)
- Download/install postgresql
- Download/install init.d script for postgresql
- Create OpenACS startup scripts -- nsd.tcl, db user, createdb.
- Download/install OpenACS software.
- Setup daemontools control directory for service.
- Initialize cvs repository.
I ran through the entire 5.0 install doc and used the defaults provided for a postgresql install. I found a number of errors in the docs by doing this.
I have to admit it is a bit tedious to setup, but it does work. All the commands which are actually executed are recorded in a log file along with any output from the commands.
I have command scripts for the following:
Usually all that is left is to start up the service:
Then install OpenACS by filling in the form.
Each application is independent, but they can be chained together and variables set previously default to the already entered values.
The main issue I need to address is to switch over to using sudo. Some commands were executed 'su'. In general it would be nice to use sudo on every command, and allow switching users. Then some configuration of the sudoers file would be required to run the script.
Sounds very useful!
I'll second what Tom said about the importance of the script stopping on error. We don't want the user to wait on the whole install process only to find out that the script bombed in step one. Also makes it easier to find the source of the error if it is the last line. :)
When you get to the point of having something ready to test, I can volunteer a fleet of disposable RH7, RH9, and Debian vservers ready to be messed up and deleted. Takes me about 10 minutes (two command lines and some waiting) to spawn a vserver, and it can be totally goofed up by shell scripts run amok without any problems. :)
A more limited script to do automatically do the OpenACS setup tasks after AOLserver, PostgreSQL, etc. are already installed seems more useful to me. Especially if it cleanly supports dropping and re-creating test instances over and over again, that sort of thing. And Joel says above that at least a first draft of this already exists...
If you want to do both the AOLserver etc. install and the OpenACS config stuff, fine, put please make sure it's two separate scripts, and that the second (OpenACS config) script is parameterized enough so that it will work correctly even if you installed AOLserver and everything else from rpm or deb packages rather than from source using the first script.
For these scripts, especially in this community, going with Bourne shell rather than Tcl is a bad idea for maintenance. I've done a lot of Bourne shell programming, and Tcl is just plain better. Perhaps do the first (AOLserver etc. install) in Bourne shell if you really want to. But through its various dependencies I believe OpenACS effectively already requires that tclsh be installed on the box, so at least for the second (OpenACS configure) script I don't see any reason at all to use Bourne shell rather than Tcl.
1) tcl/tk would be a good combo to make a combined text or graphical installet - and since we all have to be tcl experts anyway it makes sense to use what we know
2) i like the way fink works (fink is a MacOS X package manager similar to bsd ports)
on OSX I, for instance, type "fink install postgres" - fink makes sure all the dependancies exist, downloads the source, downloads os X specific patches from it's own server, does the magic and compiles/installs the whole shebang.
3) you need to decide if you are going to use the native platform package manager or not. if so, the way to check dependancies is obvious, albeit different on every platform. you don't have to be locked in to the default packages since you could roll your own, or you could make, eg. an postgres-openfts package that relies on the standard package.
I personally think that it would be best to ignore the package manager since noone will be willing to maintain the up to date packages for every distro.
Instead I would use an autoconf style script to check for the required libraries/versions etc. as a bonus this would give you the paths to everything so you don't need to hardcode specific paths for specific OSs.
Darwinports is also another target for where openacs should be able to install from.
The reason Sloan wants to build and contribute an automated installer is to make it easier for a new user with no technical background to download and install dotLRN. The long list of "stuff" that has to be downloaded from here, there and everywhere and installed is thought to be scaring away potential adopters, who don't have much time to spend on this and feel they lack the skills to get through the manual installation.
The question about using Tcl vs Bourne shell boils down to this: Is tclsh going to be installed on all systems by default? All of the different flavors of UNIX I have access to already have it, but that's no guarantee. I don't know how to find out, without trying each one, whether all the major distros include tclsh in a default install and will continue to do so - any sugestions?
I completely agree that the script should stop on error. I haven't done much shell scripting lately but it seems like it ought to be possible to trap errors somehow; on Solaris in particular many installers still use Bourne shell, AFAIK. Is there some reason why this can't be done?
My gut feeling is that it would be better to have one script (or set of scripts) that takes the user all the way to installing OpenACS, as Tom describes above, rather than having one set to do the external package installs and another (the tclwebtest-based script) to do the setup. However, this would mean there would be some duplication between my script and the exisiting one. Can someone familiar with the existing script comment on how widely used it is, who is supporting it, and what the future plans for it are? If no-one is really championing it then it's probably not such a big deal to make it potentially obsolete.
In this scenario, we still need a fairly generic script to serve as the reference across different implementations. Note that the current install script is in fact a bash script, not a tclwebtest script. Tclwebtest is required for a single call, and for some advanced features such as doing a link test after installation.
So the most useful approach would be to pick an existing distribution platform that already has all of the dependencies, and make an OpenACS package on top of that. Out of RPM, debian, bsd ports, fink, etc, which one is closest? What scripting languages do they use? Which is most useful to the broadest range of existing and immediate potential users?
The better solution, tongue in cheek, is to start charging $10,000 for installation services. Then we'd be recognizably enterprise-class.
Otoh, you can definitely exit on errors with a shell script
set -edoes that and you can use the trap command to catch the error (although I think that is a bash-ism and you cant trap ERR with vanilla sh).
I would like to see packages for .debs, .rpms, and ebuilds and any of the half dozen other package managers people use.
I suspect the best way to get there would be to have a place in CVS where we keep the source for building with the various package tools (for example there are a set of ebuilds to install all the aolserver stuff + openfts and writing an openacs ebuild would be pretty easy). Maybe we should create an openacs-4/etc/installers/DISTRO directory and under that have the source for generating the various packages/build scripts. If we get them working well then I imagine they will get pulled into more standard distributions.
Another alternative would be to try harder to become the maintainers of aolserver and openacs on various standard distributions. The problem seems to be that the people maintaining aolserver for the various distributions where it is present are not really very good at keeping up to date. I don't know how hard it would be to get commit on gentoo and debian for example, and I don't follow fedora at all.
Anyway, there are a lot of options; I think getting things into standard distributions is important but I don't know how it will happen until some people here get commit to maintain the necessary packages.
If Sloan's need is to demonstrate .LRN to those with limited technical background, why not skip any installation and use a VMWare image or .LRN LiveCD like the one Malte is working on?
There would be hugely less amount of work to maintain it (no need to select tcl vs. bash, deb vs rpm vs etc, no need to track or resolve openacs/tcl/aolserver/tdom version conflicts), and also way less work/time to install and try it out.
Also, fwiw I was planning on including a .LRN install on my next Oasis release.
This is the OpenACS Knoppix run out of the box, should not use 512MB of RAM (but might), .LRN 2.0 enabled, Jabber running demo live CD.
We have to work a little bit more on it, but are keen to collect feedback and if you want to get your custom made CD don't hesitate to contact Björn Kiesbye (the author), or myself.
It should be that hard to use a portable TCL solution for an installer I have done with customer projects, successfully, in serveral instances.
After all this discussion I'm still not sure yet what the best way to accomplish that would be.
Unfortunately OpenACS relies on many packages which the OpenACS community cannot demand be maintained, and available in rpm, etc. format. To require this is to say no to an auto installer.
Even OpenACS doesn't have a standard which adequately addresses running multiple independent installations on the same machine.
You only install sendmail once, same with qmail, daemontools, and a bunch of other standard packages. It makes sense to have rpms for these singleton packages. However AOLserver and OpenACS and PostgreSQL are often installed in multiple locations on the same machine. Plus you have to address the possibility that a user might not have root access to their machine and might want to install OpenACS with their own privileges.
Even if rpm packages existed, you still need to maintain an installation document to explain how to do the install. Users still have to find and deal with the same issues. What is needed is a tool to combine installations of multiple independent software products.
Also, note that it isn't rocket science to install any individual package. Most of the time is spent figuring out what options to use to ./configure, how to do the post install configuration, etc. The problem is in glueing the packages togeather (plus finding the up-to-date packages is a pita). OpenACS installation itself couldn't be easier, and with 5.0 it seemed to get even easier.
It is possible something as simple as 'configure' and 'gmake' could work. Some AOLserver modules have gmake targets to create release version of the module, so it is obvious it can be used to perform any command line task, handle dependencies, etc.
first create all the rpms for the supporting software. (tcl, tdom, nsd-*, aolserver) these packages would all have the appropiate dependancies with each other.
then create the openacs rpm package (again with the appropiate dependancies) This package will not create an installed instance of openacs but create what unix sys-admins refer to as a skeleton directory as well as supplying a bunch of installation-generation scripts.
these scripts would then be used to run the useradd command (or equiv) to create the openacs user, database, directory structure.
the admin would then modify the site specific .conf file and WHamm0! they have openacs installed.
to the admin this could be made extremely easy if we made all the packages avalible from a yum or apt4rpm repo.
this would be the overall view from the admin's perspective:
apt-get install openacs
this downloads all the dependancies, and installs them
svc -u testinstall
where acsinit is the propigation script and it sets up the directory structure into /srv/openacs/<username>
any thoughts to this method?
I think it would be a mistake to work on something to just help the newbie. While the tool should support a hands off automated install, it should be customizable in an easy way for the expert. Otherwise the expert will never use it and keep it up-to-date.
What would be the benefit for the expert? A record of how a piece of software was installed so it can be repeated when necessary, both for disaster recovery and service duplication.
The installer should focus on software which traditionally has not been available in an rpm or other type of compiled format, or those which are routinely installed in a non-standard location, or when multiple installations per machine is expected. Of course, an installer which could handle this situation, would easily handle installation of compiled, standard packages.
Whatever the installer, it must have two separate parts: the actual installer, and the configuration file. New applications should only require producing a new configuration file to direct the installation. In the case of configure/gmake, you only have to create the configure script and the makefile, no programming required.
The problem with gmake/configure is, I would guess, most developers here don't spend too much time writing configure/gmake scripts. What they do know is the sequence of shell commands required, plus necessary edits of files.
Unless you're thinking of writing umbrella scripts to configure and make all the C pieces (AOLserver etc).
OpenACS itself has only one configuration choice that's of interest: which DB to use. Currently we depend on the user configuring their nsd.tcl init script properly, we could have an umbrella tclsh (or whatever) script ask for which database, user names, passwords etc and write the script for the user.
Other than that, though ... is there anything that's hard OTHER THAN installing AOLserver, PG or Oracle, ImageMagick blah blah blah?
I think we should look at defining the problem space a bit more tightly here. If our goal is to lower the hassle factor for people who want to try it out, do we really need to worry about debian people? Anyone running Debian is going to know how to install debian packages.
The person running RH or Mandrake or Fedora from an end-user CD set is much more likely to need help. Maybe we should concentrate on one platform like this first? Our resources are limited...
And for anyone with a spare machine sitting around, the Knoppix CD approach is a great one, I was only partly joking when I said that it would be cheaper to just ship folks a $200 PC than to try to support every Unix variant out there.
We do. It's part of OpenACS, in /etc/install, and instructions to get it and use it are located at http://openacs.org/doc/openacs-5-0-0/openacs.html#install-with-script and linked from the top middle part of the home page of openacs.org.
I was only partly joking when I said that it would be cheaper to just ship folks a $200 PC than to try to support every Unix variant out there.
I for one took you completely seriously, sounded like a reasonable statement. Of course, I'm sure a lot of folks would sign up to try it out if we advertised our solution!
Anyway, I already wrote a tclsh program that works for linux. The only thing missing for the total newbie is a bootstrap script to install the installer.
As you said there isn't anything difficult in the task of installing the system, but my script gathers all the software and installs from scratch in about an hour (mostly postgres compile time). Still left is the boot up of AOLserver to fill out the form for OpenACS install.
The main problem with my program (besides the fact that I wrote it) is it does what you tell it, which could lead to errors, but if the install script is correct, it works. Unlike an installation document, an install script can be tested for correctness.
Does that cover everyone?
- use XML files (probably) to specify the versions of all the pieces to be installed for, say, a functioning 5.0 install and where to get them from
- allow the user to install everything we recommend, or cherry pick the things s/he wants to install
- use variables for all OS commands, even if we start out only supporting Redhat, to make it easy for someone to get it working on their OS of choice
Among other things not thought of yet. :) Your script probably does not do all of these things currently, but is it written in such a way that it wouldn't be tragic to try to add them?
I still don't have a good feeling for how well supported the tclwebtest script is. Should we rely on it or not? I see that it is a shell script (at least that is implied by the name install.sh), which has been soundly trounced here as a Bad Idea....
My next attempt to improve this will probably be to create a localized copy of all the needed files, and write an updated installation script before installation begins. I also want to use sudo to execute every command as a particular user.
I think they're planning on doing the integration during the Berlin bug bash.
1) takes about 5 critical parameters (name, ip, port, directory) and many optional ones
2) set up full working OpenACS instance or .LRN instance, supervised by daemontools, from CVS checkout of specified tag
3) drop and rebuild an existing OpenACS instance
4) run from command line
5) report success/failure in parseable xml file
It has been working well for us for several months and is no longer experimental. It is no more finicky about paths and versions than the standard OpenACS installation, which is to say that it is pretty finicky.
svc -u testinstall
and pre-determining the location of the openacs directory you would rather allow people to specify it like this:
The reason I suggest this is that the documentation used to suggest /web/ and now suggests /var/lib/aolserver/ and as you have said you use a different directory, I would guess that it is common for people to use whatever directory they see fit. I would also sugest that the command start with the name of the apt-get able package (i.e. openacs). Lastly, it would be nice if it defaulted to installing openacs, but if the script also allowed for other "prepackaged" systems to be installed such as dotlrn. Maybe a command line option for openacs-init.
Another idea would be when you run openacs-init or whatever its called it starts an interactive menu that asks with logical defaults then you don't even need to edit config.tcl. I.E.
what is the servername [default to hostname]?
what is the database name [default to directory name]?
what ip address will this server use [localhost]?
what port will we use for this server ?
should we run an ssl connection [Y]?
- if YES then: what ssl port should we use ?
what build would you like (openacs,dotlrn,dotwrk,custom) [openacs]?
- if custom ask for the install.xml file location
would you like to start openacs now [N]?
I really like the idea of the interactive init script.
the reason I suggested /srv/openacs/ was just to be LSB v1.3 compliant as /var/lib/aolserver is LSB v1.2
but yeah the idea of being able to specify the directory will make those who like to super-customize happy as well.
openACS gurus could just drop in the needed switches like this:
# openacs-init -d /some/random/dir -u username -g groupname -h localhost:8000 ...
that command would just do the install, populate the needed directories, create the database user, shell user, database etc.
and the new users/noobies could just run:
And they would be walked through all the questions interactively with all the defaults shown. Then everything would be done.
If anyone needs write access, write the OCT.
Moreover, since my question about the automated installer was never answered, I don't know if it would fit in in the future (http://openacs.org/forums/message-view?message_id=252336).
SUSE enterprise server
Red Hat Enterprise
I plan on creating all rpms for all of the above distros as well as testing the install script/installer I may include mandrake if I see a call to support that distro
I can add a vm for the following distros, however I'm not familiar with creating packages.
if anyone is interested in developing packages for these distros, please contact me mailto:email@example.com and I'll spool a vm and give you access
read up here: http://openacs.org/forums/message-view?message_id=278458