Forum OpenACS Development: Ideas for easier installation of OpenACS

One thing I noticed was that config sections look a lot like directories. It may be possible to apply a few conventions so that conversion from the current config to the new structure (and back, obviously) would be automatic. (I'm not sure exactly how much this applies, but something like the /proc filesystem seems applicable)

For instance if map = {
  HOME /usr/local/aol/config 
  SECTION ns
  SERV1HOME /web/oacs/config
  SERV1SECTION ns/server/serv1
}

Then HOME + /parameters => /usr/local/aol/config/parameters
and SECTION + /parameters => ns/parameters

SERV1HOME + /modules => /web/oacs/config/modules
and SERV1SECTION + /modules => ns/server/serv1/modules   

The current situation is that to add a server, you first add a line to a central file:

ns_section ns/servers
 ns_param server1 "Original Server"
 ns_param myserver "New Server"

# then a config section
ns_section ns/server/myserver
 ns_pram ....

The same thing is repeated for each thing you add: ns_sock modules, database pools, etc.

With everything in one file, any typo (this is a tcl script), or even certain other errors can cause the entire startup to abort and loop over and over. So adding anything to a working file is error-prone, with the potential to bring down everything using the file.

Several years ago I started experimenting with a new idea, which was to have each virtual server (server1, myserver) keep their configuration in their own directory, next to pageroot, and the modules directory:

servers/
        server1/
                pages/
                modules/
                config/
                       server.tcl
                       nslog.tcl
                       nscgi.tcl
                       nscp.tcl
                       threadpool-default.tcl
                       ...
                       

When I add a new server, I simply copy this directory and make a few changes where needed, maybe comment out a few modules I don't need. But then I'm still stuck editing main server config file so that it knows about the new server.

This is okay, because I have scripts which source the above files, and if there is an error in sourcing, the server is not added to the system, the others come up normally. The scripts also log what didn't work so the error is easier to track down.

But I also did work on the main level scripts so that it was easier to determine what the configuration was without hunting through a bunch of files. This works but is awkward and still requires editing the main file a little, but somewhat more safely.

The next 'advance' was to include a few helper procs, so that my top-level file starts like this:

# Set Main Server Variables:
namespace eval ::config {

    variable Home      /web/nsd45
    variable ConfigDir [file join $Home config]

    source [file join $ConfigDir lib config-procs.tcl]

}

# Override default values in config:
namespace eval ::config {
    variable Debug 1
    variable StartupScript [file join $Home [info script]]
    variable ErrorFile "/tmp/errors-from-[string map {/ _} $StartupScript].txt"
}

# Virtual Server / Socket Config:
namespace eval ::config {

    addServer jnm "Junom Server"
    addServer tutos "Tutos tWSDL"
    addServer server1 "Simple Server"
    addServer openacs "OpenACS" "/web/openacs-5.2.3/config/server.tcl"
    addServer oacs "OpenACS 5.3.2" "/web/openacs-5-3-2/config/server.tcl"
    addSocket nssock1 192.168.1.102 8001
    addSocket nssock2 192.168.1.102 8005
    addSocket nssock3 192.168.1.102 8000
    addSocket nssock4 192.168.1.102 8002
    addSocket nssock5 192.168.1.102 8003
    mapHostsToSocket jnm nssock1 {maria junom}
    mapHostsToSocket tutos nssock2 tutos
    mapHostsToSocket server1 nssock3
    mapHostsToSocket openacs nssock4
    mapHostsToSocket oacs nssock5
    setDefaultServerForSock jnm nssock1
    setDefaultServerForSock tutos nssock2
    setDefaultServerForSock server1 nssock3
    setDefaultServerForSock openacs nssock4
    setDefaultServerForSock oacs nssock5
}

There are several catch-22s in server startup that I avoid. One is the defaultServer for a sock. If the default server for a sock doesn't come up, the entire process aborts! For some reason it is okay to have a sock that isn't used (AOLserver removes it automatically), but not having a default server for a sock is fatal.

The above script still requires creating files in correct places, but it is easier to put in variations. Notice that I have regular old AOLserver virtual servers, and two OpenACS installations. OpenACS uses a different path structure, so the default of where server.tcl exists, is overridden by the direct path. It is very hard to use a single file (with typical variables set at the top) in this situation, and most users would be left struggling to make a copy of what they need with a few small changes, put that in a new file and try again. Then all the upstream scripts have to change, developers no longer know where to even find the configuration file for a given server. Of course once they find it, it is just like reading a tcl script and trying to imagine the output. When your server is down, this is not fun. If everyone's server is down, you have a real problem.

Anyway, even the above setup is confusing to me and I wrote the code. Fortunately, the script does print out what it is trying to do and what fails to work, so I can always look at the server.log to track down what isn't working, that is, I can track down what I did wrong. Also, the real issue is that this process cannot be automated. You cannot automate adding a line to the middle of a generic file. You can patch source code, but you can't patch configuration files. This is a big gap in AOLserver, where installation and configuration are of secondary importance to code development, or maybe just not on the radar. It is typical to write C code which checks for a config section, and if it is missing, uses a default. There is no indication to the module user that these configurations settings even exist.

For example, recently I added a Ns_Log statement to the config lookup code. Now, every call to look for something stored in the server config structure is logged (even Tcl modules). Unfortunately I don't get information on who is calling the config API, but I do get a time ordered list of what is looked for. It is amazing how many undocumented config params exist in AOLserver, and a lot of the ones being passed around in current config files are no longer used.

File based signaling of configuration could accomplish a simplification and rationalization of the above ad-hoc process and it would provide a development framework for integration of code, installation and configuration (for instance, nobody would be happy if there wasn't a 'make install' target for modules, and nobody would be happy if you didn't have revision control over your source code. If module configuration becomes part of the source code, you get this benefit with very little extra work, and you can automate it more easily.

The details are not completely obvious, but the main principle should be simplicity in design and operation. The end result of sourcing a config is a static set of data. All of this is added _before_ AOLserver starts to act on the information. All the information is added, and errors potentially occur, before AOLserver has a log file to log errors! So config processing should be verbose and explicit, but the rules for processing should be generic enough, and few enough that a simple script could produce a static config that won't crash the startup and would report any config errors and the result: db driver not found, server config skipped because 'down' exists, ip address is invalid for sock. Probably the script, if run from nstclsh would produce logging information, diagnostics, and output a static script containing only ns_section and ns_param calls. Then you could test configurations independent of the server startup, or if you had a config problem, you could run the script on the problem area so you could ask for help in diagnosis.

If everything is in one file this process would become more error prone, especially for the new user.

What is intended is an installation process which is used by novice and expert because it is easier and faster for both.

I think I need to write a little bit of code to explore and explain the idea further. Of course none of this is intended to be a requirement, but more as an additional way to install modules so that configuration is included in the process.

First, this has nothing to do with commercial mass virtual hosting, as in an Apache style web farm.

As far as informing me on the subject as related to AOLserver, you might look back over my postings and software related to virtual hosting for the past eight or so years.

Since I wasn't in on your conversations with Dossy, it is hard to comment on it. Personally, I would ask your questions in a public forum so that any misconceptions can be caught. Just a few months ago I was stupidly claiming that AOLserver's static file performance wasn't that hot. Boy, was I wrong! Other than being beat on large file performance by lighttpd, AOLserver beat every other server.

One thing I agree with in your statement, and I have said it many times: AOLserver is not designed for _mass_ virtual hosting. But why? Because virtual hosts in AOLserver take up resources even if nobody is visiting the site! They are all up and running all the time. So if you do the math, just add up the memory footprint for each of your AOLserver instances if they were to run all alone, and that is pretty close to the combined total. You will save a little bit in overhead, but not enough to think about.

Another thing you would learn (about my opinion) from reading my previous posts on the virtual host/Apache envy front is that I strongly advocate against any attempt to emulate Apache. If Apache works for you, go for it. There are some on the AOLserver list which think that is where the software should go, but it only shows up when there is the periodic call for some direction for that community. Somehow there is a severe shortage of IP addresses, and folks can't figure out how to run AOLserver and Apache on the same machine. Basically it is not a software issue. AOLserver is an integrated application server for massive applications, not for serving masses of tiny toy websites.

The thread-safety thing needs to be addressed, because someone might think there is some issue. This is insane: AOLserver (actually the original commercial product NaviServer) made Tcl thread safe, at least in AOLserver. Apparently the channel code was not quite safe, but AOLserver used C based programming in this area, so it didn't matter. Now, Tcl channels are threadsafe and they are being used more in the AOLserver core.

But it is hard to tell exactly what the comment about thread safety between virtual hosts actually means. It probably means much less than what you think it means. Obviously, AOLserver can load shared libraries which are not fully thread safe, as can any C application, and what happens is a matter of chance.

Zoran has contributed much to AOLserver and Tcl, but I have no idea what the patch is since I don't know of any problem with virtual servers and multiple servers running in one AOLserver instance.

First we need some definitions on this subject. AOLserver can do several related things:

1. One instance of AOLserver can listen on multiple IP/port combinations. Each of these listeners is a separate instance of nssock.so. But I believe you can also listen on all interfaces (0.0.0.0) with one nssock. This is typical of any server. But each instance of nssock uses a separate driver thread to handle incoming connections.
2. One instance of AOLserver can load multiple servers (like an OpenACS instance). These servers are isolated, code-wise from each other. Two different instances of OpenACS will not interfere with each other, they can run side-by-side in the same process. However, due to the ability of any of these server instances to spy on other instances, or restart the process, I have never recommended combining multiple clients in one instance of AOLserver.
3. Each loaded server can be a virtual host for incoming connections. That means that the driver threads look at the Host: header and figure out which server to use. In addition, the url is looked at to select a connection threadpool. It may seem weird at first, but drivers belong to the entire process. Each driver thread divides up the work based upon Host and url into threadpools. In addition, regardless of what threadpool is used, a server interp is placed in the thread based upon which server is used.

Naviserver is a fork of AOLserver 4.0.10 which as also ported some of 4.5 code. As others have pointed out, and I have confirmed with my testing, naviserver uses less resources. I only did static file serving, so it was a limited test. But AOLserver4.5 also outperformed naviserver by a good margin, at least 20%. AOLserver4.5 has improved driver code, and a few more threads, so that may account for the extra overhead and better performance.

If performance is critical for you, remember that it is easier and cheaper to add more ram than it is to add and maintain another machine. A product which delivers better performance with more ram is eventually going to be the better product, so the 'less resources' claim for naviserver has limited value.

On a shared machine (real virtual hosting!) you can run more naviserver instances and let those instances fight over the CPU and I/O. But as soon as you have a single application which might spill over to multiple machines, application performance becomes important, not the memory footprint.

So can AOLserver do virtual servers/multiple servers/etc.:

I decided to run an apache-bench on two servers in one instance of AOLserver. One server is running OpenACS-5-3-2 and I grabbed the home page. The other server is just a simple server. I grabbed a page which passes the request off to a worker thread, same as Gustaf uses for background delivery. It returns a 28k image file.

I hammered OACS with just 10 concurrent requests for 5000 total requests and hit the other server with 10000 request
with concurrencies of 20, 20 and 40. I've posted the logs at
http://junom.com/document/aolserver/apache-bench/

There is also the top of the startup log showing multiple servers being loaded and mapped to various Host headers.

In all cases, the concurrencies are above the number of threads in the pool.

I'll do another post today to hopefully better explain the config install proposal. In short it doesn't have too much to do with virtual servers or multiple servers in one nsd, it just has to do with distributing the configuration setup to multiple files so that the configuration can be added automatically during software installation.

😊

You are correct I should of checked posting history.

My intention was only to learn, I am out of my league on this topic.

Thank you for the input, I'm learning a lot.

OpenACS and the software it relies on are not Apache. The main difference is that you can install one version of Apache and everyone on a machine can use it. It is a system wide service like tar, less or whatever. AOLserver is not a system wide service. You can't build AOLserver with the standard Tcl distribution, you need to enable threads and other things. You may not want AOLserver to rummage around the entire system looking for Tcl packages which may also not have been compiled with the correct switches.

On the other hand, just try to install a new version of Apache on an old version of Linux. Forget it. If you have an older machine which still works very well, but has an old OS, you are totally screwed if you want a new Apache.

AOLserver has very specific, but easy to meet requirements. These requirements are not found in the standard installations. Beyond that, OpenACS and AOLserver are best considered as user applications. A single user installs and maintains all the components, and if another user wants to do the same, there is no problem. If software was installed to a central location, management would eventually, maybe quickly, become impossible.

Now, that was just a description of the basic problem, what is the solution?

App-install, although far from ready for general use, had a particular philosophy: hierarchical installation.

Each component has an associated installation file. The file contained more than just a script for installation. App-install would read the file and step through the instructions. It might prompt the user for data, displaying the default and allowing an override. The override could then affect additional defaults. Once these were set, a series of shell commands could be executed. A number of common operations were represented in the file at a higher level than simple shell commands, like creating and writing file content, changing file modes, sudo for certain operations.

Some installation files were higher up in the hierarchy. These files just contained links to other installation files, and/or the associated tar.gz file. App-install would then simply loop through these handling each in turn. In some cases the default variables in one install are reused, so you don't have to keep track of these details.

So the point was to have component maintainers create these relatively small files, and then a higher level maintainer would combine the parts they like into a new bundle. If one part of the bundle changes, the maintainer only have to change a few lines of one file and create a new version.

Once you create one of these files, you can install the sofware with the same information each time.

Then the maintainer just says which file to grab for a particular cases. When there are problems with an installation, the frustrated user can just post the entire output of the installation, because all of it was recorded to a single file. There is no mystery about what the user did or didn't do, or what failed to work as expected.

Examples of what these install file look like are here:

http://rmadilo.com/files/app-install/

I cannot stress more that you will quickly run into major trouble if you try to shortcut the process of installing OpenACS. Please consider this: this project is filled with many very talented developers. Yet, even though there are continuous calls for an easier installation, they never get done. There is a reason for this, and it isn't laziness.

Consider one simple example. I decided to compile AOLserver in my home directory to test out some feature. I thought I had discovered a massive bug where the server would run for a few requests then crash with weird errors. I wasted a day tracking down the fact that the library loader was finding the wrong version of Tcl. It used the correct version for building, but found something else at runtime. This could easily happen to anyone.

But configuration is significantly more complicated for all parts of this system. Most software components we use on our servers require no configuration, or very little. Of course as soon as you want to tweak one of these you run into the exact same issues: little documentation, no examples, don't know what to ask for or search for unless you are already an expert.

This is why I gave the example of App-install. It boils down all these decisions into a single file for each installation. The file is supposed to be written by an expert and shared with others. But it is relatively realistic: real developers compile and install even more components than just the basics. The App-install concept is also a form of documentation and an aid to recovery, or more optimistically, to duplication.

I agree about the documentation. I try to write examples of using any code I write. I actually need to do this just to test my code, so it isn't a major problem to provide examples. In fact, I like to write working examples that provide something beyond dry technical usage.

For example, over the last week I wrote a tic-tac-toe game to demonstrate a few features of a tcl templating system, and another few pages to demonstrate the use of [format] in and [assign] in the same system:

http://junom.com/document/twt/view/www/

Each file in that directory can be viewed except the .tcl files. So I simply add on a ~ to the end so your browser will display it as plain text. Examples are nice, working examples are even better.

But when I hear users asking for a download of OpenACS that just runs from a static install, I wonder how much they actually value the product. I know you can go in to Office Depot and buy a bunch of canned contracts for a few dollars, but it isn't really the same as getting good advice from an attorney. Usually the more appealing this seems to a person, the more they need a real professional.

In case I wandered around this too much, I'll stay it briefly: compiling is easy compared to configuration. Configuration is a breeze compared to running or developing an application server. Better to have a few speed bumps along the way, but the process could still be a little easier.

You are still in good company with everyone here wishing that installation was easier (and faster).

The 5 minutes or 5 days thing can still occur. It took me a week to find a Linux distribution which enabled most of my laptop features. But x86_64 is new and free software takes time. Interesting thing is that the distribution I found had a semi-full-featured boot cd, so could try before installing the whole thing.

But once that is installed, it is easy to startup the Oasis/Linux.vmw and login. I'm sure that on Windows all this works much faster since they seldom give you any choices there.

Overall, pretty cool way to check out stuff, but then what?

It would be interesting to hear what users do once they decide this stuff is cool. Does this provide the motivation to figure out the typical install?

Can you really do an easy native install on Windows?

Yes, app-install is or was not even good for a single install, multiple ones made it useful. But the basic ideas follow along with Malte's comment: provide the needed files in a known location and let the installer download them. That is a big part of the frustration is just locating the right file, then discovering I need CVS access. Try to locate the cvs instructions, etc.

When it comes to installation scripts, my interest is in developing a hierarchical system for very obvious reasons: divide and conquer. Each tiny piece is relatively easy to install, but if you have to coordinate it with other components, do it for multiple systems and maintain it over time, a mega-script becomes a liability. The hierarchical division of scripts should allow the script maintainers to copy the component scripts, make minor variations as needed for new versions and system differences, and provide expert chosen defaults. And as suggested, record the process so the process can be quickly reviewed by the community instead of needing to ask a bunch of questions or make assumptions about what might have happened.

Malte: does a full checkout of the latest release...you mean the CVS version, not a tar.gz file somewhere? This is a good point, a trap I fell into a month ago. Since then I started using git-cvsimport to maintain an easier to browse repository. One very nice feature of this is the 'snapshot'. It is a tar.gz file of whatever sub-directory you want, minus the CVS directories. It is what I would call 'instant publication.'

This is more than sufficient for many production installs of OpenACS or .LRN.

So basically you can run it that way all you want. Of course once you determine OpenACS or .LRN is the right choice, you will be more inclined to spend some extra time doing a custom install if you need it.

How difficult is it to produce these virtual installations?

I refer to my experiences in creating http://media.wu-wien.ac.at/download/vdotlrn. The difficulty is well limited, as you have options at hand:

• Create your own guest os appliance? The overall time factor is negligable, last time I did it, it was an effort of 30+ minutes. Of course, it depends, as you might consider some tweakings that should help keeping the disk footprint of the vbox as small as possible. Tweakings include:
  • Special-purpose *nix distributions: Conservative approaches are bare-bone or server installations of recent distributions (Ubuntu server is 300mb+ unpacked). Once, I also used the Debian mini flavour which reduces storage needs to 100mb+). Lately, I came across JeOS, an Ubuntu flavour, that includes some kernel (and footprint) optimisations for the use as guest os. I don't have any experiences on the latter, but it might be worth a try (especially as they are optimised for the use in vmware environments).
  • Manual clean-up after the vbox has been populated: Clear for temporary and cached data, installation and source files etc. One might also go for the manual removal of unneeded distribution elements.
  • Archiving the entire vbox for delivery: Using 7z, I managed to reduce one of my Ubuntu/OACS vboxes to ~68mb (~500mb unpacked). This gives a quite impressive compression ratio ... (and makes even unoptimised vboxes suitable for delivery)
• Take a ready-made gues-os "appliance" and built on that: some examples are available from http://www.vmware.com/appliances/.

The installation of OACS/.LRN is not that an issue, time needs correpond to an ordinary installation. With the help of Malte's installation scripts (which could be tweaked a bit) it usually takes me another 30min+ on my machine to populate my vbox. After having done so, I came up with some extras which should turn the OACS/.LRN environment (and non-*nix users):

• Some pre-configured samba shares (e.g. packages directory of OACS, home dir etc.)
• A basic shell script to localise the vbox (keyboard settings, region and language, ...) in a guided manner

Overall, once the basic environment is established, tweaks are figured out, the efforts needed are really limited for the somewhat experienced OACS/.LRN geek to come up with up-to-date releases.

[...] but it shouldn't be difficult to build this with a simple script unless the VMWare folks complicate things.

to the best of my knowledge, there is no scripting environment for production/deployment of vboxes availabe (i.e. in the sense of the vmware authoring environments). this does not mean that the overall effort can not be substantially automated but not beyond what is done by e.g. Malte's script suite, I fear ... some remains handicraft.

Btw., you find a pointer to create your own images without any vmware authoring environment (as the server console, workstation etc.) at http://media.wu-wien.ac.at/download/vdotlrn.

Hope I could shed some light on the perceived complexity ...

Can these vboxes run without a console?

Yep, just get a local installation of the vmware server which will be the "ghost" run time environment for your images. The role of the admin ui (which is a tty terminal to the vbox) , as taken by the VMWARE player, is then played by the VMWare server console. You, then, can arbitrarily attach to and detach from vboxes. The vmware player is admin console and rte in one. The rte will take care of running vboxes whenever your host system fires up or the other way around. It comes with some advanced features and is definitely the way to go in your case ...

I just installed vdotlrn and like it a lot because of it clean interface, no errors so far and I like the "Profiling Information" on each page. That helps to learn how it all works. The only shortcoming is the the installation is outdated. How difficult is it to update it to the latest version?

We fully admit there are problems. One problem of course, is that most of use who regularly use OpenACS have intuitive knowledge of how to install it, and don't see these issues.

I applaud your perseverance!

You might try http://oacsrocks.org which provides a preinstalled demo that you can use.

It's an adaptation of cognovis and old oacs scripts.
Plus tutorials published on oacs website by Vinod and some others good guys from oacs community.

It's just a few confirmation steps to get your oacs box ready to browse.(apt-get Y/N questions and postfix setup)

The troubles i faced: repos are constantly down, plus aplications such as tdom, tcllib,... version updates without publishing

Also, there are settings such as ip address and homedir, service user, in case you would like to change the defaults.

Please check it out at:
http://69.67.174.135:8000/file-storage/index?folder%5fid=2572

https://openacs.org/storage/

Here it is the link
https://openacs.org/storage/?folder%5fid=492324

Thanks for the link Dave.

Another idea for easier installation of OpenACS maybe to make an ubuntu install cd/dvd with openacs installed. I've been playing with Remastersys and it's easy and fast create distributable cd's. I've done an iso dvd http://www.davidam.com/ubuntu-oacs.iso.

Just for fun!