Forum .LRN Q&A: .LRN can now be installed automatically with the help of tclwebtest

I have written up a set of scripts that can install a .LRN server from scratch non-interactively (without human interaction).

My vision for doing this was that we would have a canonical .LRN test server that gets recreated every morning with the latest source code and gets some standard setup with a few classes. If something breaks responsible people would be notified by email. This server would thus provide valuable regression tests and alert us early when things are broken. The server would also make it possible for developers to compare their own .LRN servers against the latest state of .LRN development. If you do download the TCLWebTest tool I recommend doing a cvs update -PAd since a lot seems to have been changed since the 0.1d version that is there.

My sincere gratitude and admiration goes out to Tilmann Singer for creating the excellent TCLWebTest tool. It's extremely easy to use, it's documented (not much software can claim that) and the code is well designed. I think the OpenACS community should decide on a testing framework and TCLWebTest is a strong candidate for being in such a framework. I encourage other people to try this tool and write some .LRN tests.

I will post here again once I have the test server up and I feel that I can publish my installation scripts.

I would love to have a little subgroup arise to work on testing.

We already have one in the sense that Simon Millward and OpenMSG have organized testing and have provided some tools of their own to enable folks to track testing, and to write automated tests for procs and the like.

There's been poast discussion of integrating TclWebTest into this.

What we need are a small group of people who can sit down and make it happen.  OpenMSG is perpetually busy with client work (they'll help organize testing for 4.6 but don't have additional cycles to spare beyond that at the moment).  Tilmann's busy with the Greenpeace Germany project.  I imagine Collaboraid's busy.

So ... there's a resource problem here.

Don, this is a slightly different situation... the boundaries on dotLRN testing are a little more defined.

I think it is needed and there are a lot of people that would test it.

I would like to see:

1. Two or three demo courses that show off the system (Sloan might have a course or two with content that someone would be willing to share with the world... we would also be willing to put some resources into a nice demo course)

2. Different test users (student, staff, professor, admin)

3. A big fat link to the bug tracker on the start page

I should point out that we're not doing ANY testing on dotLRN...

I'm assuming that Sloan will have padi to have their product tested, and therefore there is no value in the OACS community repeating that effort

peter: very cool!

carl: I would put the bug tracker link on _every_ page :) maybe in the footer

simon: what's "padi"?

I think Simon meant "paid" ;-)

the energy you have put into OpenACS testing is more than great (I wish I had more time to help), but I think it is important to state that dotLRN will not survive if we look at it as "their product"... the most valuable part of dotLRN is the OpenACS community and the dotLRN subset that is slowly building.

As far as I know the energy that Peter has put into the demo server is not going to result in money flowing from Cambridge, MA to København... but it might result in money from other sources flowing into København and into the OpenACS community at large.

Simon - no I know OpenMSG doesn't have time to take on dotLRN testing, don't worry.  But we do need common tools and I was just speaking in the broad sense.

Of course a lot of the base packages dotLRN is built on will be tested in the OpenACS sphere as part of our 4.6 release effort but the dotLRN package stuff itself needs to get tested by other means.  Fortunately Sloan's exercising the hell out of the Oracle version as you know.

Now ... that doesn't mean that OpenACS community members aren't going to offer to help test and fix the PG version (of course they will, we've seen a lot of this already).  It does mean that you're off the hook for organizing it and that it's not part of our OpenACS 4.6 release effort.

We (Infiniteinfo) have been actively testing (via our QA department) the postgres version for a few weeks now.  Though it's more because it's a paying job so we have our hands full here.  Our base install, with customizations, are also being tested by the client before they launch officially.  So we also get bug reports from them.  I'm making sure that patches/fixes to the bugs we found on the stock install are being submitted to appropriate persons.  That's why there's a slew of fixes and bug reports both on the dotLRN forum and the openforce dotLRN bug tracker from our end.  We just can't deliver a consolidated bug list of all the bugs we encountered because we fix them right away on our end, that's why I tend to report them in bits and pieces.

Apologies. I didn't mean to suggest that we didn't *want* to, or wouldn't *like* to test dotLRN, I was merely highlighting the fact that we simple don;t have the resources or volunteers required. I was also hihglighting the fact that dotLRN is far more specific as a platform, and therefore any test effort we do apply, will be more valuable if applied to the core.

As for organising I'd have no objection to including dotLRN in my remit, but..... how can I justify that when we as a community are not yet able to fully address our own core offering?

Now.... if I was overwhelemed with volunteers who would do testing (and do a fairly complete job ;), then this would not be the case.

Also, although I don't wish to give off the 'its their product' impression I do have to offer some gentle chastisment.

dotLRN is a proper, commercial and well backed effort with a paying customer. I don't think it is at all unreasonable to expect that OF, as the contracted firm, would perform and produce a reasonably comprehensive test plan/effort.

That is not to suggest that they are *obliged* to, but what concerns me is that they are going to be feeding a great deal of code into the OpenACS (hurray for that), but without any visible or transferable test effort. If this is the case then they are essentially increasing our QA burden....

If one was extremely synical one could argue that this is almost leaving the crappy jobs to the community! I don;t think thats what open source is about. Peer review provides many benefits, but it should not be substituted for a proper test effort.

Lets assume we were able able to perform a level of testing commensurate with the level of development... this would mean that the dotLRN stuff would add months and months of effort to a new release!

Basically the point is simple...

People are not taking testing a quality seriously enough!

Everyone vocalises its importance, but sadly there isn't the same actual test effort to back it up.

Lets get this sorted.... its good for everyone :)

Peter - great that tclwebtest is useful for you!

Something like your scripts would be extremely nice to have for the OpenACS toolkit as well, unfortunately we never got around on making that happen. Maybe after you post your scripts ...

I'd be particular interested in if/how you manage dependencies, e.g. is there an install-package.test, mount-package.test, do-stuff.test, do-some-more-stuff.test, delete-package-instance.test, uninstall-package.test kind of chaining, where it's possible to say that some tests can only happen after other tests (particularly with site-wide stuff, such as creating test users), or is it just a huge ordered list of tests that get executed one after each other?

Having dependencies would make it more reliably possible to take out sequences, e.g. for a single package, and run them without having to go through the whole install process over and over, I guess (maybe it's much easier in practice, hopefully you'll be able to tell us ;-)

My apologies for not putting up a newer release. I hope in the near future I'll have a bit more time and look into it again.

FYI, there is also this OpenACS package in new-file-storage, that acts as a web frontend for tclwebtest:

Concerning automated installation it would of course be nice to have the test script check out the most recent version from CVS automatically. In case you haven't written it already, here is a little test script that does that:
set DIR "/tmp/"
cd $DIR

# Checkout a fresh copy from CVS. Note extra colon behind anonymous
# for empty password
exec cvs checkout tclwebtest >> $DIR/tclwebtest.cvs.out 2>> $DIR/tclwebtest.cvs.err

# run selftests using the newly downloaded version
catch {
    exec sh $DIR/tclwebtest/tclwebtest $DIR/tclwebtest/selftest/ >> $DIR/ 2>> $DIR/

puts "----- output of checked out version: -------"
set f [open $DIR/]
puts [read $f]
close $f
set f [open $DIR/]
puts [read $f]
close $f
This checks out a copy of itself and starts an instance of the checked out version that runs the selftests. Oops, 1/17 failing, damn!

Instead of the second exec call one could for example set it up to call recurse $DIR/modulename/tests/ to run the suite of tests that was included in the checkout.

I have cleaned up the install scripts and made them available via CVS:

cvs -d checkout dotlrn-install

Currently .LRN is installed from scratch and a couple of users are added. What I would like to do next is set up terms, departments and classes etc.

Tilman, I am not tracking dependencies at the moment, that is something we should think about though. Right now my test scripts all build upon eachother so if one fails I want to abort all testing.

The idea about checking out an up-to-date version of TclWebTest is good, I didn't find the time to add that though.

I had two issues with TclWebTest when adding users. First of all TclWebTest doesn't seem to support file uploading. The other problem I had was that TclWebTest seems to select options in an HTML select based on labels rather than the value attribute. Tilman - would this be hard to change?



I did not mean to check out a new version of tclwebtest on each test run, I was just trying to make the automatic CVS check out work at all. I meant to get the latest version of the software being tested, e.g. in your case dotlrn. Checking out the test software itself does not really make sense, except being a funny example, and certainly added some confusion.

File upload is important and it's on the TODO list, but I don't have time right now to add it (I will accept patches of course ;-).

Regarding the select widgets - instead of doing a

field select $labeltext

you can directly set the value of a select field with

field fill $new_value

This will also accept values that are not in the option list.

I have updated the script so that it now sets up some users and classes. Also, I now have a server running at that will be recreated every workday morning and if there are errors I will be alerted. If you need access to the server, let me know.
I added two switches (--postgresql and --oacs-only) to your script so it works on PG and so you dont have to install .LRN if you don't need it. I sent you a tarball.

Thanks for writing this script, Peter - mighty useful;-)

Thanks Ola! I have tried to incorporate your changes and I have tested that Oracle installation with the --oacs-only switch works. Ola and others - please help me test the --postgresql switch and keep the feedback coming!