Forum OpenACS Development: tclwebtest with openacs-4/etc/install tests -- help getting started

Some time real soon, and maybe that's now, someone needs to do the work to integrate tclwebtest into acs-automated-testing.  Any work we do in the meantime is kinda wasted effort.  I know all of the likely suspects for this work are booked solid at the moment, and 5.0.0 should take precedence, but maybe we can tackle this in one of the rotating code bashes?  (Next one 13-14 Dec in Copenhagen:))

Hi Kolja,

Thanks for sharing your setup + use of the new testing stuff. My setup and use right now pretty closely resembles yours. My only new finding is that I can install tclwebtest to /usr/local/src and just write a short shell script in my packages/my-package/tests dir that runs all my tests by first cd'ing to /usr/local/src/tclwebtest. I like this better since I don't have to muddy by OACS source tree with tclwebtest source.

####  Set these to your settings.

set tclwebtestDir = "/usr/local/tclwebtest"

####  Stop here.

set testsDir = `pwd`
echo "Running tests from $testsDir"

# Get in tclwebtest dir to run tests.
cd $tclwebtestDir

tclwebtest -config_file ${testsDir}/nsd-config.tcl ${testsDir}/inbox-procs.test

Right now I only have the one test file but I can obviously loop here later and run all tests for this package from this shell script.

Then I added the full path to the new OACS /etc/install test suite to my local *.test scripts so I can re-use all the OACS testing procs.

    # Set full path to OACS tclwebtest harness scripts.
    set oacs_script_dir {/usr/local/src/oacs/OpenACS/openacs-4/etc/install/tcl}

    # Source OACS tclwebtest harness scripts.
    # We use their ::twt::* procs.
    source ${oacs_script_dir}/test-procs.tcl

So it is working nicely for me right now. I know as you say Surely this will have to move once there is a real testing framework in place for the whole of oacs.. So I am trying to keep good comments with this stuff in my CVS repository!

One thing that I noticed is that you said I did not put my files into automated-testing and that may be something I am doing wrong. Do the unit tests (for acs-automated-testing) all go in packages/acs-automated-testing/tcl/test or do they go in packages/my-package/tcl/test on a per package basis? This might explain why I have to watch my unit test tcl files in the APM to get them loaded in the acs-automated-testing package UI ! Glad you mentioned that.

Also I am thinking of moving tclwebtest tests to a new directory under server_root/etc/integration-tests -- at the level of the tclwebtest install suite -- instead of in per package subdirectories.

The new install suite (/etc/install) complements install.xml.  Install.tcl takes a parameter which is the install.xml file to copy in during install.  Loading of test data and running regression and integration tests is experimentally supporting by the tclwebtest scripts in /etc/install, and it's great that people are finding and using them.  We really want to get them all into acs-automated-testing, so that all of our testing is in one framework.  That should provide the separation between install code and testing code, because it should all move to /packages/whatever/tcl/test (or, better, to /packages/whatever/test, but that's probably another TIP).  We do need to put in some categorization so that tests which create test data that you don't want torn down are isolated.  I can see "automatic regression testing" as a category - these tests have no impact on any existing data and leave no new data behind, and "populate with test data," which would get run a lot for stress testing.  What other categories will we need?

A few unordered answers:

Sorry about the silly bug regarding how to start tclwebtest. As far as I know it is currently only possible to start it by doing

$ cd tclwebtest_install_dir
$ ./tclwebtest /path/to/dir_that_contains_the_tests/

which is the result of some strange way its library loading is set up
- I hope I can rearrange that one day. But that does of course not
mean you need to make a copy of tclwebtest under the web instance you want to test, you can just download it to /usr/local/, your homedir or
whatever and cd there to run it.

The -config_file switch of tclwebtest has nothing at all to do with the aolserver config file. Instead it allows to specify a .tcl file that will be sourced at the beginning of the running of each test so that you can specify common variables such as the url of your server, passwords and whatever in there. The file myservice/etc/install/install.tcl in an openacs checkout appears to be such a file (could be called test-config.tcl or whatever for more clarity).

Regarding the location of the .test files there has been no agreement yet. It would be propably the easiest if you create a myservice/test/ directory and lump them all together in there for now, before you settle on a package specific location that gets changed later on by a general decision for openacs. Also this way you don't have to worry where to put tests that are not really specific to a single package, such as logging in etc. And this way you can easily run them all in one row by just specifiying the directory as parameter to tclwebtest.

Also please note that the automatic installation of OpenACS with tclwebtest is not intended to become the recommended way of doing an installation. If you want a specific setup (package mounts, parameter settings etc.) upon installation for your service, use the install.xml feature. The tclwebtest stuff is just there to enable automated testing of the manual installation process.

Thanks, Til, for the unordered answers:)

I think the theory behind unit tests is that each of them tests a single encapsulated functionality and thus should be able to run independently from each other and not require a specific order to run. To set up prerequisites that a unit test depends on there is some functionality in acs-automated-testing as far as I remember.

Client emulation testing naturally requires a much longer series of actions to be performed in a specific order. When you run tclwebtest from the command line it runs the tests of a directory in sorted order, so you could call your tests sth like 00-register.test, 01-login.test etc.

How to deal with that when integrating tclwebtest in acs-automated-testing I don't know - maybe add a sorting facility, or an optional dependency information for each test, or instruct test writers to create long, self-contained tclwebtest tests that don't have any outside dependencies (I wouldn't like that).

Hi, I think what you said about unit testing is great as a concept and would like to work towards it:

I think the theory behind unit tests is that each of them tests a single encapsulated functionality and thus should be able to run independently from each other and not require a specific order to run. To set up prerequisites that a unit test depends on there is some functionality in acs-automated-testing as far as I remember.

But I have a few issues implementing such a test suite!

I want my unit tests to be independent and just based on the object (namespace-mimicked) that they are written for but... am running into some questions of dependencies and valid/useful assumptions to make.

Let me try to explain. Say I write unit tests for all my "objects" in my package. I would still want to be able to be assured that the unit tests for acs-kernel were run prior to my package unit tests being run. Though they are somewhat encapsulated functionality of my package objects, they do rely on at least the kernel being functioning.

Then the next lower level. Within a package, well it's easiest to do with an example so let me pretend I'm writing unit tests for the CR. I would want to have the unit test for the cr_folder complete before I ran my unit test for the cr_item and then the cr_revision. I can't even create a cr_revision without first having a cr_item, which itself requires a cr_folder to exist. And I'd personally, want to see the "right" test fail first: e.g. I would want cr_folder unit test to fail first if I can't create a folder, not the cr_item test -- it'd just be more obvious and hopefully easier to debug.

Also, my unit tests would still want to load data that assumed other packages were loaded and tested first. Like users have to be created to do just about anything worthwhile since acs_objects have creation_users. And I'd want the test users to be there for all my package's unit tests and then get deleted.

So I would want to be able to:

1. Specify package unit test suite execution order.
2. Specify data load/teardown proc per package test suite. (data is available until all unit tests in package suite have completed)
3. Specify object test order within my package test suite.
4. Specify data load/teardown/config proc per combinations of object tests within my package test suite. (think this is what aatest init-classes are for -- though I don't think this works if the unit tests are not in the same tcl file.)

I am not familiar with any commercial test frameworks and I'm just getting starting with the concept of unit testing... so anyone wants to suggest resources or recommended reading (RTFM), fine. Tips or examples how someone has addressed similar issues in another test environment would be swell!

I did just order a test driven development book so there is hope if I'm totally off base here;)