Forum OpenACS Development: Re: Uses of tclwebtest with openacs-4/etc/install tests

5: Re: Uses of tclwebtest with openacs-4/etc/install tests (response to 1)

Posted by tammy m on 12/08/03 07:03 PM

The new test suite looks like it will perform 3 separate but related and really useful functions for OACS-based servers:

Automated install of OACS. Which, btw, will this complement the existing install.xml process for this or replace it?
Loading of test data.
Running regression/integration tests.

I just wanted to mention up front that all 3 of these are also useful to be performed separately at varying times in the life of an OACS installation. Like right now, I'd like to load test data and run integration tests (but already have a server running without a need to rebuild it). Later, I'll want to just be able re-run the integration tests with or without reloading my test data. So for what it's worth (and it's probably already been thought of), it'd be nice to keep some level of separation between install scripts and the useful tcl "lib" of support procs being built to test various features and load test data, etc.

e.g. Right now, since I don't use dotlrn, I have created my own users-data.csv file to test my server using users that already exist on my test server. This way I can still use the already written ::twt procs for logging in users, etc.

6: Re: Uses of tclwebtest with openacs-4/etc/install tests (response to 5)

Posted by Joel Aufrecht on 12/08/03 11:16 PM

The new install suite (/etc/install) complements install.xml. Install.tcl takes a parameter which is the install.xml file to copy in during install. Loading of test data and running regression and integration tests is experimentally supporting by the tclwebtest scripts in /etc/install, and it's great that people are finding and using them. We really want to get them all into acs-automated-testing, so that all of our testing is in one framework. That should provide the separation between install code and testing code, because it should all move to /packages/whatever/tcl/test (or, better, to /packages/whatever/test, but that's probably another TIP). We do need to put in some categorization so that tests which create test data that you don't want torn down are isolated. I can see "automatic regression testing" as a category - these tests have no impact on any existing data and leave no new data behind, and "populate with test data," which would get run a lot for stress testing. What other categories will we need?

7: Re: Uses of tclwebtest with openacs-4/etc/install tests (response to 6)

Posted by Joel Aufrecht on 12/09/03 10:26 AM

We also need a category for tests that are added in advance of the features (or bug fixes) they test, so that we know these are "expected" failures.

8: Re: Uses of tclwebtest with openacs-4/etc/install tests (response to 7)

Posted by Tilmann Singer on 12/09/03 03:27 PM

A few unordered answers:

Sorry about the silly bug regarding how to start tclwebtest. As far as I know it is currently only possible to start it by doing

$ cd tclwebtest_install_dir
$ ./tclwebtest /path/to/dir_that_contains_the_tests/

which is the result of some strange way its library loading is set up
- I hope I can rearrange that one day. But that does of course not
mean you need to make a copy of tclwebtest under the web instance you want to test, you can just download it to /usr/local/, your homedir or
whatever and cd there to run it.

The -config_file switch of tclwebtest has nothing at all to do with the aolserver config file. Instead it allows to specify a .tcl file that will be sourced at the beginning of the running of each test so that you can specify common variables such as the url of your server, passwords and whatever in there. The file myservice/etc/install/install.tcl in an openacs checkout appears to be such a file (could be called test-config.tcl or whatever for more clarity).

Regarding the location of the .test files there has been no agreement yet. It would be propably the easiest if you create a myservice/test/ directory and lump them all together in there for now, before you settle on a package specific location that gets changed later on by a general decision for openacs. Also this way you don't have to worry where to put tests that are not really specific to a single package, such as logging in etc. And this way you can easily run them all in one row by just specifiying the directory as parameter to tclwebtest.

Also please note that the automatic installation of OpenACS with tclwebtest is not intended to become the recommended way of doing an installation. If you want a specific setup (package mounts, parameter settings etc.) upon installation for your service, use the install.xml feature. The tclwebtest stuff is just there to enable automated testing of the manual installation process.

9: Re: Uses of tclwebtest with openacs-4/etc/install tests (response to 8)

Posted by Tilmann Singer on 12/09/03 03:50 PM

Joel: I would love to do the acs-automated-testing integration of tclwebtest, but realistically won't find time for that before the end of this year. But I don't see that it is that urgent, it's a 5.1 thing anyway.

And yes, we need to think about the locations for the .test files, and propably more about complex dependencies of tests since black box client tests test to have more dependencies like inserting test data before running stuff etc., compared to api unit tests.

I hope I did not contradict you (Joel) when saying that tclwebtest isn't used for automated installation under normal circumstances. In my opinion it shouldn't, but I haven't looked closely at etc/install/ yet.

10: Re: Uses of tclwebtest with openacs-4/etc/install tests (response to 8)

Posted by tammy m on 12/10/03 12:19 AM

Thanks, Til, for the unordered answers:)

11: Re: Uses of tclwebtest with openacs-4/etc/install tests (response to 6)

Posted by tammy m on 12/10/03 04:46 AM

Joel,

I don't know if these would be categories of tests or not but they would be useful IMHO.

Some way to specify the order in which tests are run within test suites. I think that right now, the tests I wrote that use the acs-automated-testing package are executed in the order in with they are defined in my test-procs.tcl script. (I don't know what order individual *-test-procs.tcl script files get loaded though. Probably best not to depend on this since you would want the scripts to be able to execute independently.)

For instance, if I have a test case that verifies that object_type_a can be created via database function and a tcl proc, I want to be sure this test runs before any other tests that create object_type_a's and run tests on them. Obviously if the an object_type_a can't be created, it'd be easier to glean that from the test results for the object_type_a creation test (and not other tests that do other things with the object).

Anyway, just some current thoughts that came up while trying to organize my tests...