Forum OpenACS Development: tclwebtest with openacs-4/etc/install tests -- help getting started

Hi

I'm starting to write tclwebtest acceptance tests for my first package! I ran across this post about up and coming tclwebtests with oacs. I grabbed a beta copy from cvs and it looks like these test are mostly intended for automating installation of OACS (how cool). But there are lots of really useful procs defined there for general OACS testing with tclwebtest.

So long story, somewhat shorter, I am trying to use this testing framework to implement some basic acceptance tests for my new package. I can outline how I started doing this (below) and what I'm hoping for is some sort of guidance/feedback on my (mis)use versus the way this neat new test harness is intended to be used. Keep in mind this my first use of tclwebtest too so any tips are welcome really;)

The testing stuff is located in /etc/install under OACS root. Looks like my *.test files should be created here to take advantage of the extensive test procs that have been written for use with tclwebtest + oacs.

Do I have to copy /usr/local/src/tclwebtest-0.3/tclwebtest and /usr/local/src/tclwebtest-0.3/lib to openacs-4/etc/install to use this?! Otherwise tclwebtest can't find either its own lib (unless run from its source dir) or the openacs-4/etc/install/tcl files (uses path relative to dir tclwebtest is exec'd from) to source them:(

Next problem I encounter is, if I pass my nsd config to tclwebtest (using -config_file as I see done in other places here), well then I get errors about ns_log in my nsd.config (not being defined of course). There is an ns_log in the sample config provided with the testing stuff so I am missing something here... are these tests supposed to be run from within aolserver? It'd be much nicer for me if not!

Yet if I don't pass -config_file to tclwebtest, I will get this sort of error for various variables defined in nsd config:

can't read "serverroot": no such variable
I can make a stripped down copy of server nsd conf file containing only the variables I think are needed by the test procs (from openacs-4/etc/install/tcl/config-procs.tcl global vars;).

I ended up needing at least this to run a basic test (just to source openacs-4/etc/install/tcl/config-procs.tcl mostly):

#---------------------------------------------------------------------
# change to 80 and 443 for production use
set httpport                  8000
set httpsport                 8443

# The hostname and address should be set to actual values.
set hostname                  myserver.mydomain.com
set address                   127.0.0.1

set server                    "openacs-dev"
set servername          "New OpenACS Installation - Development"

set serverroot                "/www/nsroot/${server}"

set server_url                "http://myserver.mydomain.com:8000"

set admin_last_name "admin_last_name"
set admin_email "admin_email"
set admin_password "admin_password"
set alert_keyword "alert_keyword"

#---------------------------------------------------------------------
So anyway, I know this stuff is hot off the press, but can I get some direction from the pros in the trenches please? What is the manner in which tests are recommended to be developed for new packages? Where do things go? I'll be happy to document my experiences getting up and running... TIA:)
Some time real soon, and maybe that's now, someone needs to do the work to integrate tclwebtest into acs-automated-testing.  Any work we do in the meantime is kinda wasted effort.  I know all of the likely suspects for this work are booked solid at the moment, and 5.0.0 should take precedence, but maybe we can tackle this in one of the rotating code bashes?  (Next one 13-14 Dec in Copenhagen:))
hi, I'm trying on tclwebtest for acceptance tests as well, and like you I like to run it outside of aolserver.
I got the server parameters in this (really ugly) way:

# determine the server address
# open the config file to determine the server parameters
set file [open "../../../etc/config.tcl" r]
while {[gets $file line]!=-1} {
    if [regexp {set address} $line] {
        eval $line
        # address is set
    }
    if [regexp {set httpport} $line] {
        eval $line
        # httpport is set
    }
    if [regexp {set port} $line] {
        eval $line
        # port is set
    }
    if [regexp {set server} $line] {
        eval $line
        # server is set
    }
}
if {![info exists address] || ![info exists httpport]} {
  if {[info exists server] && [info exists address($server)] && [info exists port($server)]} {
      set server "http://[set address($server)]:[set port($server)]"
  } else {
      error "Error parsing config file!"
  }
} else {
    set server "http://${address}:${httpport}"
}

And I did not put my files into automated-testing, but in packages/<my-package>/tests/ instead so it is bundled with the package it's meant for. Surely this will have to move once there is a real testing framework in place for the whole of oacs.
As for the contents of the testsuite I just kind of "click around" through the pages, seeing that I cover all the pages, all the functionalities, some/most of the error messages on wring input etc.
The main concern I have which made me using this is breaking some code somewhere without noticing it, like you change a proc somewhere or mass update something in 20 files and in 1 page it breaks for some obscure reason. I don't want the client to find that out.

Hi Kolja,

Thanks for sharing your setup + use of the new testing stuff. My setup and use right now pretty closely resembles yours. My only new finding is that I can install tclwebtest to /usr/local/src and just write a short shell script in my packages/my-package/tests dir that runs all my tests by first cd'ing to /usr/local/src/tclwebtest. I like this better since I don't have to muddy by OACS source tree with tclwebtest source.


####  Set these to your settings.

set tclwebtestDir = "/usr/local/tclwebtest"

####  Stop here.

set testsDir = `pwd`
echo "Running tests from $testsDir"

# Get in tclwebtest dir to run tests.
cd $tclwebtestDir

tclwebtest -config_file ${testsDir}/nsd-config.tcl ${testsDir}/inbox-procs.test
Right now I only have the one test file but I can obviously loop here later and run all tests for this package from this shell script.

Then I added the full path to the new OACS /etc/install test suite to my local *.test scripts so I can re-use all the OACS testing procs.

    # Set full path to OACS tclwebtest harness scripts.
    set oacs_script_dir {/usr/local/src/oacs/OpenACS/openacs-4/etc/install/tcl}

    # Source OACS tclwebtest harness scripts.
    # We use their ::twt::* procs.
    source ${oacs_script_dir}/test-procs.tcl

So it is working nicely for me right now. I know as you say Surely this will have to move once there is a real testing framework in place for the whole of oacs.. So I am trying to keep good comments with this stuff in my CVS repository!

One thing that I noticed is that you said I did not put my files into automated-testing and that may be something I am doing wrong. Do the unit tests (for acs-automated-testing) all go in packages/acs-automated-testing/tcl/test or do they go in packages/my-package/tcl/test on a per package basis? This might explain why I have to watch my unit test tcl files in the APM to get them loaded in the acs-automated-testing package UI ! Glad you mentioned that.

Also I am thinking of moving tclwebtest tests to a new directory under server_root/etc/integration-tests -- at the level of the tclwebtest install suite -- instead of in per package subdirectories.

Collapse
Posted by tammy m on

The new test suite looks like it will perform 3 separate but related and really useful functions for OACS-based servers:

  • Automated install of OACS. Which, btw, will this complement the existing install.xml process for this or replace it?
  • Loading of test data.
  • Running regression/integration tests.
I just wanted to mention up front that all 3 of these are also useful to be performed separately at varying times in the life of an OACS installation. Like right now, I'd like to load test data and run integration tests (but already have a server running without a need to rebuild it). Later, I'll want to just be able re-run the integration tests with or without reloading my test data. So for what it's worth (and it's probably already been thought of), it'd be nice to keep some level of separation between install scripts and the useful tcl "lib" of support procs being built to test various features and load test data, etc.

e.g. Right now, since I don't use dotlrn, I have created my own users-data.csv file to test my server using users that already exist on my test server. This way I can still use the already written ::twt procs for logging in users, etc.

Collapse
Posted by Joel Aufrecht on
The new install suite (/etc/install) complements install.xml.  Install.tcl takes a parameter which is the install.xml file to copy in during install.  Loading of test data and running regression and integration tests is experimentally supporting by the tclwebtest scripts in /etc/install, and it's great that people are finding and using them.  We really want to get them all into acs-automated-testing, so that all of our testing is in one framework.  That should provide the separation between install code and testing code, because it should all move to /packages/whatever/tcl/test (or, better, to /packages/whatever/test, but that's probably another TIP).  We do need to put in some categorization so that tests which create test data that you don't want torn down are isolated.  I can see "automatic regression testing" as a category - these tests have no impact on any existing data and leave no new data behind, and "populate with test data," which would get run a lot for stress testing.  What other categories will we need?
Collapse
Posted by Joel Aufrecht on
We also need a category for tests that are added in advance of the features (or bug fixes) they test, so that we know these are "expected" failures.
Collapse
Posted by Tilmann Singer on
A few unordered answers:

Sorry about the silly bug regarding how to start tclwebtest. As far as I know it is currently only possible to start it by doing

$ cd tclwebtest_install_dir
$ ./tclwebtest /path/to/dir_that_contains_the_tests/

which is the result of some strange way its library loading is set up
- I hope I can rearrange that one day. But that does of course not
mean you need to make a copy of tclwebtest under the web instance you want to test, you can just download it to /usr/local/, your homedir or
whatever and cd there to run it.

The -config_file switch of tclwebtest has nothing at all to do with the aolserver config file. Instead it allows to specify a .tcl file that will be sourced at the beginning of the running of each test so that you can specify common variables such as the url of your server, passwords and whatever in there. The file myservice/etc/install/install.tcl in an openacs checkout appears to be such a file (could be called test-config.tcl or whatever for more clarity).

Regarding the location of the .test files there has been no agreement yet. It would be propably the easiest if you create a myservice/test/ directory and lump them all together in there for now, before you settle on a package specific location that gets changed later on by a general decision for openacs. Also this way you don't have to worry where to put tests that are not really specific to a single package, such as logging in etc. And this way you can easily run them all in one row by just specifiying the directory as parameter to tclwebtest.

Also please note that the automatic installation of OpenACS with tclwebtest is not intended to become the recommended way of doing an installation. If you want a specific setup (package mounts, parameter settings etc.) upon installation for your service, use the install.xml feature. The tclwebtest stuff is just there to enable automated testing of the manual installation process.

Collapse
Posted by Tilmann Singer on
Joel: I would love to do the acs-automated-testing integration of tclwebtest, but realistically won't find time for that before the end of this year. But I don't see that it is that urgent, it's a 5.1 thing anyway.

And yes, we need to think about the locations for the .test files, and propably more about complex dependencies of tests since black box client tests test to have more dependencies like inserting test data before running stuff etc., compared to api unit tests.

I hope I did not contradict you (Joel) when saying that tclwebtest isn't used for automated installation under normal circumstances. In my opinion it shouldn't, but I haven't looked closely at etc/install/ yet.

Collapse
Posted by tammy m on
Thanks, Til, for the unordered answers:)
Collapse
Posted by tammy m on
Joel,

I don't know if these would be categories of tests or not but they would be useful IMHO.

Some way to specify the order in which tests are run within test suites. I think that right now, the tests I wrote that use the acs-automated-testing package are executed in the order in with they are defined in my test-procs.tcl script. (I don't know what order individual *-test-procs.tcl script files get loaded though. Probably best not to depend on this since you would want the scripts to be able to execute independently.)

For instance, if I have a test case that verifies that object_type_a can be created via database function and a tcl proc, I want to be sure this test runs before any other tests that create object_type_a's and run tests on them. Obviously if the an object_type_a can't be created, it'd be easier to glean that from the test results for the object_type_a creation test (and not other tests that do other things with the object).

Anyway, just some current thoughts that came up while trying to organize my tests...

I think the theory behind unit tests is that each of them tests a single encapsulated functionality and thus should be able to run independently from each other and not require a specific order to run. To set up prerequisites that a unit test depends on there is some functionality in acs-automated-testing as far as I remember.

Client emulation testing naturally requires a much longer series of actions to be performed in a specific order. When you run tclwebtest from the command line it runs the tests of a directory in sorted order, so you could call your tests sth like 00-register.test, 01-login.test etc.

How to deal with that when integrating tclwebtest in acs-automated-testing I don't know - maybe add a sorting facility, or an optional dependency information for each test, or instruct test writers to create long, self-contained tclwebtest tests that don't have any outside dependencies (I wouldn't like that).

Hi,

Yeah there are init-classes in acs-automated-testing that I think can be used to set up prerequisites and tear them down too. I think maybe I need some more knowledge of unit testing. Thanks for listening;)

Hi, I think what you said about unit testing is great as a concept and would like to work towards it:
I think the theory behind unit tests is that each of them tests a single encapsulated functionality and thus should be able to run independently from each other and not require a specific order to run. To set up prerequisites that a unit test depends on there is some functionality in acs-automated-testing as far as I remember.
But I have a few issues implementing such a test suite!

I want my unit tests to be independent and just based on the object (namespace-mimicked) that they are written for but... am running into some questions of dependencies and valid/useful assumptions to make.

Let me try to explain. Say I write unit tests for all my "objects" in my package. I would still want to be able to be assured that the unit tests for acs-kernel were run prior to my package unit tests being run. Though they are somewhat encapsulated functionality of my package objects, they do rely on at least the kernel being functioning.

Then the next lower level. Within a package, well it's easiest to do with an example so let me pretend I'm writing unit tests for the CR. I would want to have the unit test for the cr_folder complete before I ran my unit test for the cr_item and then the cr_revision. I can't even create a cr_revision without first having a cr_item, which itself requires a cr_folder to exist. And I'd personally, want to see the "right" test fail first: e.g. I would want cr_folder unit test to fail first if I can't create a folder, not the cr_item test -- it'd just be more obvious and hopefully easier to debug.

Also, my unit tests would still want to load data that assumed other packages were loaded and tested first. Like users have to be created to do just about anything worthwhile since acs_objects have creation_users. And I'd want the test users to be there for all my package's unit tests and then get deleted.

So I would want to be able to:

  1. Specify package unit test suite execution order.
  2. Specify data load/teardown proc per package test suite. (data is available until all unit tests in package suite have completed)
  3. Specify object test order within my package test suite.
  4. Specify data load/teardown/config proc per combinations of object tests within my package test suite. (think this is what aatest init-classes are for -- though I don't think this works if the unit tests are not in the same tcl file.)

I am not familiar with any commercial test frameworks and I'm just getting starting with the concept of unit testing... so anyone wants to suggest resources or recommended reading (RTFM), fine. Tips or examples how someone has addressed similar issues in another test environment would be swell!

I did just order a test driven development book so there is hope if I'm totally off base here;)