Forum OpenACS Development: Automated Testing Guidelines

Posted by Simon Carstensen on
It seems to me that a lot of people have been asking for some guidelines on how to write automated tests. I've done several tests by now and have found the process to be extremely easy and useful. It's a joy to work with automated testing once you get the hang of it.

I just wrote a test script for the acs-service-contract package and thought I'd might as well post a step-by-step run-through, since some people have been asking for this. Here goes.

1. Create the directory that will contain the test script(s):

$ cd /web/simon/packages/acs-service-contract/tcl
$ mkdir test
2. Create the .tcl library that holds the test procs:
$ cd test
$ emacs acs-service-contract-procs.tcl
3. Write the tests.

This is obviously the big step :)

The script should first call ad_library like any normal -procs.tcl file:

ad_library {
To create a test case you call aa_register_case test_case_name. Once you've created the test case you start writing the needed logic. Let's say you just wrote an API for adding and deleting notes in the notes packages and wanted to test that. You'd probably want to write a test that first creates a note, then verifies that it was inserted, then perhaps deletes it again, and finally verifies that it is gone.

Naturally this means you'll be adding a lot of bogus data to the database, which you're not really interested in having there. To avoid this I usually do two things. I always put all my test code inside a call to aa_run_with_teardown which basically means that all the inserts, deletes, and updates will be rolled back once the test has been executed. A very useful feature. Instead of inserting bogus data like:

set name "Simon"
I tend to generate a random script in order avoid inserting a value that's already in the database:
set name [ad_generate_random_string]
Anyway, here's how the test case looks so far:
aa_register_case acs_sc_impl_new_from_spec {

   aa_run_with_teardown \
       -rollback \
       -testcode  {
          ... logic ...
Now let's look at the actual test code. That's the code that goes inside -testcode {}. In my case I had added a new column to acs_sc_impls (pretty_name), which meant that I had to change the datamodel and the Tcl API to support this new change. To make sure I didn't screw up, I wrote a test that created a new service contract, then a new implementation of that contract, and called acs_sc::impl::get to check that the data in the new column had been added correctly and then finally verified that the pretty_name was actually what I had tried to insert. It looked something like this:
set spec {
     name "foo_contract"
     description "Blah blah"

# Create service contract
acs_sc::contract::new_from_spec -spec $spec

set spec {
     name "foo_impl"
     description "Blah blah blah"
     pretty_name "Foo Implementation"

# Create implementation
set impl_id [acs_sc::impl::new_from_spec -spec $spec]

# Get the values of the implementation we just created
acs_sc::impl::get -impl_id $impl_id -array impl

#Verify that the pretty_name column has the correct value
aa_equals "did column pretty_name get set correctly?" $impl(pretty_name) "Foo Implementation"
Now you might not know how acs-service-contract works, but that doesn't matter. I'm basically inserting data into the database, then querying for the database to check that it got inserted and then finally, using aa_equals, I compare the result with what I inserted to verify that everything is correct.

There are number of other useful procs for determening whether a test case was successful or not, namely:

aa_true "is this true?" [expr ![empty_string $foo]]
aa_false "is this true?" [empty_string $foo]
There a number of other useful procs and I will encourage you to look at the few packages for which tests have already been implemented. That is perhaps the best documentation we have so far.

Hope this helps.

- Simon

Posted by Joel Aufrecht on
simon, that's awesome.  That's the document I've been trying to get the time to learn enough to write for five months.  Can you show how you would set up a test file to check the front page of a module, like, say forum?  I.e., how would you make sure that the default install worked?  You could do an http get of the default mount location and check for a specific string, but that would break if the page was ever customized.  Could you work through the .tcl pages and ignore the .adp to create a more reliable test?
Posted by Simon Carstensen on
Can you show how you would set up a test file to check the front page of a module, like, say forum? I.e., how would you make sure that the default install worked?

I'm not sure I understand what you mean by this. Do you want to check whether the package installed correctly (i.e. that the install scripts aren't broken)? Or do you just want to make sure that some page of a package is not throwing a Request Error?

Automated testing is generally meant for testing the Tcl API, not so much the logic of individual pages. TclWebTest should be for individual pages (disclaimer: haven't used TclWebTest and don't know much about it).

One way to get around it, of course, is to put the logic into a Tcl API, which is considered best-practice anyway.

But please let me know what is you mean by "checking the front page of a module".

- Simon

Posted by Randy O'Meara on
Oh, this is wonderful stuff! Thank you, Simon.


Posted by Walter McGinnis on
Seems like a prime candidate for a FAQ to me.  Once the FAQ is in place, it would be nice if it was linked to from the documentation pages...

Thanks for the write-up Simon,


Posted by Peter Harper on
Hi Folks,

I've been away from the OpenACS community for quite a while, and have just happened upon this posting.  Just to say that from the automated testing packages humble beginnings when I initially threw it together, I'm glad to see that some people are beginning to use and develop it.  I never had the time to document it unfortunately, so its also good to see some tutorials beginning to appear.

Have many tests actually made it into any releases yet?


Posted by Peter Marklund on
Hey Pete,
good to see you back! 😊 We are using your acs-automated-testing package quite heavily here at Collaboraid and we're very grateful that you contributed it! We've used it in all service packages we've developed - workflow, acs-lang, and most recently acs-authentication. You can see the test status here:
Posted by Jade Rubick on