Forum OpenACS Development: Regression Testing in OpenACS/.LRN

OpenACS/.LRN should have extensible regression testing to prevent changes to break other unnoticed parts of the system.
Nobody likes upgrading a system and getting new errors that weren't there before.

I'm using WWW::Mechanize because it works really well and it's maintained and extensively used in regression testing.
I already have a quick script to install OpenACS/.LRN from scratch only using the web interface (as a human with the browser would do).
You can find it here: http://e-lane.org/pub/openacs_assessment_regression_tests
It's not polished but it works quite well.

Some said to use TCL Webtest. But many people including me totally NOT like it 😉

The links to the perl WWW::Mechanize:
http://search.cpan.org/~petdance/WWW-Mechanize/

Web Testing with HTTP::Recorder (so you just have to browse the site to get a test script)
http://www.perl.com/pub/a/2004/06/04/recorder.html

There are other options like:
http://httpunit.sourceforge.net/ (also with http://junit.sourceforge.net/)
http://htmlunit.sourceforge.net/

HttpUnit seems to also have javascript and that would ease some things I can't totally test with WWW::Mechanize. But not having javascript it not a big problem and you also need to test pages with javascript disabled so you can also test servers checks and not only javascript client checks.

More about Web Testing in this slashdot discussion:
http://ask.slashdot.org/askslashdot/02/09/13/1724254.shtml

Some past references:
.LRN can now be installed automatically with the help of tclwebtest (I can't get the scripts. Anybody knows where to get them?)
https://openacs.org/forums/message-view?message_id=50346

Do you also feel that OpenACS/.LRN needs some real regression testing?
What do you suggest?

Having tests is only half of the solution. Having a way to distribute the tests, run the tests, and collect the results is the other half. As of OpenACS 5.1 (or possibly 5.2 - we did some of the work on HEAD many months ago and it still isn't released yet), the basics of the second half are in place. You can use any framework you like to write tests, and still run the tests from within OpenACS and distribute them as standard parts of the packages. If your test require tclwebtest, then the test will simply not run on machines without those prerequisites. Writing tclwebtest tests is documented. If your test requires other frameworks, then we need to extend OpenACS testing to handle them, but this can be done fairly simply by extending the -libraries flag of aa_register_case.

So, yes, I do feel that OpenACS needs some real regression testing. I've outlined a path to add it via tclwebtest. I am not at all opposed to other test platforms. If someone wants to add another test platform, they should produce (and I'll help):

• Documentation to install the test package (example)
• Documentation to write tests (example)
• At least one test for each of the core packages
• Extention to aa_register_case for the new package (example from aa_register_case:

  # run library specific code
      foreach library $libraries {
          if { $library == "tclwebtest" } {
  
              # kludge: until tclwebtest installs itself in the proper
              # place following the tcl way, we use this absolute path
              # hack.
              set tclwebtest_absolute_path "/usr/local/tclwebtest/lib"
              if { ![info exists ::auto_path] || [lsearch $::auto_path $tclwebtest_absolute_path] == -1 } {
                  lappend ::auto_path $tclwebtest_absolute_path
              }
              if { [catch {
                  package require tclwebtest
                  package require http
              } err] } {
                  set error_level metatest
                  append case_error "tclwebtest is not available. Not registering this test case.\n\nError message: $err\n\n"
              }
          }
      }