Forum OpenACS Development: Sould we add acs-automated-testing into acs-core?

1: Sould we add acs-automated-testing into acs-core?

Posted by Peter Marklund on 10/01/02 04:08 PM

I just wrote my first unit test with OpenACS (hooray!) that tests a
new acs-lang procedure that we are adding. My first inclination was to
use the acs-automated-testing package that Open-MSG kindly has
provided us with but since that package is not part of acs-core I
cannot use it to test core procedures. Shouldn't we add this package
to acs-core (aka the OpenACS kernel)?

I would like to see procedure level tests being used more
broadly in the kernel. As a Java programmer I used JUnit a lot and the
same idea is quite applicable to the OpenACS TCL API. Some of
the procs do non-trivial things and if you edit them you want (at
least I do) to run a regression test to make sure you didn't at least
break the main functionality of the proc.

2: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by defunct defunct on 10/01/02 04:14 PM

Peter,

I couldn't agree more! The whole idea was to encourage people to provide TCL based regression tests for just the circumstance you describe, and also figuring that developers might get more out of testing (and be more inclined to do it) if it was essentially a development exercise.

Not sure why you can't use acs-automated-testing to test acs-core functions? But perhaps Peter Harper (erstwhile author) can answer that for you?

3: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by Don Baccus on 10/01/02 04:36 PM

I think he's saying he can't depend on it being there. I have no problem adding it. It doesn't break anything to have it there, right?

4: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by Peter Marklund on 10/01/02 04:49 PM

Thanks for the support Simon! I think you misunderstood me, I wasn't saying you can't use acs-automated-testing to test acs-core procs. It's just that I can't create a dependency on the acs-automated-testing package from the acs-lang package as long as acs-automated-testing is not part of core (we are moving acs-lang into core).

I discussed the testing issue briefly with Lars and his concern was to avoid cluttering or bloating the code base with tests especially on production systems. For example you probably don't want to run your tests on every server startup on production systems. On dev servers on the other hand I would argue that you *do* want to run the tests on server startup. To obtain a distinct and simple separation it might make sense to keep the TCL tree with the tests separate from the production TCL tree. For example we could keep TCL tests under a package-name/test directory. We could then have a ProductionSiteP parameter that indicates if those tcl files should be sourced on server startup or not.

I haven't looked into how acs-automated-testing deals with these issues yet. Any thoughts?

5: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by Lars Pind on 10/01/02 04:57 PM

The keys thing for me are:

1) If I don't install acs-automated-testing on a server, the tests that exist in various packages should have no effect on me. They can sit in the file system, but they should just be ignored. In other words: It should be possible to have an OACS server without acs-automated-testing installed.

2) If I do install acs-automated-testing, I think it's reasonable that that package has an option that says "should I run all tests on server startup". It belongs in acs-automated-testing, not in core, correct?

3) I myself would much rather run the automated tests every time I make a change. So it should be possible to run them without restarting the server. Also, it should be possible to run only the tests registered for a single package to save time, so I can run them more frequently.

/Lars

6: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by defunct defunct on 10/01/02 05:46 PM

ok guys, I think I can answer these, but Pete's the authority.

My understanding of the current version is that I think it does all your asking.. i.e. you can re-rund tests, on demand from the package via a simple interface (you can select sub sections or whatever).

I also thought that it was possible to essentially *ignore* tests a-la production env. Although this change may not have worked its way in yet.

So basically I think Lars' points 2 & 3 are already covered. 1 maybe but I need peter confirm that.

Calling Mr. Harper! can you pitch in when you get chance?

7: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by Don Baccus on 10/01/02 06:02 PM

#1: Certainly this must be true. I want tests in a separate directory, actually, so I can easily delete them if I want to (not that I necessarily see the need to do so). This may already be true.

#2: I think there's already a parameter to do this? And yes, it should be a parameter to the testing package, not systemwide.

To be honest I don't care much if its distributed as part of our core tarball (when we move to modularized releasees) or not. Releasing it as part of core doesn't imply it needs to be mounted and enabled by default, we could install it but not mount it and that would solve Peter's dependency issues, right???

8: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by defunct defunct on 10/01/02 06:12 PM

Sounds right to me....

9: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by Lars Pind on 10/01/02 08:35 PM

I think we need a small glossary here ... "Kernel", "part of core", and others ... When I hear "part of core" i hear "required in an OACS install, i.e., you can depend on this package without explicitly havoing a require section for it in your package". Don, apparantly, takes that to mean "distributed with the core tarball". Whether or not "part of core" means "release process for this package should be linked to releases of this other package" to various people is still unclear to me.

Glossay would be good. :)

/Lars

10: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by Don Baccus on 10/01/02 11:44 PM

Yeah, you're right, Lars. If you want to get super-technical about it, my definition is "the set of packages gotten by grabbing openacs-4-core from CVS". This is essentially the ACS 4.2 core tarball with a couple of additions made by us over time (things that need to be there to resolve dependencies, mostly or entirely or something like that).

There are already packages that aren't mounted by default in that set, though they should all be installed at startup I think (I haven't checked this in a long, long time and might be wrong).

Revisiting "what's in core and what's not" seems like a good idea for 4.7, which hopefully is when we'll have download options other than "download everything in existence in one big tarball".

11: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by Peter Harper on 10/02/02 11:43 AM

Hi Folks, My plan was to integrate acs-automated-testing with the core system for the 4.6 release. I've quite spectacularly missed this deadline, and am now aiming to get this into the 4.7 release. The plan is:

1. Store all tests in the /tcl/test directory.
2. Modify the bootstrapper to source these files if a)Acs-automated-testing is installed, and b) An acs-automated-testing package exists with a "source-tests" flag set.

The current interface allows one to browse and run tests based around package-key's, and also provide some basic forms of categorizing tests. I have no plans to provide functionality for running tests at server startup. My thinking is that although you'd hope tests should be written to run standalone (i.e, leave the system in exactly the same state as it was before the test was ran), this is a dangerous assumption to make. Perhaps I should create a category called "startup", which marks those tests that should be run...... Also we might want to consider than some tests might be written for performance testing. Wouldn't want those run during startup.

Whats the estimated release date for 4.7 again????

12: Response to Sould we add acs-automated-testing into acs-core? (response to 1)

Posted by Janine Ohmer on 10/05/02 09:16 PM

Pete, we don't have an estimated release date for 4.7 yet - the Gatekeepers need to figure out what's going into it first. It will be prominently announced when we figure it out.