Premise: OpenACS Needs Automated Testing
As you probably all know, a new Acceptance Test Cycle is coming
up for release 4.6.
This has prompted a discussion on how best to do testing, what the
current issues are, and what kind of long term scheme will help us do
this in future and improve the overall quality of the toolkit, whilst
retaining manageability.
I'm proposing that the underlying solution to all our issues in this
area is to begin using automated testing in all new
development.
I'm pretty keen to get feedback from the community on this, but past
postings and discussions have made it obvious that very few people in
the community properly understand the concept, know how to implement
it or have used it before.
Most seem to think that its about writing scripts and testing web
pages in an automated fashion. Its isn't! (Or at least its not just that)
Therefore, I'll set out what I mean by it, and then put forward what I
think should happen next.
Automated Testing Description
Automated Testing is the only way in which a developer can
demonstrate his code functions as intended. Testing cannot prove the
absence of all bugs, but it can demonstrate the correct
behaviour of code.
Code without automated tests is meaningless!. This is an
absolute truism. If you can't prove what you claim abut your code,
then no-one can have any faith it will work. The only way to
demonstrate that it does, is to provide automated tests that can be
run locally.
I know that that comment is going to set the cat amongst the pigeons,
but whether you like it or not it is the case. Developers often say
things like 'I've tested it and its really stable'. But what does that
mean? How can I verify that? How does the developer know that? Guesswork?
So you can see the issue.
Ok, so the solution needs to be a method whereby we can do the following:
- Create automated tests that are unbiased
- Verify the functioning of code in an automated way
- Test all elements individually and collectively
- Prevent code from introducing errors
- Build-in demonstrable quality
- Save time during Acceptance/Delivery testing. For example its
estimated that a bug fixed at development time is nearly 4 times less
expensive in terms of effort, than doing it after acceptance testing.
The Steps in Automated Testing
The following steps describe how to create/perform automated testing.
Its important to note that its not a pick-and-choose approach. This
works only if *all* of the following are done:
- An automated test is created for every distinct
function/operation of significance.
- Tests are created 'bottom-up' i,e you write a test for a function
before that function is used in a higher function.
- Tests are always written before the code. Developers must
create tests before they write their function. The test therefore
defines the operational requirements of the function.
- Test should be written for every significant
operation/possibility the function has, this includes failure cases
and wierd stuff.
- Tests must be capable of running in isolation. i.e. a test
must not depend on the existence of another. Essentially this means
any test should create the 'system-state' required, perform the test,
record the result and then put the system back into its original
state. This means no test can affect/corrupt the result of another.
- Test are evolutionary. Whenever you think of a new thing you
could test for, an automated test should be added.
- A function/code segment, cannot, be accepted and
cannot be used or developed further until it passes all its
tests. In this way we avoid bug-overrun, where a bug in one function
causes side effects in another.
- Changes to code, must pass all existing tests as well as
new ones. If the changes imply a change is needed in the test, this
must be done before the code change.
- Tests must be deliverable as part of the product, to allow end
users to examine them and to run them themselves. In this way we also
provide regression tests as a by product.
This is the process I am strongly recommending become part of OPenACS
governance and future development. but..... I suspect you're about to
ask........
Applying A-Testing to Existing Code
Before everyone panics I am not suggesting we retro-fit or
create automated tests. The method for doing this is...
- We accept a cut-off point from which all new development must
conform to A-Testing.
- We only create a-tests when we want to make changes to
existing code, creating a test for each new function we create.
- When a bug is discovered in existing code, then we create a-tests
for it, and fix the bug.
In this way we only create the retro-a-tests, as they are required.
There is little point in creating them if we don;t think things are
failing.
The Automated Test Package
I would never recommend anything I wouldn't do myself. As a
company we use this method. It works.. We have less bugs, take
less time to develop and create better code. When we've strayed and
not done it... we have suffered. Once you start you will NEVER want to
go back to non-a-testing!
Why does this work?
There are many reasons, but primarily....
- Its works 'with' human nature not against.
- It turns testing into a development exercise, thus appealing to
coders and hackers.
- It saves huge amounts of time.
- It makes it easier to Acceptance Test releases.
- It encourages developers to write good code. and Immediately
demonstrates when they haven't.
- It builds self and product confidence.
- It makes it much easier for disjoint communities such as this to
collaborate AND improve quality.
- Its a LOT more fun than traditional approaches.
The Automated Test Package has been part of OpenACS for a while now
(how many of you have used it I wonder?).
It provides a neat tool to manage, organise, run and view the results
of automated tests. It also helps with organisation and delivery. It
even makes a great 'customer confidence builder'.
Therefore we have the tools, the code, and the method (and the proof
it works) so my recommendation is that we adopt this, ASAP.
Comments welcome :o)
(but please don't bother telling me that 'testing can't be done in
OpenACS community because.. blah blah blah..)
It can be done, we just need to find the right method. So lets
figure it out.
Cheers
PS: one of the things I've found is that most people immediately
equate this with more work!.. It isn't. You've just been fooling
yourself in the past! ;).... when you adopt this method, the effort
savings on the really boring stuff... like Acceptance Testing, Release
Management, Bug Fixing, live updates, migration scripts etc... are
massive. And I for one would rather be coding, than messinmg about....
and here you have it. The coders solution to QA!
