Forum OpenACS Q&A: Call for testers ... OpenACS 4

Collapse
Posted by Don Baccus on
We're getting close to an alpha release of the OpenACS 4 core and a
subset of the packages being worked on.  While this first release will
primarily be developer-tested, we'll be wanting to institute more
formal testing as we roll on towards a solid beta release.

Several folks have volunteered to help in the past, but it has been so
long since my first call for volunteers that I thought it would be
best to issue another call for testers, specifically.

We have a volunteer willing to help organize the testing effort - more
on that next week - but will need bodies willing to help do thorough
testing of various packages for Oracle and Postgres.

Please e-mail me directly if you can help out.  Though I've kept the
names of those who've offered in the past, don't be bashful about
e-mailing me again.  My record-keeping's not perfect, after all!

Collapse
Posted by Yon Derek on
I think that the crucial point in testing is fixing bugs that are reported. So the question is:
  • how to report bugs/small patches to make sure they get fixed
  • what is the plan for making sure that reported bugs get fixed
I know that the standard answer is "use sdm" but so far it isn't working.

I don't want anyone to take it personally, but that's a fact: 3.2.x was a disaster in this area (tens of bugs reported, many of them with clear descriptions and even patches; not many of them fixed). OpenACS 4 is heading into the same direction: there are already a few pages of bugs reported.

My point: there's little point asking for bug reports while there are still known bugs.

My second point: there must be a dedicated "sdm watcher", someone whose priority would be going through bugs in sdm and fixing them, closing invalid bugs etc. Otherwise sdm will become useless very quickly. Clearly not many people qualify for this position: this someone would have to have full cvs access and ability to close bugs in sdm.

Collapse
Posted by Roberto Mello on
Yond, 3.2 was not a disaster in this area. For the 3.2.5 release I walked through EVERY bug report on SDM. I read all of them, and closed all the ones that were relevant.

The thing is, many (most?) of those bug reports were from the alpha/beta days of OpenACS and they were not closed, but they got fixed. I had 2 choices: spend a lot of time closing every bug OR spend that time fixing other things in OpenACS. I opted for the latter.

Now I'm not saying that I was perfect. One or more bugs may have slipped, but I did walk through all of them.

But your point is valid. We need to be attentive and close the bugs in our modules. Porters need to watch their modules (click on the "I'm interested" button) so they receive notification when new bugs/patches are posted.

This is very important because if not done, the community will lose confidence in the process, as evidenced by your post. I don't see a better way of reporting bugs. Any other reporting method will suffer the same fate if the person responsible for that module/package does not take care of his/her responsibility.

But we should not lose confidence in the process. You could instead, e-mail the maintainer of the package and try to work this out with him/her.

So, may this be a heads-up for all those helping in the port: Check your SDM reports, please!

Collapse
Posted by Don Baccus on
<blockquote><i>My second point: there must be a dedicated "sdm watcher", someone whose priority would be going through bugs in sdm and fixing them, closing
    invalid bugs etc. Otherwise sdm will become useless very quickly. Clearly not many people qualify for this position: this someone would have to have
    full cvs access and ability to close bugs in sdm. </i></blockquote>
Well, if in your opinion there *must* be an SDM watcher, back up your words by offering to volunteer your time to take on this role!  As always, I find that people are more willing to offer criticism than to step up and offer ten or so hours a week of volunteer time.
<p>
And, yes, I do appreciate the fact that you've passed along some bug fixes as well as bug reports.  Still, we're volunteers here and my long experience with volunteer crews tells me that there will be little patience with folks *demanding* that certain roles be filled unless those folks are willing to volunteer to pitch in and help.
<p>
Roberto's answered your criticism of 3.2 very well.  If you or someone
else wants to volunteer to go through the SDM and mark fixed bugs as being closed, that would be great.
<p>Any takers?  Just e-mail Roberto, I'm sure he'll welcome the logistical help.
<p>As far as OpenACS 4 goes, no, we won't fix all reported bugs in this release.  Hope to burst anyone's bubble, but aD left us with a lot of shortcomings, unfinished pieces, and out-and-out bugs in the code base.  While I do hope we can fix many of the out-and-out bugs, the first priority is to make sure we have a solid replica, if you will, of the aD 4.2 code base running on both Oracle and Postgres.
<p>Of course we will want to move quickly on to address shortcoming and to fix as many bugs as possible in the coming months, but unless folks want to wait until 2002 for a release, we're going to have to accept the fact that we can't make all the improvements nor fix all the shortcomings in the code base before making our first release.
<p>Does this mean that testing and bug reporting is therefore a fruitless task?  My guess, Yon, is that you're just about the only person who will look at it that way.
<p>Part of our test plan will include working to make sure package porters are kept up to date on bug reports on their packages, that we get some idea as to how and when bugs will be fixed, etc.  I can tell you right now, though, that the person who has offered to *coordinate*  this effort isn't going to have time to do all of the work themselves.
<p>They will need help.  We've already been talking about the need for  things like regression testing, good performance testing and the like  (most of which *won't* happen during this first release cycle, but which we want to get started on).  In other words a more formal testing program.  These things don't happen by hand-waving on the part of consumers, they happen by virtue of hard work on the part of producers.
<p>Now let me repeat: we're looking for volunteer testers.  Let's not turn this into another argument over whether or not the project's on the right path, OK?  If you want to discuss such issues, please start another thread so folks don't lose sight of the purpose of my post.
Collapse
Posted by Matthew Terenzio on
Don,
Perhaps you can expand a little on what will be required of testers. For instance, I haven't been able to help much around here because I'm still learning some basics. (Everyone has been a great help! :)) But I have built a few sites using OpenACS and I certainly can afford the hardware and some time. Are you looking for OpenACS users who have a general understanding of things, or would testers need to be at the level of developing and porting modules themselves?
Collapse
Posted by Don Baccus on
I think we'll be looking first and foremost for testers who will be willing to install the Oracle and Postgres versions of OpenACS, and to test for basic functionality and congruence between them.

It's intentionally a bit vague because I've been begging for a "testing czar" to volunteer for some time, and I think we have one.  I'm into delegation, so I would expect to work with him to put together a plan that includes detailed expectations of volunteers.

In short, I guess, if you think you might want to help e-mail now, and if the strategy ends up looking like too much work or commitment, feel free to drop out later.

It will likely be less than the commitment level asked of porters, but not much less...

Collapse
Posted by Tilmann Singer on
I am currently working on a this regression testing tool for web applications, which I use for testing an openacs 3 based ecommerce app. I would love to see it used for automated testing of OpenACS 4 and hereby offer to do my best to make it suitable for that purpose.

tclwebtest is sort of a clone of HttpUnit - the java web app testing tool that was used in a few places of ACS 4.2 - just with way less features but hopefully with an easier syntax to write tests in (well, it's tcl after all). The HttpUnit tests that can be found in the openacs-4 cvs have been already ported to tclwebtest and work fine.

Don, your post reminded me that I promised to produce some examples for simple openacs tests, sorry for forgetting about that. I have added some examples in the tclwebtest cvs now - you can look at their output here: oacs4examples. Those include also the ported ones from HttpUnit.

Is there a need for a regression testing tool at all? Or does the nature of this project demand a different approach to testing (no idea, since I have no QA experience). Second - are there people here that would be willing to help me in working on the open issues of tclwebtest? There are a lot of bugs/unfinished features and some design decisions to be made - most notably a flexible way of structuring many single tests into a suite is still missing.

It would be great if some people involved with testing here could take a look at it and decide if they think it could be used for OpenACS.

BTW, I do have already a test script working that does an automated install of openacs 4, including a drop and create of the db, restart of aolserver, data-model installation etc. It could easily be extended to grab the latest cvs version, install it and run some tests on that. Currently it is very specific to my local setup and I would need some help to make it more generally usable.

Collapse
Posted by Don Baccus on
Well, not entirely by coincidence I downloaded your tool on Saturday and took a look at it.  I think it can be *very* useful, even in its primitive state.

I do believe regression testing using a tool like this can be very useful for development of something like OpenACS 4.  It can be especially useful for driving the development of scripts that can be used to help establish confidence that the PG and Oracle versions work
the same way.

Stay tuned, expect more news in a day or so ... and for those who've e-mailed offering to help thanks.  Be a bit patient ... we'll be back to you in a few days at most.

Collapse
Posted by defunct defunct on
Hi Tilmann,
Yes, looks v.useful. One of my guys is currently looking at it as is pretty interested. I suspect he'll want to pursue this with you further.
Regression tests... absolutely! Apart form anything else AD left quite a few holes in acs4.2 so I guess for you could easily look at regression in this case as being acceptance testing 😉
I'm also pretty interested in your installation script. Quick, simple, automatic installation is (IMHO) a great asset for any product. Its usually the first experience a new user has, and can therefore really turn them on (or off) to it. I'd be happy to pitch in and do what I can if you wanted to send it over/post it up somewhere.

Cheers
Simon

Collapse
10: Install Script please (response to 1)
Posted by Jonathan Marsden on
Tilmann,

Please upload your install script (even just as it is) to a file in /new-file-storage so we can all see it.

This could be very useful for me in trying to build OpenACS4 RPMs, which IMO offer an appropriate route to an "easy" installation for many (posibly most) newcomers to OpenACS.

Thanks!

Collapse
Posted by Peter Harper on
As Simon mentioned above, we've been doing some work with automated testing as well. We as an organisation use an XP development methodology, i.e. we write test scripts before the actual code.

I've been working on a tool for testing the functionality of Tcl functions within the system. This is slightly lower level to Tilmann's system, which is testing the system from the perspective of a user. Really, each approach complements the other.

This package is currently based on the AD-ACS 4.2, but I'm in the process of porting it to OpenACS 4.

To summurise, this tool allows you to:

  • Register testcases (chunks of code)
  • Report test results in a standard way
  • Web tool for running complete set of tests, or individual tests.
  • View testcase bodys and test results through the web-based admin tool.
  • Stub Tcl functions for driving the logic-flow and testing incoming parameter values.
We've used this tool for a little while now, and have found it extremely useful, and has significantly reduced overall development effort by trapping problems early. I'll post up more information once the port is complete, but if anyone would like any other info in the meantime, just drop me an email.
Collapse
Posted by Don Baccus on
Yes, this stuff is complementary to Tilman's stuff.  That's great.

Simon and friends will be coordinating the testing effort and we're working on putting together a plan (mostly Simon's working and I'm being thankful, to be honest).

Collapse
Posted by Tilmann Singer on
Jonathan, the experimental installation script can be downloaded from here: install.test

Customize the values at the top and then run it with

$ ./tclwebtest install.test

There is some stuff in it to save an existing parameters/nsd.tcl file, delete that if you don't need it.

There is one major drawback - tclwebtest is not able to access webservers on other ports then the standard web port 80. This is due to a bug in the tcl http package (already reported) I believe. So you need to run this with root privileges if you want to start your webserver to run on port 80. (And of course an installation script must be able to restart the webserver). Or you have some virtual hosting set up.

No guarantee that it works on any other machine then mine though - I would like it to be more general but it definitely needs more work.