Forum OpenACS Q&A: Release 4.6 Testing

Collapse
Posted by defunct defunct on
Hello one and all!

Yes its that time again! A new release, 4.6, is fast approaching, and to ensure its worthy, working and behaving we need volunteers to once again take part in the testing.

For those of you who remember the last phase, you'll probably recall we didn't get as much help/support as required. That was a real shame, but I suspect we'll have better luck this time!

For this phase we now have a couple of Test Servers for people without suitable resources, and I may even have an offer of some Oracle installations in a similar vein.

So........
The real question is 'Who's gonna volunteer?

There are going to be a few changes from last time. Most notably last time I asked people which ones they'd test. What I ended up with is authors/users testing packages, and too many people volunteering for the same package. So this time I'll allocate differently.

If you volunteer I will assign you a package(s) based on priority and need (and then on circumstances).

What we require is people to perform Acceptance Testing! This is not a highly technical exercise and largely speaking, if you can use a mouse and a web browser then you can test a package.
I want people who will sit down with a package, install it, use all its functions, and record the results. You don't have to fix bugs, understand code, or even figure out what went wrong. Thats the job of the developers! All you need to do is report it when it looks wrong.

Deliverables
Each volunteer will need to do the following:

  • Produce a test plan/checklist. This will be a simple list of test performed, that can be re-used each release.
  • Exercise each test for the given package.
  • Record the results for each test.
    We'll then compile up all the results and produce an Acceptance Test Report that gives everyone a resonable idea of the state of the release.

    It couldn't be simpler!

    We estimate that testing the average package should not take much more than a couple of hours. Once initial plans are produced it will be even quicker!

    I know many of you could spare this amount of time. We have about 15 separate items to be tested. So We don't need an army of volunteers, and handful of dedicated persons, could really make the difference.

    Lets remember, code without testing is meaningless! This is a great way for anyone to contribute and add real value to the community......

    Also, If any of you out there work for companies who've done testing on OpenACS before as part of your work (for a specific delivery lets say) we'd be very, very glad of any test plans/details and so forth you might have generated. These might make an excellent start point! Think of all those dull hours you spent preparing them... why not put them to some more altruistic use and donate them!

    Looking forward to you help

    Warmest regards
    Simon

    PS: I won't be contacting people till nearer the time, so just let me know your keen and I'll be in touch.

    PPS: Any of you who helped out last time (many thanks) and who have suggestions/experience/advice on how we can improve our ability to carry out this function, please let me know.

  • Collapse
    Posted by Gilbert Price on
    Simon,

    I'll be happy to help out in any way I can, sign me up...

    Collapse
    Posted by Torben Brosten on
    Simon,

    "if you can use a mouse and a web browser then you can test a package"

    This statement seems to contradict these statements:

    "I want people who will sit down with a package, install it.. "

    "Each volunteer will need to ..Produce a test plan/checklist. This will be a simple list of test performed, that can be re-used each release."

    Volunteers also need to know:

    1. where to find available packages

    2. how to prepare a package for install (what needs to be decided operationally before reaching this point)

    3. documentation on how to install the package --including understanding the jargon and concepts presented in the package documentation

    4. other underlying assumptions about how packages fit into the system.

    These implicit assumptions may be clear to developers, but without clear documentation, I don't see how "anyone" who can use a mouse and browser can volunteer for testing.

    Collapse
    Posted by Kjell Wooding on
    Is there a preferred format for these test plans?
    Collapse
    Posted by Torben Brosten on
    I volunteer, Simon. I look forward to your instructions.

    I should be able to provide some basic feedback on docs and default functionality, but won't be able to create a complete test plan, since I have limited experience in knowing and using the great customizing features associated with each package.

    Preliminary observation: missing links

    I've been skimming the package docs, and note that some links still point to arsdigita.com (and are subsequently redirected to a nonspecific RedHat.com page still). Making these arsdigita docs available at some alternate location could really help clear up some "missing links" in the package presentations (pun intended).

    Collapse
    Posted by defunct defunct on
    Torben,

    Apologies if you've missed earlier postings on this subject, but I *did* make it very clear that we have a couple of test servers loaned to us by Hub.org that you can use for testing... i.e. we can install and make a package available for you to test on...

    Hence you *can* do testing with just a mouse and a browser (oh and maybe a pencil & paper ;).

    Hope that helps and thanks for volunteering..

    Kjell: I'm just finalising a simple format at the moment. We want to keep it to high level, simple test descriptions... for example.

    General Comments Test: Submit a general comment as a registered user, then withdraw the permissions for that user to 'write'. Attempt to post again

    Expected Result: User should not be permitted and receive an error message....

    Actual Result: Entire system fell over with db_transaction error.. (see log file)

    That about all we're aiming for as theses are 'user' or 'acceptance' or 'customer' tests.

    The secondary objective is to try and build up a set of reusable test plans, some I'm particularly keen for people to keep a record of tests thats clear and easily repeated...

    As you prpbaly know e already have an acceptance test server for recording suc things, and we hope to offer another instance for this cycle.

    I'll be in touch nearer the 15th when the release is clarified

    Cheers
    Simon

    Collapse
    Posted by John Norman on
    We would be happy to do some testing, because it can be part of our learning about OpenACS functionality. However, since we are new to OpenACS we may need help with the test plan or to be given a package whose intended functionality is well documented. If we can help within these constraints, count us in.
    Collapse
    Posted by defunct defunct on
    John,

    I'd be very grateful of whatever assistance you can offer.

    We can offer help with setting it up, test plan outline and I will try and find a package for you that is reasonably well documented in terms of functionality.

    Thanks for the offer!

    Interestingly this highlights the lack of documented functionality for each package.

    Don: Perhaps this is somehting we could suggest to package developers. i.e. a hitlist of package functionality that should be tested?

    Collapse
    Posted by Torben Brosten on
    Simon, I'm sorry for any critical tone in the first post. I really meant just to clarify. I know some people who would like to beta test, and they can be really great end-user beta testers, but their conceptual understanding of openacs varies to none, and therefore I believe precludes them as testers based on posted expectations. --Someone would need to document (express) their findings, questions etc.  I'll get them to try it out (and translate their observations). It should be fun.

    Testing on hub.org? I was hoping to use that service. Great!

    Collapse
    Posted by Brad Punt on
    You can add me to the list of volunteers, Simon
    Collapse
    Posted by Ben Koot on
    Simon,

    If there''s any thing I can do, let me know.

    Collapse
    Posted by David Kuczek on
    Simon, you can also put me down on the list.

    It sounds great that the toolkit gets reinstalled on the hub.org servers daily!

    Collapse
    Posted by Luigi Martini on
    I'd also contribute, though I am afraid Simon is just cheating when saying that anybody can contribute.
    I'm actually at a stage of not even being able to install OACS 4.5 for the first time (see my last post, still waiting for help).
    I feel much involved in the thing, but I'm just a beginner.
    So, if you are serious, Simon, and the task is easy: here I am.
    Collapse
    Posted by Jun Yamog on
    Hi Luigi,

    Welcome to OpenACS.  I would like to note that such strong words against Simon or any community members especially those who have contributed a lot is not advised.  Simon has done a lot to OpenACS even put in resources for free for the community.

    I am sure Simon will give you a task especially if you ask him kindly.

    Collapse
    Posted by defunct defunct on
    I'm sure luigi meant that in a light hearted way ;)

    Even a begginer can help. Jon has a couple of test servers we can use, so we can set up the installation/package for you.

    What we need volunteers to do is step through all of the funcitonality of a package and record their observations.

    The new acceptance test recording method Jon and I are proposing will also allow you to indicate your level of experience, so others can evaluate the completeness of testing...

    I'll be in touch next week (assuming everything else is on track)

    Collapse
    Posted by Luigi Martini on
    Jun and Simon,
    I did mean what I wrote: it is maybe possible that my poor knowledge of your language prevented me from clearly expressing my thoughts. I retry with different words:
    1) Simon says it is easy to contribute.
    2) Torben Brosten suggests that maybe it is not so easy...
    3) I'm a beginner, not yet able to install plain OACS 4.5, despite my many efforts.
    4) People writes about testing on hub.org: something unknown to me. This increases my fear not to be able...
    5) Still, I would like to contribute.
    6) Thus, Simon, please add me to the list.
    As you see, no strong words; if any, just something agaist me: fearing to be a burden for the community instead of bringing some help.
    Collapse
    Posted by defunct defunct on
    Luigi,

    Even if your new, testing is a good place to start, and a good way to familiarise yourself. And the more people who become experts the better!

    I'm sure you'll be able to make a useful contribution...

    Don't worry about the hub thing. Jon Pike will assist you with that next week (mailto:jon@open-msg.com)... it means you can test a package without the hassle of installation etc...

    Collapse
    Posted by Torben Brosten on
    Welcome, Luigi,

    You write:

    2) Torben Brosten suggests that maybe it is not so easy...

    My clarification does not apply to beginning administrators or developers, such as yourself.  I meant to clarify that: this testing is not for end-users. The role-objectives of end-users are very different from beginning and experienced OpenACS administrators and developers.

    Lack of experience can be an asset when testing a system for completeness. Both testers and the OpenACS system can benefit from the testing experience. Where else can you try things you are unsure of; be certain to get useful, immediate, feedback; and immediately contribute to the project?

    ciao,

    Collapse
    Posted by Ava Jean Aldovino - Negrillo on
    In response to your alert about the Release testing of OpenACS 4.6, our company is pleased to extend our testing capabilities before
    the release of your new package. It would be a great chance for us to give back our appreciation since our company is using it as our main platform.

    We could offer you our testing cycles and resources that we normally use.
    We could deliver the following items as needed.

    1. QA Testing Mechanics -  type of testing that should be done
    2. Status Bug Report - list of issues and bugs found
    3. Test Plan
    4. Checklist

    Please let me know your thoughts and from there maybe we can start discussing the items.

    Collapse
    Posted by Richard Hamilton on
    Delighted to work with the community on testing the new version. Looking forward to the release very much. Regards Richard