Forum OpenACS Development: Simplification of Assessment

Collapse
Posted by Carl Robert Blesius on
In the interest of simplifying things for end users Anny added the option to create "assessment types" (example: if I want to create an assement of type "survey" I do not want to have to fill out 3 pages of exam options).

Here are Anny's addtions:

http://xarg.net/tools/cvs/change-set-details?key=19299
http://xarg.net/tools/cvs/change-set-details?key=19298

Great idea, but then Eduardo adds these types:

New_Assessment
New_Survey
New_Review_Assessment
New_Partial_Assessment
New_Quick_Assessment
New_Assessment

http://cvs.openacs.org/cvs/openacs-4/packages/assessment/www/asm-admin/index.tcl?r1=1.6.2.6&r2=1.6.2.5

and now we have even more complexity in the UI (6 times as much imho): most have enough problems with "Assessment" (the word). I am not even sure what a "Partial Assessment" or a "Quick Assessment" is in this context (local needs Eduardo?).

I want to remove these options from the UI for now and pursue a different model in the future (e.g. assessment just provides questions, sections, groups of sections/questions, and library files for use in other packages that provide a UI for differnt use cases). For now getting the standard/advanced "Assessment" UI to work should be our main goal.

Anyone using those types? Objections to getting rid of those options?

Collapse
Posted by Bernd Simon on
We are using assessment in two portals, http://www.lerntransfer.at/gfp/cycles and http://www.evaluate.at as a survey tool.

Based on this experience I would love to see an assessment UI that distinguishes between survey and assessment at all levels (assessment, section and question), because now have to take out the assessment specific elements for our clients "by hand". However, I do not see a need for a more detailed distinction at the moment.

BTW, why not calling "Assessment & Survey" instead of "Assessment". Could be that a majority of usages are in "survey mode" although this is not reflected in the name.

Collapse
Posted by Malte Sussdorff on
I agree, one "assessment" which is the current complex UI and one "survey" mode would probably be enough. Also we could think about reducing the complexity by making good guesses when a new question is added (aka: the default ones), and let the user edit the parameters if he wants to see them changed.

Last but not least, currently the assessment system has (had) the major hickup that it was nearly impossible to edit once entered data. I took care of that one for one of our clients.

Once Heidelberg moves to postgresql I'd suspect that some effort can be put into making all the nice changes from HEAD (not sure what has been backported, I'd count better performance and lickert scales among the major ones) work back in the postgresql 5.2 version of assessment, while retaining any changes done in your simplification work.

Collapse
Posted by Rocael Hernández Rizzardini on
Why Eduardo is always messing all the things???

I guess at Galileo we got the pre-eduardo's commit asm version, which has no many option, though we still think needs to be simplified.

Now we are working on making it scalable, for serving asm and procesing answers, more comming soon!

Collapse
Posted by Caroline Meeks on
Roc,

If you are doing a bunch of work on assessment then we should make sure we are on the same code base so we don't fix the same bugs and we can use your work. We are using the openacs-5-2 branch last updated for MGH on 2/4/06.

Collapse
Posted by Carl Robert Blesius on
After finding out that Heidelberg's fixes were not in the OpenACS CVS I asked Malte to add them to HEAD. We merged these changes with 5.2 and are now bug fixing together.

We will be cooperating on getting assessment tidy for .LRN 2.2 so please commit any local fixes (not features) you have soon to the oacs-5-2 branch.

For now we are tending towards removing all the options mentioned in my previous post and leaving "New Assessment".

Please complain or fix now, if you are using this untested/broken "feature" and are willing to help get it in.

If we work on it we will fight for just going with 2 or a maximum of 3 different assessment creation options: "New Simple Survey" (as per Bernd's post), "New Simple Quiz/Test", and "New Custom Suvey/Quiz/Test" (this one would give you all the options) for example.

Speak now, or wait for the next release round.

Collapse
Posted by Rocael Hernández Rizzardini on
We are working on oacs-5-2, though we are not commiting by now until further notice. Feel free from our side to do whatever you want with assessment (i.e. take out the assessment creation options).
Collapse
Posted by Derick Leony on
At Galileo, we are analyzing some approaches in order to improve Assessment performance and decrease the amount of deadlocks at peak times.

After error log analysis and some stress tests (which do not really simulate multiple users due to the lack of this feature in tclwebtest) our approach to follow is to remove the item data versioning (i.e. not to use the CR for item data) and to offer either a "Light Assessment" option to the user or parameter to the administrator.

We will keep on posting our results in order to help others with similar issues.

Collapse
Posted by Dave Bauer on
Could you give some more detail on what aspects of the content repository cause deadlocks?

I would assume you'd have to try updating the same row to cause a deadlock. What is inherent in the content repository that requires this?

Perhaps it is the way the assessment packages uses the CR and that it can be fixed without totally rewriting it.

At least one example of one query that causes this problem and why it deadlocks would be very helpful.

There are many experts in the OpenACS community. Let us work together to understand this and fix it.

Collapse
Posted by Derick Leony on
Since in the tests we try to simulate 10 simultaneous users answering an assessment, the deadlocks show up when creating a new session for the third or fourth time in average:

ERROR: deadlock detected
DETAIL: Process 11236 waits for ShareLock on transaction 343145231; blocked by process 11227.
Process 11227 waits for ShareLock on transaction 343145260; blocked by process 11236.
SQL:

select content_item__new('25300952-25302114-5da3ce40-1f35-4da0-9221-6c719d3b3d94','24950685',NULL,NULL,NULL,'25300952',NULL,
'10.0.0.45','content_item','as_sessions',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,'file')

but this is in a development instance.

In production, we get deadlocks at several queries:
Creating a new answer to a question:

ERROR: deadlock detected
DETAIL: Process 2988 waits for ShareLock on transaction 3017009599; blocked by process 2989.
Process 2989 waits for ShareLock on transaction 3017009471; blocked by process 2988.
SQL:

select content_item__new('24696117-24696253-25620217','25516644',NULL,NULL,NULL,'24303066',NULL,
'10.0.0.20','content_item','as_item_data',NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,'file')

Inserting a session revision:

ERROR: deadlock detected
DETAIL: Process 2975 waits for ShareLock on transaction 3017009950; blocked by process 3015.
Process 3015 waits for ShareLock on transaction 3017010057; blocked by process 2975.
SQL: insert into as_sessionsi
(revision_id,object_type,creation_user,creation_date, creation_ip, title, description, item_id, mime_type )
values('25620811','as_sessions','24266584',NULL,
'10.0.0.20','24266584-25620119-1ba075ea-2d08-4f81-b35f-2b8baca8d210',NULL, '25620807', NULL )

Inserting a item_data revision:

ERROR: deadlock detected
DETAIL: Process 2975 waits for ShareLock on transaction 3017009950; blocked by process 3015.
Process 3015 waits for ShareLock on transaction 3017010057; blocked by process 2975.
SQL: insert into as_sessionsi
(revision_id, object_type,creation_user, creation_date, creation_ip,title, description, item_id, mime_type )
values ('25620811', 'as_sessions', '24266584', NULL, '10.0.0.20', '24266584-25620119-1ba075ea-2d08-4f81-b35f-2b8baca8d210', NULL, '25620807', NULL )

and all of the above were from the same day (last Saturday).

As many people had said, I don't know why a simple insert could cause a deadlock, I wonder if it being an insert to a materialized view could be related, or if indexes like these:

"as_item_data_pk2" btree (session_id, section_id, as_item_id)
"as_item_data_pk3" btree (as_item_id, section_id, session_id)

could affect a DML operation performance. Some help from PG experts would be really appreciated :-)

I agree on the possibility of the Assessment package misusing the CR, because everything related to the CR (File-Storage, Evaluation, LORS) was working great here until Assessment was installed.

Collapse
Posted by Dave Bauer on
I wonder if its this code in the insert rule

"update cr_dummy set val = (
select content_revision__new("

cr_dummy is a one row table that is used to make the rule work in postgresql.

If its called in a transaction would it lock that table and not allow another update to that table for a new revision until the first transaction is done?

If you do an entire assessment anwer insert in one transaction and there are many questions, it could take several seconds or more to finish the transaction.

Maybe Don can comment on this.