Forum .LRN Q&A: RFC: Evaluation System for .LRN

Vision

Professors and students need a system that lets them upload the tasks and evaluate on line. With the evaluation system the professors can administer the grades on line, the assignments and have access to the grade reports. The students can submit the tasks on line and see their grades at any given time.

These are the basic requirements of the system:

  1. Post assignments.
  2. Categories for the assignments.
  3. Edit the ratios of the categories and assignments.
  4. Assignments groups.
  5. Evaluate the assignments.
  6. Grades reports.
  7. Submitting answers.
  8. Reports for the student.
  9. Tracking changes / Audit.
  10. Give each user different privileges.
  11. Group creation workflow (extra).

Please review the full specification here:

http://www.galileo.edu/public/files/Evaluation_System_Spec.pdf

We would like to hear your feedback to keep the development going on!

Collapse
Posted by Jose Pablo Escobedo Del Cid on
Also, you can check the current version (based on ACES) that is used in the Galileo University here:

http://jopez.galileo.edu

user: jopez
pass: 123
(The system is in spanish)

In "Mis Grupos" portlet you will have access to the course "Tecnologias de Aplicaciones Web" (jopez is the proffesor of the course), you click on this link and you will go to the page of the course where you can post assignments and evaluate the students. You can do anything you want in this demo site, and it will not send emails to the students ;)

Our goal is to migrate all the functionallity to .LRN in 4 weeks (starting next week)

Any comments/suggestions will be appreciated

regards, jopez

Collapse
Posted by Malte Sussdorff on
This looks like a good step into the right direction. You might want to coordinate though with the .LRN Assessment team, as the assessment system shall include most, if not all of your functionality. It would be sad if we, yet again, develop multiple systems to do the same task.

If you are not sure what I'm talking about, take a look at https://openacs.org/projects/openacs/packages/assessment/ (and especially the last part at https://openacs.org/projects/openacs/packages/assessment/rfc/).  An assignment can be seen as a question of type "file".

In any case, you should use grade_id in your evaluation (instead of grade, if you are using grade table). Add a locale to the grade table, to allow multiple grading scales, if this is wanted. But I better stop here, just read the specs ;).

How quickly do you need to start on this, as we already have a couple of interested parties in the whole assessment/grading/assignment area. Sadly the sponsors so far had different priorities, but it would give the opportunity to pool the money (as well as developers/experience) and get the packages working right and beat commercial systems at it.

Collapse
Posted by Rocael Hernández Rizzardini on
Yes, sure, we would like to coordinate with any the effort going on. Our deadline is tight and we want to have it finished for testing by the time stated by jopez, anyway extending this module to use a grading service should be more less easy, and will be good if the evaluation & assessment effort can be shared and coordinated.

The actual design is for evaluating and grading projects, assignments, tests or any other category that the professor might want to use, no matter if the assignment is uploaded or its a test done in the classroom, a project done by a group, etc.

The grade_id could be used if a service is provided for grading, and I think that service can also handle internally what kind of grading scale is used by that grade_id. (in Guatemala is usual the 1-100 scale)

I don't find any spec for such grading service? (just Scoring/Grading paragraphs)

Collapse
Posted by Rocael Hernández Rizzardini on
BTW, I forgot to mention that this effort will be done by Galileo University (www.galileo.edu) in our long term commitment to use and improve .LRN.
Collapse
Posted by Rocael Hernández Rizzardini on
As a matter of fact, we plan to develop also an online testing tool, specialized for distance learning (non-presential), in this sense the assessment tool will be more than useful. This is not a priority in the very short term, but is something that we for sure will need in this year at Galileo.
Collapse
Posted by Caroline Meeks on
Hi Roc,

I had the oportunity to revisit the idea of ePortfolios

(https://openacs.org/forums/message-view?message_id=52750)

When someone mentioned they are trying out the open source version of this product.

http://www.rsmart.com/ePortfoliodemo.php

It seems useful and not difficult to implement in OACS. Do you anticipate your evaluation project having portfolio like funtionality?

Collapse
Posted by Rocael Hernández Rizzardini on
Hello Caroline,
at the moment we're not trying to address this functionality, though seems not difficult and useful, but more related to a file-storage service, don't you think?

The evaluation system is more intended to "gradding book" plus handling assignment, project, etc.

Collapse
Posted by Caroline Meeks on
Hi Roc,

I realize you aren't trying to create a portfolio system I'm just interested in understanding both portfolio systems and what you are building better.

Portfolios are intersting because there is so much in OACS that is a lot like one yet we don't actually have the functionality.

For instance, in the Homework package a student uploads a document as evidence of having done an assignment.  Then the instructor uploads comments. So far this is very similar to a portfolio system. But in a portfolio system there is a view of all of a student's work not just assignment by assignment and there is addional flexibility to control who has read on both the documents and the comments.

You're system will  have the student centered view of assignemnts right?

Can you view a grade report by student across classes or just for all assignments within a class?

Does the instructor just submit a grade or can he also submit comments?

In your system, does a student submit a file storage document as thier assignment, this is the way the homework package works, or are assignments submitted through a more survey like interface?

I'm not trying to add any requirements, you have already bitten off plenty! Just trying to understand what you are doing and think about different ways the same functionality might be reframmed for marketing purposes.  Maybe, if its somewhat close, the portfolio products might give you some UI ideas.

Thanks
Caroline

Collapse
Posted by Nima Mazloumi on
Hi Rocael,

I was just wondering how far you are with the development of your evaluation system. We started our dotLRN instance here at Mannheim and almost every professors asked me if something like an assesment tool is offered or not.

Will you make your tool available over cvs? Do you have a deadline for your project?

Greetings,
Nima

Collapse
Posted by Rocael Hernández Rizzardini on
Hello Caroline,

<blockquote> In your system, does a student submit a file storage document as thier
assignment, this is the way the homework package works, or are assignments
submitted through a more survey like interface?

</blockquote>

Will work more in the sense of how the homework package does, instead of the survey like, since the experience of running this package for almost 2 years has shown that that interfase is the best. The UI of this package, specially, has been used / tested by several professors, TAs, students, so its quite polished (that doesn't mean that it cannot be improved, but at least we will be for sure that is a good one)

Of course, a cross-reporting class functionality will be really useful and not difficult to make. Actually the reporting area of the tool is one that has been taken a big amount of the work, so better wait to see it or try it online, and then give us feedback to improve it.

Collapse
Posted by Rocael Hernández Rizzardini on
Hello Nima,

<blockquote> I was just wondering how far you are with the development of your evaluation
system. We started our dotLRN instance here at Mannheim and almost every
professors asked me if something like an assesment tool is offered or not.

</blockquote>

Sure, we'll be happy having you using it! And you'll be able to use it quite soon ... =)
I think there's a word that is making some misunderstaning in some of us, rigth now we understand:
assessemnt = online testing tool
evaluation = gradding book + a tool for handling assigments, projects, etc

<blockquote> Will you make your tool available over cvs? Do you have a deadline for your
project?

</blockquote>

Yes, this will be GPL and will be available to cvs.openacs.
We got several delays related to events that we were having here. Our plan is to have the beta version by the end of march. I'll let you know when we have something to test, so you might help us on that.

Collapse
Posted by Jose Pablo Escobedo Del Cid on
Hello Caroline

You're system will  have the student centered view of assignemnts right?

-Yes, and it will support group assignments too (parties) for the assignments thar are in groups.

Can you view a grade report by student across classes or just for all assignments within a class?

-Currently we only have thought about reports for the professors for all assignments within a class, but an administrator can see the reports of the students across classes.

Does the instructor just submit a grade or can he also submit comments?

-He can also submit comments and decide whether the student can see the grade or not.

regards, jopez

Collapse
Posted by Caroline Meeks on
Thanks Roc and Jose I think I have a clearer picture of what you are building and I am looking forward to seeing it.

For what its worth it might well be considered a "Portfolio Service" for evaluation as defined by the ePortfolio Conference. :)

http://www.eife-l.org/portfolio/Default.htm

Collapse
Posted by Nima Mazloumi on
I will be happy to help you out with testing and i18n.

Also, I have now a better understanding of your evaluation system aswell. First I thought of something like an extended survey tool that serves as an assessment tool.

One important thought I had on your evaluation system was the possibility to assign tasks to groups rather than single students. Thus a bunch of students can form a group that works on the tasks given by the professor.

Why this feature you may ask:

When a professor simply wants to improve the interaction between him and the students on the one hand and his own course on the other without thinking of grading in first hand he could offer a kind of a competition between groups of his students who participate. The is a deadline for submission the students have to stick with. The professor will score the replies and students can see the ranking of their group for this competition. This has a great fun and motivation factor.

We have something similar here at the university written in php and the course where this tool is used is one of the best visited because students identify themselves more with the subject and there exists a play element in the course.

The professor can use the replies of the groups to improve his course by focusing on those subjects the groups had most problem with.

Greetings,
Nima

Collapse
Posted by Nima Mazloumi on
I just read your specification and saw that you support groups already. Fantastic. Do you have a group report for the groups with something like a ranking?

Greetings,
Nima

Collapse
Posted by Jose Pablo Escobedo Del Cid on
mmm, we have grades reports for the assignments (in groups or individual), and you can order the reports by grade, but those reports can only be seen by the professor.

greetings, jopez

Collapse
Posted by Nima Mazloumi on
Is someone of you able to come to the .LRN Spring Meeting in Heidelberg, 23 April 2004? When do you expect your evaluation package to be finished?

Greetings, Nima
Collapse
Posted by Rocael Hernández Rizzardini on
Well, my plan is to go, but not sure right now, anyway, we are doing good progress on the system and as I said, probably by the end of this month we'll have the beta version of the system ready.
Collapse
Posted by Nima Mazloumi on
If you need a place to stay let me know :)
Collapse
Posted by Cathy Sarisky on
Hi Rocael,

I've tested out the evaluation package from head (actually a few weeks ago now).  That's a very slick package!

May I offer some help and suggest a feature?

Offer to help:
- The en_US translations maybe need a little bit of work.  I was confused a few places by the terminology used.

Feature suggestion:
- It would be very useful to be able to insert scores into the gradebook from outside the evaluation package (for instance, as the result of a survey/assessment/whatever, but I couldn't find any public tcl API for doing so.  I hacked in a work-around for some code that assumes it is being called from within the evaluation package.  (Find the evaluation package within the dotlrn class rather than relying on [ad_conn package_id] to return the evaluation package_id.)  However, then I got stuck on trying to insert a grade, probably because the CR makes my head spin.

Collapse
Posted by Rocael Hernández Rizzardini on
<blockquote>The en_US translations maybe need a little bit of work.  I >was confused a few places by the terminology used.
</blockquote>

feel free to change the "terminology" in CVS.oacs", we do need that  to make more sense in some words in the english context.
Any other suggestion is very welcomed!

<blockquote>Feature suggestion
</blockquote>

we'll provide a service-contract to deal with that, then easier integration with another applications, like assessment, etc....