Question Catalogue: Summary:

The question catalogue is a central part of the assessment system. It deals with the storing of the various questions that can be used in a survey. You are able to add/edit/delete a question of a certain type to a certain scope. Furthermore, it allows you to search and browse for questions for inclusion in your assessment as well as import and export multiple questions using various formats. This concept is new to survey 0.1d and changes the design of the survey module considerably. No mockups available.

Spec:
All questions have some common ground.

• Questions must be scopeable, scope should be:
  • site-wide for questions that are useful for the whole site
  • package-wide, which is useful for dotLRN communities, so you can have questions which are just useful for your class
  • survey-wide, so you can add questions, that are only useful for one survey. This is the default if you add a question using the normal survey interface
• Questions should be able to be assigned to multiple categories with multiple hierarchies (e.g. catogorize by symptoms or by classes). These categories must be editable site wide.
• Questions will have a title, so they can easily be found again.
• Questions have to be versionable. This allows a survey to use the older version of a question, if the question is changed.

For each of the various question types, there will be a separate input form instead of the currently used method. A user selects a question type to add and is then redirected to the question type add form.

• True for all Question types
  The following fields are true for every question type:
  • Category
  • Title
  • Scope
  • Question: An richtext HTML widget (look at bug-tracker for an example of this widget)
  • Graphik: A file for upload from the local harddisk to the content repository
  • Use old answers: boolean (yes/no). If yes, the latest response of the user to this question will be given as a default value
  • Data validation steps are fairly complex because we need two layers of data validation checks:
    • Intra-item checks: the user input { exactly matches | falls within narrow "target" bounds | falls within broader "acceptable" bounds with explanation}
    • Inter-item checks: if { a user input for item a is A, item b is B, ... item n is N } then { user input for item z is Z }

    Both levels involve stringing together multiple binary comparisons (eg 0 < input < 3 means checks that 0 < input and input < 3), so we need to express a grammar consisting of

    • comparison1 conjunction comparison2 conjunction ... comparison n
    • appropriate grouping to define precedence order (or simply agree to evaluate left to right)
• Open Question
  Besides categories and title (which is the same for all questions), open questions have the following entry fields:
  • Size of the reply box: Radio buttons (small/medium/large)
  • Prefilled Answer Box: richtext widget. The content of this field will be prefilled in the response of the user taking the survey
  • Correct Answer Box: richtext widget. The person correcting the answers will see the contents of this box as correct answer for comparison with the user response.
  • [NTH]: Button to add predefined comments next to the correct answer box. This would be shown to the manual corrector to quickly choose from when manually scoring the answer.
• Calculation:
  This type of question will not be supported. But we should make sure we can take care of that type while importing the data. Therefore, we have to know the values. And while we are at it, we can as well just generate the input form :-).
  • Formula: string
  • Units
  • Value (in %): integer
  • Required (boolean)
  • Ignore Space (boolean)
  • Ignore spell checking (boolean)
  • General Feedback: richtext
• Short Answer Question:
  
  • Number of Answerboxes: Integer Selectbox. This will control how many answer boxes the respondee will see.
  • Upper/Lowercase: Radio boolean. This will control, whether we treat the response case sensitive when comparing it to the correct answers or not.
  • The questioneer has the option to define multiple possible correct answers that will be matched with the response of the user in various ways. For each of the possible answers the following fields are given:
    • Answer: short_text. This contains the answer string that will be matched against the response
    • Value in %: short integer: How many percentage points a match will awarded.
    • Size: Integer Select: size of the input box (small, medium, large)
    • Compare by: Select (equal, contains, regexp). This defines how the comparison between the answer string and the response shall happen.
    • Allow in answerbox: (multiple select box with "All" and the numbers from 1 to x where x is the number of answerboxes from above. For sure this only works with JS enabled :)). Defines the answerboxes the user can fill out that shall be matched with this answer. w
• Matching Question:
  Matching questions are useful for matching some items on the left with pull down menus on the right hand side of a survey. The number of the items is identical to the number of items on the right hand side.
  • Settings:
    • Distribution of points: boolean (all or nothing / increasing). All or nothing will either give 100%, if all correct answers are given, or 0% else. Increasing will give (number of correct matches / number of total matches) *100% points.
    • Allow negative: boolean (yes/no). This will allow a negative percentage as well (as the total result).
  • A couple of match entries will be presented below the settings. Each one will consist of:
    • Match item: This is the item which will be displayed on the left side of the question presented to the respondee.
    • Matched item: This is the correct item from the select box on the right side. For each match item on the left there will be a select box on the right with ALL the matched items (when taking the survey, that is...)
  • In addition to submit, there is another button to allow further answers to be filled in. Typed in values shall be remembered and 4 more answerboxes be shown.
• File upload question: A file upload question will allow the respondent to upload a file. No additional settings but the usual for every question.
• Multiple Choice question: Multiple Choice questions allow the respondee to choose from multiple alternatives with the possibility to answer more than one at a time.
  • Settings:
    • Allow Multiple: boolean (yes/no). This will determine if the respondee has the option to choose multiple possible answers for his response.
    • Select Box: boolean (yes/no). Will display a select box or radio/checkbox otherwise.
    • Distribution of points: boolean (all or nothing / increasing). All or nothing will either give 100%, if all correct answers are given, or 0% else. Increasing will give (number of correct matches / number of total matches) *100% points.
    • Allow negative: boolean (yes/no). This will allow a negative percentage as well (as the total result).
    For each (possible) answer we have a couple of fields:
    • Correct answer: boolean, radio with grafik (red x, green y) (yes/no). This marks if the current answer is a correct one.
    • Answer: Richtext widget.
    • Value: percentage value this answer gives to the respondee
    • Reply: richtext widget. This is a reply the student can see at the end of the test giving him some more information on the question he choose.
  • In addition to submit, there is another button to allow further answers to be filled in. Typed in values shall be remembered and 4 more answerboxes be shown.
  • Additionally there is a button "copy", which copies the contents of this question to a new question, after you gave it a new title.
  • [FE]: Possibility to randomly choose from the options. This would add a couple of fields:
    • To each answer: Fixed position: Select Box, Choose the mandatory position when displaying the option (e.g. "none of the above").
    • Number of correct answers: integer, defining how many correct options have to be displayed. Check if enough correct answers have been defined.
    • Number of answers: integer, defining how many options shall be displayed in total (correct and incorrect). Check if enough answers are available.
    • Display of options: Numerical, alphabetic, randomized or by order of entry.
• [FE]: Rank question.
  Rank questions ask for the answers to be ranked.
  • Rank Type: Boolean (alphabetic, numeric). Shall the rank be from a to z or from 1 to n.
  • Only unique rank: Boolean (yes/no). Shall the ranking only allow unique ranks (like 1,2,3,5,6 instead of 1,2,2,4,5)
  • Straight order: Boolean (alphabetic, numeric). Shall the rank be in a straight order or is it allowed to skip values (1,2,3 vs. 1,3,4)
  • For each answer we ask the following questions:
    • Answer: Richtext widget.
    • Rank: correct rank
  • In addition to submit, there is another button to allow further answers to be filled in. Typed in values shall be remembered and 4 more answerboxes be shown.
• [FE]: Matrix table (blocked questions)
  A metric table allows multiple questions with the same answer to be displayed in one block. At the moment this is done in the section setup (if all questions in a section have the same answers they would be shown in a matrix). One could think about making this a special question type on it's own.

Only site wide admins will get to see the following question types:

• Database question:
  The answer to this question will be stored in the database. The question has the following additional fields:
  • Table Name: short_string. This is the name of the table that is being used for storing the responses.
  • Column: short_string. This is the column of the table that is used for storing the responses.
  • Key Column: short_string. This is the column of the table that matches the user_id of the respondee.

Concerning permissions here is the current thinking:

• A question can be changed only by the creator or any person that the creator authorizes. To keep it simple for the moment, a person that is authorized by the creator has the same rights as the creator himself.
• If a question is changed, all survey administrators, whose survey use the question, are notified of the change and given the opportunity to upgrade to the new version, or stick with their revision of the question.
• If an upgrade happens we have to make sure that the survey gets reevaluated. Unsure about the exact procedure here.

There needs to be an option to search the question catalogue:

• Search term: short_text. What shall be searched for
• Search type: select {exact, all, either}. Search for the exact term, for all terms or for one of the terms given.
• Search in: Checkboxes (Title, Question, Answer, Category, Type). Search for the term(s) in the Title, Question, Answer, Category and question type (multiple, short_answer ...). Obviously search only in these areas which have their checkbox set to true.
• Browse by category (link). Link to allow browsing for a question in the category tree.
• The result should show the question title, the type of the question, a checkbox for inclusion in a survey. The following actions are possible:
  • Include the marked questions to the current section, if section_id was delivered with the search.
  • Delete selected questions
  • Change scope of selected questions
  • Export questions in CSV, Blackboard, WebCT or IMS format.

Operations on questions:

• View. View the question in more detail (all settings along with a preview of the question)
• Edit. Edit the current question. On submit:
  • Store a new version of the question.
  • Mail all current survey administrators using this question about the update.
  • Include a link which allows the administrators to update their survey to the latest revision of the question.
  • Don't relink the survey to the latest revision if not explicitly asked for by the survey administrator.
• Copy. Copy the current question and allow for a new title. The edit screen should be presented with an empty Title field.
• Delete. Delete a question. On the confirmation page show all the Possibility to include images in answers. Currently this can be done using HTML linking. A more sophisticated system which links to a media database is thinkable, once the media database is ready.

For the future we'd like to see a more sophisticated way to include images in questions. Currently this can be done using HTML linking, but a media database would be considerably more helpful and could be reused for the CMS as well.

Calculation and Database Questions

I'm not clear from your description what these are. If by Calculation questions you mean questions that produce some calculated result from the user's raw response, then IMHO this is an important type of question to support now and not defer. This is the main type of question we use in quality-of-life measures (see demo here ). These are questions scored by the Likert scale algorithm. If there are five potential responses (1,2,3,4, and 5) for a question, and the user choose "1" then the "score" is calculated as 0; if "5" then 100; if "3" then 50, and so on -- a mapping from raw responses to a 0-100 scale. Is this what you mean by a "calculation" question?

By Database questions, do you mean free text input (via textboxes or textareas) questions for which there is a "correct" answer that needs to be stored during question creation? Then when the teacher is reviewing the student's response, she can inspect the student's response against the stored answer and determine what degree of correctness to assign the response?

-- Stan Kaufman on November 09, 2003 06:29 PM (view details)