Proposal for Survey Expansion
This document is intended to foster community discussion about the best ways to expand the survey package to make it more fully meet the needs of the community, as expressed in this thread from two weeks ago. Throughout this document I will use the following terms:
Admin(s) = Survey Administrator
User(s) = The End User (i.e. the person taking the survey)
The current 4.x based survey module provides a simple and effective way for Admins to collect information from Users and for Users to revise their submissions. It permits Admins to read individual responses submitted by Users and analyze them, which is likely the most effective way to analyze unique answers (i.e. text fields, text areas and file attachments). However, this system leaves much to be desired when it comes to Multiple Choice answer as well as Dates. Although it permits downloading information (through the use of CSV export) to programs such as Microsoft Excel, which will allow the Admin to perform a more sophisticated mathematical analysis of the material, this is not very Admin or User friendly. The methods of data analysis are insufficient for large sites where the information collected could not possibly be reviewed by the Admins (due to the sheer volume), or when one wants to provide immediate and useful feedback to Users about their submissions. We need to improve the Survey package in the following ways:
- There needs to be a way for Admins to analyze Multiple Choice/Dates survey data without pulling it off the site and using an external program.
- There needs to be a way for Users to receive an immediate customized response from the system based on information presented.
- There needs to be a way for submissions to be available to other packages (if this is desired by the site owner). Sophisticated Analysis of Surveys
In order to think about useful ways to analyzed submitted survey responses, I would like to ask you to consider the following possible uses for the survey module. Although I am sure there are other types of surveys people may want to conduct, I think that you will see that they can be captured by a comprehensive "test grading system":
- Graded Surveys in Online Classroom Use (i.e. DotLRN sites)
- Health Tests (similar to Personality Tests)
In the online classroom, the Admin (i.e. teacher) would administer tests to Users (i.e. students). If the test were a written essay, or a number of short answers to questions, there is currently no feasible way to automatically analyze responses (i.e. give a grade), and the teacher would be forced to read all the responses. Although each response grading_points_default | integer | default null (this is the default number of points given for a question - which multiple choice questions found in Survey_question_choices, grading_points field values would override - this would be a way of saying, in a essay response test, the teacher may want to give 10 points per question, and he/she will assume they get the full point value unless he/she specifies it otherwise... or something like that. must be read by the teacher, there are "tools" the survey package could provide to the teacher to make their job easier. There should be a way for the teacher (i.e. Admin) to provide feedback after having read the responses. This feedback could take two forms, either a simple "grade" (i.e. point value - which could hopefully be automatically added up for all short answers to give the complete grade for that assignment), but also possibly in the form of comments about the Users answers. For example, in response to a students short essay on how all leaders with facial hair are horrible people (Stalin, Hussein, etc.), the teacher might want to comment and tell the student to consider leaders such as Gandhi or Martin Luther King. In having talked to some professors at the school i work for, they also expressed interest in having "predefined" answers for questions, so that they can have a template with 4 or 5 different standard responses which can be customized.
If the test administered by the teacher is a multiple choice test. We should provide a way to automatically grade the test by cross referencing the users responses with a key. This could be as simple as the response to question number 1 is multiple choice value 5, thus if the User entered the "correct" value return a 1, if not return a 0 and then adding all the scores up to give a total. Although this would provide a "grade", it would be nice to also provide an explanation for incorrect answers. Thus, if the test is automatically graded, it should not only say you missed 7 of 50 questions, but also tell you which questions you missed and give you an explanation for the correct answer. In testing one things that has little to do with Grading but more to do with the design of a test, which would be useful for professors is the ability to write out a short essay for display and then follow it with a section of multiple choice question... although the grading wouldn't change, this has an impact on the way information is displayed. Likewise, it would be nice to assign subcategories for grades (i.e. questions 1-15 are to be part of your "section 1 on the text book grade", whereas questions 16-30 are part of you "section 2 in the text book grade"). Thus a single test (i.e. survey) could provide independent scores for various sections of a test.
Another service teachers would need is the ability to translate point values into grades. For example, in the United States it is common practice to give an "A" to people with 92% of the total possible or higher, and an "A-" for 90-92%, a "B+" for 88-90%, a "B" for 82-88%, etc. In Germany on the other hand, grades are given in the form of numbers 1-6 (with 6 being the lowest grade). We should have a way of taking the compiled score (which would be a total number of points), and translating this into either a percentage or a grade which is defined by that percentage or the total number of points.
Then, there are tools for the teach. A Teacher may want to have an overview of the entire test. What he/she care may care about is "what are the points breakdowns for all students" or "which questions were most frequently missed by students" (this way the teacher could discern if a certain question was possibly written poorly - if for example almost everybody missed a particular question). The best/most visually appealing way to present this information is through bar graphs (which are relatively easy to program), one may want to use a pie chart to show grade breakdowns, but I do not know how we could auto generate this in code. If you have a suggestion please let me know.
Finally, if a teacher were administering an online class... he/she, and or his/her teachers assistants may want to receive notification about a test having been completed. So we should add notification for a survey having been completed. There already appears to be the functionality to notify yourself, but this is not working in postgresql, and it would be good to be able to specify other people as well.
Many websites offer their Users the ability to take health tests, Stan Kaufman (another openacs developer), has written an excellent example of health tests on OpenACS3.x demos of which are available at http://www.cvoutcomes.org. In these surveys one asks a number of questions and then come up with an estimate of risk for certain health concerns. Each question is part of a subsection of the primary survey. So, for example one may ask 10 questions about a persons diet, 10 questions about a person physical activities, 10 questions about a family history of disease, etc. At the end of the survey, it would then respond to these sections and say "it looks like you need to eat a more balanced diet", "you seem to be above average in terms of physical activity so keep it up", and "since your family has a history of concerns with this disease you should be extra cautious and be sure to inform your doctor that they should check for any irregularities". One could also envision a scenerio in which these three responses woudl be given and a summation would also be given, something like "since you are below average in 2 of the 3 categories you are at risk for ________ disease".
Luckily these customized responses and categorization are equal to the "graded surveys" translation from points to grades. I have suggestions for data model modification to accommodate these systems below.
The Admin should have the ability to determine the amount of information returned to the user, and at which point in time this information is returned. In purely multiple choice tests, it would be nice to automatically grade a test, and return the incorrect responses with an explanation of why their response was incorrect. In written tests we would want to take the submission, and then once the teacher has graded the students work, automatically notify the student, either by sending them the teachers responses with grades or sending them an email message encouraging them to come back to the site and look at the grades as well as teachers comments. In Health surveys it would be nice to automatically respond with statements based on scores in subcategories. If one took a personality test (similar to Health survey) would could return statements based on scores in subcategories, but one might also want to respond with a graph that says the average person that took the test on this website had "x" scores in these categories or something similar. None of this would be new in terms of functionality, but it does mean that we need to have a selective way of determining how response are made.
Integration with other packages
Since I am new to OpenACS I am not intimately familiar with the various ways in which packages can be integrated with one another. However, i am certain that the Survey package should not be dependent on other packages like curriculum, or similar. Thus something like a service contract that can be activated and deactivated (possibly though the use of parameters) seems like the way to go. I would like suggestions on the technical work required to implement this.
Data Model Changes
I have thought long and hard about the Data model changes needed, and think i have come up with a simple non-invasive and elegant solution. Please let me know what you think.
In order to facilitate grades we should add the following tables and some columns to preexisting tables.
d survey d surveys Table "surveys" Column | Type | Modifiers --------------------+-------------------------+----------- survey_id | integer | not null name | character varying(4000) | not null description | text | not null description_html_p | boolean | not null enabled_p | boolean | not null single_response_p | boolean | not null editable_p | boolean | not null single_section_p | boolean | not null type | character varying(20) | display_type | character varying(20) | package_id | integer | not null ADD randomize_questions| boolean | default 'f' (this lets you randomize the display of questions from various questions... and bypasses survey_questions sort order and groupings by section. this makes it possible in health surveys to ask 100 questions with each section dispersed throughout the survey, this helps people to not sub- consciously 'cheat' the test by knowing how questions relate to one another and knowing that question 73 actually relates to 74 and thus drawing different conclusions about the implied meaning then the question objectively asks for - this is much more difficult to do if the questions "jump" around in relation to various sections) =============== d survey_sections Column | Type | Modifiers --------------------+-------------------------+----------- section_id | integer | not null survey_id | integer | not null name | character varying(4000) | not null description | text | not null description_html_p | boolean | ADD graded_section | boolean | default 'f' (if this is 't' the default value in survey_questions is 't' otherwise it is 'f') point_translation | boolean | default 'f' (if this is 't' then the system will not just create a sum of points, but will convert that sum into a comment - like a "A" or "B+" or "you need to watch your diet more carefully") ============= Create table survey_point_translation survey_id -integer -references surveys overarching_translation -boolean -default 'f' - (this says if this is to be the the comment that sumarizes sub_section comments) section_id -integer -references survey_sections total_min -integer (the minimum number of points needed for a certain comment) total_max -integer (the maximum number of points permitted for a certain comment) comment | varchar(4000) Then to get a comment we will reference the view survey_grading_section_totals or survey_grading_total_score depending on whether or not it is the "overarching_translation" and do a select spt.comment from survey_point_translation spt, survey_grading_section_totals sgst where sgst.total_score > spt.total_min and sgst.total_score < spt.total_min ============= d survey_questions Column | Type | Modifiers ------------------------+-----------------------+----------- question_id | integer | not null section_id | integer | sort_order | integer | not null question_text | text | not null abstract_data_type | character varying(30) | required_p | boolean | active_p | boolean | presentation_type | character varying(20) | not null presentation_options | character varying(50) | presentation_alignment | character varying(15) | ADD grading_points_p | boolean | default is set by survey_sections (this tells the system whether or not this is one of the questions to be graded - if it is, it will wait for input from the teacher on text and varchar responses but automatically input values from the grading_points field in survey_question_choices) grading_comments_p | boolean | (this tells the system whether or not there comments are permitted - it would then look in the newly created survey_grading_comments table and find the comment to use - or give the Admin a selection) grading_comments_one_response_p | boolean | (this says whether or not there are multiple pre-defined comments which can be selected by the teacher) ============= create table survey_grading_comments comment_id -integer (sequence) question_id -integer (refernces survey_questions) comment_sort_order -integer comment_text -text (This table holds the comments on survey questions for graded surveys, if there is one possible response (i.e. in a multiple choice test, if somebody gets a question wrong there is a standard response giving justification for the answer) - as defined by the grading_comments_one_response_p field in the table survey_questions. If the teacher is grading a short essay test, he/she may want to have 5 standard responses... these would be defined here - and something like a radio button list would let them choose a standard response, define a new standard response or give a response specific to that user/question combo) ============= d survey_question_choices Column | Type | Modifiers ---------------+------------------------+----------- choice_id | integer | not null question_id | integer | not null label | character varying(500) | not null numeric_value | numeric | sort_order | integer ADD grading_points| integer | default null (this would be the number of points a certain selection automatically gets - i.e. if there is only one right answer for a question that one would get a "1" and the rest would get "0") ============== d survey_question_responses Column | Type | Modifiers -------------------+--------------------------+----------- response_id | integer | not null question_id | integer | not null choice_id | integer | boolean_answer | boolean | clob_answer | text | number_answer | numeric | varchar_answer | text | date_answer | timestamptz | attachment_answer | integer | ADD grading_points | integer | default null (this gets filled in from the survey_question_choices grading_points field, but on text and varchar responses, can be manually entered by admin - i.e. when grading a essay response to a test, etc.) grading_comment_id | integer | references survey_grading_comments, default null (this gets filled with pre-defined admin comments -i.e. when responding to a short essay - from the survey) grading_custom_comment | text | default null (if the admin wrote a custom comment applicable only to this one users response, this comment goes here)Since Survey_Sections already exists in the data model, all we need to do is activate this feature (at least in postgresql it is not possible to create mulitple sections for a survey). So categories seem to be taken care of. we will need to create a number of views like this to create graphs, reports and custom responses to surveys:
Create view survey_grading_section_totals SUM of "grading_points" from survey_question_reponses per user per question_id where question_id is part of survey_section. (this would a pretty complicated view - I hope it can be done) ============= Create view survey_grading_total_score SUM of "grading_points" for all subsections of a survey info will be pulled off of the survey_grading_section_subtotals view which is mentioned above. ============
Sophisticated Survey Templates
Given the various ways in which Survey can be used, it would be a good idea to have a number of new templates to deal with this new functionality, and to make it easier to integrate survey into each developers applications. Something like edit-this-page/templates, each of which can be individual selected based on the use of a particular survey is what i had been thinking about.