Introduction

The Assessment Package unites the work and needs of various members of the OpenACS community for data collection functionality within the OpenACS framework. We're using the term "Assessment" instead of "Survey" or "Questionnaire" (or "Case Report Form" aka CRF, the term used in clinical trials) because it is a term used by IMS and because it connotes the more generic nature of the data collection system we're focusing on.

There has been considerable recent interest in expanding the capabilities of generic data collection packages within OpenACS. Identified applications include:

• Educational settings. The dotLRN project has updated the Simple-Survey package to the Survey package now in the current distribution. A number of groups in the OpenACS community are interested in adding capabilities defined in the IMS Global Learning Consortium's specs for Question and Test Interoperability and Simple Sequencing.
• Clinical research settings. The Epimetrics Group has created an enhanced version of the Simple-Survey package that adds a variety of scoring and scheduling tools for use in health-related quality-of-life assessments. This Questionnaire package has not been ported to OpenACS 4.x yet, however, and it also lacks a wide variety of other features that are necessary for use in formal clinical trial data collection applications, certainly for those that intend to create data sets acceptable for new drug applications to the US Food and Drug Administration and equivalent European regulatory agencies.

  Of note, there are large and well-funded vendors of clinical trials data management systems. Phase Forward, Outcome Sciences, and PHT Corporation among others. A standards body called CDISC (Clinical Data Interchange Standards Consortium) formed a few years ago and is developing data models for clinical trials data derived from schema contributed primarily by Phase Forward and PHT. These vendors provide "electronic data capture" (EDC) services at considerable cost -- a 18 month study of 2500 patients including about 500 data elements costs nearly $500,000. There is clearly interest and opportunity to craft systems that bring such costs "in house" for organizations doing clinical research.

• Data collection services for other OpenACS packages. Most other OpenACS packages invoke some form of data collection from users. While developments such as ad_form and the templating system in OpenACS 4.x ease the construction of data collection forms, it may be possible to expose a focused data collection package via acs_service_contract mechanisms to other packages. In particular, incorporating Workflow and a new data collection package would be key to creation of new vertical-application tools like dotWRK. Such integration would also be immensely useful for a clinical trials management toolkit.

Historical Considerations (Work Done So Far)

Several OpenACS efforts form the context for any future work. These include:

• Survey. This package (largely written/revised by Dave Bauer) doesn't currently have any documentation in the documentation section of the OpenACS.org site, but it is in any current OpenACS installation at /doc/survey/. Dave has added internationalization capabilities (in the version of Survey in CVS HEAD) and cleaned up the administrative UIs very nicely. This package was thoroughly debugged prior to the 4.6.1 release. It supports simple one-section surveys, though the data model has as-yet unimplemented provisions for multiple sections within a survey.
• Exam. This package (written by Ernie Ghiglione and Malte Sussdorff) is currently an Oracle-only tool with capabilities not much different from Survey.
• Surveys. This package was written a while ago by Buddy Dennis, and the source code package has dropped from view. However, we've posted it here. Presumably this package has been further developed, since it appears to be in use at the iQ&A site, though current source doesn't appear to be available there. Surveys included several important enhancements to the data model:
  • Conditional branching within a survey (though how well worked out this is remains unclear)
  • "Folder" based repositories of questions and sections

  However, Surveys has some important limitations:

  • Surveys are "published" as static HTML files which are served out to users when they complete the survey
  • The package doesn't use a templating system
  • Oracle-only

  Still, this package adopts some naming conventions consistent with the IMS spec and definitely represents the closest effort to a "complex survey" done to date.

• "Complex Survey". This is the descendant of "Survey" and Buddy's "Surveys" written by Malte Sussdorf. It currently is in the /contrib branch of the OpenACS 5 distro and represents the currently most advanced package for OpenACS 5+. If you want to start looking at surveys in OpenACS right now, this is the package to get. It incorporates a number of the features of Surveys. We discuss it in greater detail here.
• Questionnaire. This is a 3.2.5 module developed by Stan Kaufman at The Epimetrics Group in order to support complex scoring of a particular type of clinical measure. (You can see a demo of this here, and if you register at the site and join the Bay Area OpenACS Users Group, you can play with the intuitive administrative pages for creating and editing questionnaires, defining scoring mechanisms, setting up user scheduling and reminder features, and configuring results reporting/graphing capabilities.) This module runs within OpenACS 3.2.5, though, and will need a substantial rewrite to work within the new 5.x infrastructure.
• Simple-survey. This package remains in the OpenACS distribution but it is now obsolete, supplanted by Survey

Competitive Analysis

The number of competing products in this area is *huge*. Starting with the usual suspects Blackboard and WebCT you can go on to clinical trial software like Oracle Clinical or specialised survey systems. When writing the specifications we tried to incorporate as many ideas as possible from the various systems we had a look at and use that experience. A detailed analysis would be too much for the moment.

Functional Requirements

An overview of the functional requirements can be found here . It is highly encouraged to be read first, as it contains the use cases along with a global overview of the functionality contained within assessment. Additional requirements can be found in the specific pages for the user interface.

Design Tradeoffs

The assessment system has been designed with a large flexibility and reuse of existing functionality in mind. This might result in larger complexity for simple uses (e.g. a plain poll system on it's own will be more performant than running a poll through assessment), but provides the chance to maintain one code base for all these separate modules.

API

The API will be defined during the development phase.

Data model

The data model is described in detail in the design descriptions .

User Interface

The UI for Assessment divides into a number of primary functional areas, with the entry page located here . It is split up into multiple sections:

• Assessment Authoring: all the pages involved in creating, editing, and deleting the Assessments themselves
• Section Authoring: all the pages involved in creating, editing, and deleting the Sections themselves. Includes the page to browse for items to include in sections
• Item Authoring and Catalogue: all the pages involving the item creation and the item catalogue.
• Assessment Delivery: all the pages involved in deploying a given Assessment to users for completion, processing those results, etc; these are user pages
• Section on Tests: Currently still split away, some notes on additional user interface for test. Shall be integrated with the rest of the pages.
• Assessment Review: all the pages involved in select data extracts and displaying them in whatever formats indicated; this includes "grading" of an Assessment -- a special case of data review; these are admin pages, though there also needs to be some access to data displays for general users as well (eg for anonymous surveys etc). Also, this is where mechanisms that return information to "client" packages that embed an Assessment would run.
• Session Management: pages that set up the timing and other "policies" of an Assessment. This area needs to interact with the next one in some fashion, though exactly how this occurs needs to be further thought through, depending on where the Site Management mechanisms reside.
• Triggers and Action Execution

The  Page Flow page is diagrammed below and should give a very rough and outdated overview, but still good for getting an impression.

Authors

The specifications for the assessment system have been written by Stan Kaufman and Malte Sussdorff with help from numerous people within and outside the OpenACS community.