Package Developer's Guide to Workflow

Workflow Documentation : Package Developer's Guide

By Lars Pind

Introduction

The workflow package manages a collaborative process surrounding some object.

Examples of the object could be a bug in a bug tracker, a ticket in a ticket tracker, a content item in a content management system, a user contribution in a moderated forum/comments/whatever application, a user's request for particpation in a course/event/whatever.

For example, when a new bug is submitted, someone's assigned to fix it, and whoever submitted it is assigned to verify the fix and close the bug. Once the bug's fixed, the submitter will get notified, and the bug will wait in the 'resolved' state until the submitter has verified and then closed the bug.

In order to make use of workflow in your own application, here are the things you need to consider:

Define your default process. The idea typically is to allow your end users to modify the process to suit their needs, but you'll want eto provide a process which they can use as a starting point.
Identify, declare, and implement the callbacks that your application will need.
Write the code to set up the initial process, and to clone that process for each package instance.
Integrate workflow support into your application's API.
Integrate workflow support into your application's user interface.
Integrate workflow into your application's queries

Let's first look at some concepts before getting into the technicalities of how you actually do this. For a working example of building workflow-based applications, we recommend you take a look at bug-tracker.

Workflow Concepts

What's in a workflow

In its broadest, most conceptual sense, a workflow is defined as (and this does get a little hairy, so feel free to skip if you just want to start developing your application):

A number of actions that can be executed (open, edit, comment, resolve, edit, publish, approve, reject, etc.) Whether or not an action is enabled, meaning it can be executed at this point in time, will depend on the current state of the workflow. Executing an action may change the state of the workflow.
A definition of the possible states that the workflow can be in. This could simply be an enumeration of states, such as "Open", "Resolved", "Closed", in which case we're talking about a "Finite State Machine" (because there's a finite number of possible states). Other models, such as petri nets, have an infinite number of possible states. This makes things a lot more complicated, and you don't need to bother since this first implementation of workflow only supports finite state machines.
People participate in a workflow through their role. For a bug tracker, the roles could be "Submitter", "Resolver", and "Tester". For a publication workflow, they'd be "Author", "Editor", and "Publisher". Actions will be to roles, which in turn will be linked to users or groups of users in a particular workflow case. The reason we link people to actions through the concept of roles is to allow you to specify that two or more actions should be done by the same party. In bug-tracker, for example, the same user who opened the bug should also close it.

Finite State Machines (FSMs)

As mentioned, workflow is designed to support workflows based on any model, however the only model currently implemented is the finite state machine.

The workflow API is designed so that whenever you're using features that are specific to finite state machines, the procedure name will include the letters "fsm" in the name, as in workflow::case::fsm::get. That way you'll know when you're hard-wiring a dependency on the particular workflow model.

It's considered good practice to only use non-fsm API calls, but in practice, it can be hard to create good user experiences without. So feel free.

Notation:
[Action] (State) {Role}
(State1) -> [Action1] -> (State2) -> [Action2] -> (State3)

Cases

So much for defining the workflow. When you start "running" your workflow, that's called a workflow case, or simply a case. A case is concerned with a particular object, it's always in a particular state, and it has specific people or groups assigned to its different roles.

In-flow and out-of-flow

When defining actions, we differentiate between in-flow and out-of-flow. In-flow refers to the normal idealized flow of the workflow, out-of-flow are the rest. Concretely what it means is that if you're assigned to an in-flow action, we'll bug you about it through notifications, and potentially get mad at you if you don't come and do something to get the workflow moving along. We don't do that with out-of-flow actions. So we'll send a notification that says "Please come back and resolve this bug", whereas we'll not notify everybody who are allowed to comment saying "Please come back and comment on this bug".

For bug-tracker, the normal flow (in-flow) is this:

(Open) -> [Resolve] -> (Resolved) -> [Close] -> (Closed)

Other actions not in the normal flow are [Edit] and [Comment], which are always enabled, but never change the state. And [Reopen] which throw you back to the (Open) state. And finally [Resolved] is in-flow when in the (Open) state, but out-of-flow when in the (Resolved) state, meaning that you can re-resolve a bug if you need to, but you're not required to.

In-flow and out-of-flow depends on the action, the state, and the user's role in the case. For example, it might be that users in the {Submitter} role are allowed to resolve their own bugs, but the [Resolve] action will still only be considered in-flow to people in the {Assignee} or {Resolver} role.

The Six Steps Conceptually

The recommended way a workflow is linked to an application is this: As part of developing your application, you define your default workflow, which will be used as a template for customization by the users of your applications. This workflow will be installed using the APM after-install callback, and will be linked to your application's package-key.

Then when a new instance of your application is created, your default workflow will be cloned, and the clone linked to the new instance, so that your users can customize the workflow for each instance of your application individually. The default copy installed along with your package is never actually used except for cloning when creating a new instance. This means that your users can customize this default workflow, and the modified version will serve as the boilerplate for all new package instances.

In order to integrate workflow with your application, you'll want to implement one or more of the callback service contracts. These can do things like determine default assignees based on certain data in your application, get information about your application's object for use when sending out notifications, or perform side-effects, such as actually changing the publication state of a content item when you execute the [Publish] action.

When integrating the workflow with your application's user experience, what workflow will give you is essentially the list of actions that the given user can perform on the given object at the given time. In bug-tracker, for example, bug-tracker takes care of displaying the form displaying and editing a bug, while workflow takes care of displaying the buttons that say [Comment], [Edit], [Resolve], [Reopen], [Close], etc., along the bottom of the form. Workflow also has a place to store which form elements should be opened for editing depending on the action being executed.

Your application should typically have an API for creating a new object, editing an object, etc. This application object API will need to be workflow-aware, so when a new object is created, a new workflow case will be started as well. And when the object's edited, that should generally only happen through a workflow action, so that needs to be taken into account as well.

The final step is integrating workflow into your application's queries when you want to filter an object listing/count based on the workflow state. This is the only place where you'll directly be dependent on the workflow data model.

Defining Your Process (FSM)

The current version of workflow only supports workflows based on a finite state machine (FSM). Support for other models, such as petri nets and directed graphs are possible, but not currently designed or implemented.

An FSM-based workflow consists of a set of states, actions, and roles.

You define a new workflow like this:

set spec {
    workflow-short-name {
        ...
        roles {
            role-short-name {
               ...
            }
            ...
        }
        states {
            state-short-name {
               ...
            }
            ...
        }
        actions {
            action-short-name {
               ...
            }
            ...
        }
    }
}

set workflow_id [workflow::fsm::new_from_spec -spec $spec]

All the items (workflow, roles, states, actions) have a short-name, which should be lowercase and use underbar (_) instead of spaces. These are mainly used to refer to the items in other parts of the spec.

The workflow short name can be used to refer to the workflow in your application, which is useful if you have several different workflows. The bug-tracker, for example, could have a workflow for bugs and another one for patches.

Finally, you can also refer states, roles, and actions in your application using short names, although this creates a dependency in your application on a particular form of the workflow, and there's currently no mechanism for ensuring that your workflow contains the states, roles, and actions you'd refer to. This is on the todo-list.

Workflow

These are the attributes you can specify for the workflow itself:

Workflow Attributes
Attribute	Description
`pretty_name`	Name used in the user interface.
`package_key`	The package that defined this workflow.
`object_type`	The parent object type which this workflow can be applied to. If your workflow applies to any object, say 'acs_object'. This is used in relation to callbacks when we build the user interface for defining workflows. More on this in the section on callbacks.
`callbacks`	Callbacks that apply to the whole workflow. If you add side-effect callbacks, these are executed every time any action is executed.
`roles`	Denotes the section of the spec that defines the workflow's roles.
`states`	Denotes the section of the spec that defines the workflow's states.
`actions`	Denotes the section of the spec that defines the workflow's actions.

Internationalization Note:
When we make workflow internationalized for OpenACS 5.0, pretty names will contain message keys in the form "#message-key#". More about this in the package developer's guide to internationalization.

Roles

Attributes for roles:

Role Attributes
Attribute	Description
`pretty_name`	Name used in the user interface.
`callbacks`	Callbacks that define how assignment of this role to users is done.

States

A few typical examples of states:

Examples of States
Application	States
Ticket Tracker	(Open),(Completed), and (Closed)
Bug Tracker	(Open), (Triaged), (Resolved), and (Closed)
Content Management System Publication	(Authored), (Edited), and (Published)
Simple Approval	(Requested), (Approved), and (Rejected)

These are the state attributes in the workflow specification:

State Attributes
Attribute	Description
`pretty_name`	Name used in the user interface.
`hide_fields`	A tcl list of form elements/object attributes that don't make sense in this state. In bug-tracker, the element "Fixed in version" doesn't make sense when the bug is (Open), and thus not yet fixed. It's currently up to your application to do incorporate this into your application.

Actions

Actions are what the workflow allows you to do to your object.

Terminology:

Enabled
The action is allowed to be executed in the workflow's current state.
Allowed
The given user is allowed to execute the action given his current relationship to the workflow case and the object.
Assigned
The same as allowed, but the action is in-flow for this user.
Available
The action is both enabled and allowed for this user.

Some actions will always be enabled. In bug-tracker, for example, we have [Comment] and [Edit] actions, which are always allowed, regardless of whether the bug is (Open), (Resolved), or (Closed).

Other actions, however, will only be enabled in certain states. In bug-tracker, for example, the [Close] action is only available in the (Resolved) state.

Another distinction is that some actions change the state, and others do not. [Comment] and [Edit], for example, do not. [Resolve], [Close], and [Reopen] do. For an FSM, when an action changes the state, you simply specify what the new state should be.

There's a special action called the initial action. This is implicitly executed when a new case is started for this workflow, and must always specify the "new_state" attribute to define which state new cases start out in.

Attributes for actions:

Action Attributes
Attribute	Description
`pretty_name`	Name used in the user interface.
`pretty_past_tense`	This is used in the case log to say "<pretty_past_teense> by <user> on <date>", for example "Resolved by Jeff Davis on April 15, 2003".
`new_state`	The short_name of the state this action puts the case into. Leave out if the action shouldn't change the state.
`initial_action_p`	Say 't' if this is the initial action. Leave out or set to 'f' otherwise.
`allowed_roles`	A list of roles that are allowed but not assigned to perform this action.
`assigned_role`	A single role which is assigned to this action.
`privileges`	A list of privileges. Users who have been granted one of these privileges on the case's object will be allowed to execute this action.
`always_enabled_p`	Say 't' if this action should be enabled regardless of the case's current state. Say 'f' or leave out otherwise.
`enabled_states`	If not always enabled, enumerate the states in which this action is enabled but not assigned.
`assigned_states`	Enumerate the states in which this action is enabled and assigned.
`edit_fields`	A tcl list of fields which should be opened for editing when the user is performing this action. Again, it's up to the application to act on this.
`callbacks`	Side-effect callbacks which are executed when this action is executed.

Putting A Workflow Together

When you put this all together, here's a real live example of what defining a workflow could look like:

ad_proc -private bug_tracker::bug::workflow_create {} {
    Create the 'bug' workflow for bug-tracker
} {
    set spec {
        bug {
            pretty_name "Bug"
            package_key "bug-tracker"
            object_type "bt_bug"
            callbacks { 
                bug-tracker.FormatLogTitle 
                bug-tracker.BugNotificationInfo
            }
            roles {
                submitter {
                    pretty_name "Submitter"
                    callbacks { 
                        workflow.Role_DefaultAssignees_CreationUser
                    }
                }
                assignee {
                    pretty_name "Assignee"
                    callbacks {
                        bug-tracker.ComponentMaintainer
                        bug-tracker.ProjectMaintainer
                        workflow.Role_PickList_CurrentAssignees
                        workflow.Role_AssigneeSubquery_RegisteredUsers
                    }
                }
            }
            states {
                open {
                    pretty_name "Open"
                    hide_fields { resolution fixed_in_version }
                }
                resolved {
                    pretty_name "Resolved"
                }
                closed {
                    pretty_name "Closed"
                }
            }
            actions {
                open {
                    pretty_name "Open"
                    pretty_past_tense "Opened"
                    new_state "open"
                    initial_action_p t
                }
                comment {
                    pretty_name "Comment"
                    pretty_past_tense "Commented"
                    allowed_roles { submitter assignee }
                    privileges { read write }
                    always_enabled_p t
                }
                edit {
                    pretty_name "Edit"
                    pretty_past_tense "Edited"
                    allowed_roles { submitter assignee }
                    privileges { write }
                    always_enabled_p t
                    edit_fields { 
                        component_id 
                        summary 
                        found_in_version
                        role_assignee
                        fix_for_version
                        resolution 
                        fixed_in_version 
                    }
                }
                reassign {
                    pretty_name "Reassign"
                    pretty_past_tense "Reassigned"
                    allowed_role { submitter assignee }
                    privileges { write }
                    enabled_states { resolved }
                    assigned_states { open }
                    edit_fields { role_assignee }
                }
                resolve {
                    pretty_name "Resolve"
                    pretty_past_tense "Resolved"
                    assigned_role "assignee"
                    enabled_states { resolved }
                    assigned_states { open }
                    new_state "resolved"
                    privileges { write }
                    edit_fields { resolution fixed_in_version }
                    callbacks { bug-tracker.CaptureResolutionCode }
                }
                close {
                    pretty_name "Close"
                    pretty_past_tense "Closed"
                    assigned_role "submitter"
                    assigned_states { resolved }
                    new_state "closed"
                    privileges { write }
                }
                reopen {
                    pretty_name "Reopen"
                    pretty_past_tense "Reopened"
                    allowed_roles { submitter }
                    enabled_states { resolved closed }
                    new_state "open"
                    privileges { write }
                }
            }
        }
    }

    set workflow_id [workflow::fsm::new_from_spec -spec $spec]
    
    return $workflow_id
}

Defining Callbacks

There are a number of different types of callbacks, each of which applies to different workflow items (workflows, roles, states, actions). They're all defined as service contracts.

In order to make use of them, your application will need to implement these service contracts, and register the implementation with the relevant workflow item through the 'callbacks' attribute in the spec above.

Here are the types of callbacks defined, how they're used, and the workflow items they apply to.

Service contracts
Service Contract	Applies To	Description
`Workflow.Role_DefaultAssignees`	Roles	Used to get the default assignees for a role. Called for all roles when a case is started. Also called for roles with no assignees, when that role is assigned to an action. Should return a list of party_id's.
`Workflow.Role_AssigneePickList`	Roles	Used when the users wants to reassign a role to populate a drop-down list of the most likely users/groups to assign this role to. Should return less than 25 users/groups. Should return a list of party_id's.
`Workflow.Role_AssigneeSubQuery`	Roles	A subquery used to limit the scope of the user's search for a new assignee for a role. Could typically be used to limit the search to members of a particular group, organizers of a particular event, etc. Should return a subquery ready to be included in the from-clause of a query, which will be used when querying for users, for example '(select * from parties where ...)', (sub-selects must be in parenthesis), or simply 'cc_users' or 'parties'. Defaults to 'cc_users'.
`Workflow.Action_SideEffect`	Workflows, Actions	This is executed whenever the given action is executed. If specified for the workflow, it will be executed whenever any action is executed on the workflow. For details about how to use this in conjunction with log entry data and format log title, see below. Side-effects are executed after the application object has been updated, after the workflow state has been changed, after the log entry has been crated, and roles assigned, but before notifications have been sent out.
`Workflow.ActivityLog_FormatTitle`	Workflows	Used to format the title of the case log. In bug-tracker, this is used to get the resolution code displayed in the case log as "Resolved (Fixed)" or "Resolved (Not Reproducible)". The implementation should return the text string that should go into the parenthesis. The parenthesis are automatically added if the returned string is non-empty.
`Workflow.NotificationInfo`	Workflows	Allows the application to supply information about the case object for the notification. Should return the notification information as a 4-element list containing: url one-line summary details about the object in the form of an array-list with label/value tag for the notification subject (optional). If present, it will be put inside brackets [].

All the service contracts have 3 operations each. The first two are the same for all service contracts, and they really just act like static variables:

Common service contract operations
Operation	Input	Output	Description
GetObjectType	None	object_type:string	Get the object type for which this implementation is valid. If your implementation is valid for any object, return 'acs_object', otherwise return the object type.
GetPrettyName	None	pretty_name:string	Get the pretty name of this implementation. This will be used in the user interface to let the workflow designer pick which implementation to use.

The third operation is the one that does the real work. Here are the inputs and outputs:

Specific service contract operations
Contract	Operation	Input	Output
Workflow.Role_DefaultAssignees	GetAssignees	case_id:integer object_id:integer role_id:integer	party_ids:integer,multiple
Workflow.Role_AssigneePickList	GetPickList	case_id:integer object_id:integer role_id:integer	party_ids:integer,multiple
Workflow.Role_AssigneeSubQuery	GetSubquery	case_id:integer object_id:integer role_id:integer	subquery:string
Workflow.Action_SideEffect	DoSideEffect	case_id:integer object_id:integer action_id:integer entry_id:integer	None
Workflow.ActivityLog_FormatTitle	GetTitle	case_id:integer object_id:integer action_id:integer entry_id:integer data_arraylist:string,multiple	title:string
Workflow.NotificationInfo	GetNotificationInfo	case_id:integer object_id:integer	info:string,multiple

For the most up-to-date information about the service contracts, your safest bet is to refer to the user-visible pages of the acs-service-contract package itself, which will let you view your currently installed contracts and implementations.

Log Entry Data and Log Entry Titles

One noteworthy thing that side-effects can be used for, is to record information about a log entry for use later in displaying a more detailed log entry title, or can be used to e.g. tie a workflow log entry to a particular content repository revision, etc.

Using workflow::case::add_log_data, you can add arbitrary key/value pairs to a log entry. These can the be retrieved later using workflow::case::get_log_data_by_key, and workflow::case::get_log_data.

Installing and Instantiating (APM Tcl Callbacks)

Here are the workflow-related operations that you'll typically want your application to do from the APM Tcl Callbacks:

after-install

Register service contract implementations
Create default workflow (workflow::fsm::new_from_spec)

before-uninstall

Delete default workflow (workflow::delete)
Unregister service contract implementations

before-upgrade

Add new service contract implementations
Add new workflows
Make changes to existing default workflows (if the installed version is not modified) (this isn't yet supported on the workflow API)

after-instantiate

Clone default workflow to create a new workflow attached to the instance (workflow::fsm::clone)

before-uninstantiate

Delete the workflow attached to the instance (workflow::delete)

To see what this could look like in practice, check out packages/bug-tracker/tcl/install-procs.tcl.

Integrating With Your Application's API

Newer applications will define a namespace for each of the objects it defines, which will contain procs like "get", "new", "delete", etc., to manipulate these objects.

Given such a setup, here are the procs that you want to integrate workflow into for your workflow-integrated objects. For a real-life example, see packages/bug-tracker/tcl/bug-procs.tcl and search for "workflow::".

get

In addition to your application's object data, you'll want to call workflow::case::get_id to get the case_id for your object, and then either workflow::case::get or workflow::case::fsm::get in order to get state information from workflow to include in the data set returned by your API proc.

new

When creating a new object, you should also start a new workflow case for that object using workflow::case::new.

edit

Editing an object should only happen through a workflow action, if you want to have a complete audit log (workflow case log). Thus, your edit proc should take the following arguments, in addition to the object_id and the array containing the object data:

      -action_id:required
      -comment:required
      -comment_format:required
      {-entry_id {}}

(entry_id is for double-click protection).

First, you should update your application object data as normal. Then you'll probably want to use workflow::case::get_id to find the case_id. If you have assignment integrated in your form, you'll want to call workflow::case::role::assign to pass these on to workflow, and finally you'll say workflow::case::action::execute to execute the action, including state changes, side-effects, and notifications.

Integrating With Your Application's User Interface

Usually, you'll want one page that lists all the objects in your package instance, and another that lets the user view/edit one object, called the object form page. This section is about the object form page, the next section is about the object listing page.

For a real-life example, look at packages/bug-tracker/www/bug.tcl. You may want to have that handy while reading through this.

We're hoping to make some streamlining of both ad_form and workflow to make this form page integration even easier at some point. But no promises.

Use workflow::case::get_id to get the case_id.

If you want buttons along the bottom of the form like bug-tracker, use the new 'action' feature of the form builder. What you do is pass a list of possible actions to the form builder as a list-of-lists with { label value }. These will be displayed as buttons at the bottom of the form.

When one of these buttons are clicked, the form will be in edit-mode, and you can use form get_action to get the value of the action chosen.

So up top, you'll want to ask the form if an action is in progress, and which one it is, by saying set action_id [form get_action form-id]. If no action is in progress this will return the empty string.

Then you should check that this action is currently available to this user by saying workflow::case::action::available_p.

To get the currently available actions so you can offer them to the user, use workflow::case::get_available_actions which will return the action_id's, then workflow::action::get to get the details about each of the actions.

If you're using ad_form, and you want only one assignee per role, and you want assignment integrated with your form, use workflow::case::role::add_assignee_widgets to add the widgets. It'll do an ad_form -extend, so they'll appear at the point at which you call this proc.

To set the editable fields as defined for the action, do this:

if { ![empty_string_p $action_id] } {
    foreach field [workflow::action::get_element -action_id $action_id -element edit_fields] { 
        element set_properties bug $field -mode edit 
    }
}

Similarly, on submit, you'll want to set the value of the editable fields.

To populate values of the assignee widgets, use workflow::case::role::set_assignee_values in your on_request block.

To add the case log to the comment field, use workflow::case::get_activity_html and feed it to the before_html property of a textarea.

Integrating With Your Application's Queries

Here's an example of how the bug-tracker integrates with workflow for information about the current state of bugs.

select b.bug_id,
       ...

       st.pretty_name as pretty_state,
       st.short_name as state_short_name,
       st.state_id,
       st.hide_fields,

       assignee.party_id as assignee_party_id,
       assignee.email as assignee_email,
       assignee.name as assignee_name

from   bt_bugs b,
       workflow_cases cas left outer join
       (select rpm.case_id,
               p.party_id,
               p.email,
               acs_object__name(p.party_id) as name
          from workflow_case_role_party_map rpm,
               parties p
         where rpm.role_id = :action_role
           and p.party_id = rpm.party_id
         ) assignee on (cas.case_id = assignee.case_id),
       workflow_case_fsm cfsm,
       workflow_fsm_states st 
where  cas.workflow_id = :workflow_id
and    cas.object_id = b.bug_id
and    cfsm.case_id = cas.case_id
and    st.state_id = cfsm.current_state
and    b.project_id = :package_id
order  by $order_by_clause

Note the outer join to get the assignee(s). The joins to get information about the current state is straight-forward.

Good Luck!

That's all I think you'll need to know to start developing workflow-enabled applications.

Let me know how it goes, or of something's missing, by posting on the OpenACS Forums.

lars@pinds.com