Weblog

E-Mail: Event Handling

Created by Dave Bauer, last modified by Gustaf Neumann 24 Aug 2020, at 01:43 PM

Sending email on certain events in OpenACS/.LRN is done very haphazardly. This needs to be rewritten so there is a simple way to figure out when an email will be sent, and allow proper handling of user preferences, administrative parameters, and customization.

There are cases where the system (OpenACS or .LRN) needs to send out email, for example, when a new user joins, requests a password reset, or is added to a subsite or .LRN community.

Right now there isn't any system-wide way to mange this email. In some cases, the administrator is notified an email will be sent and is given the option to edit the email before it is sent, but there is no one way this is done.  There are several pages that call  ns_sendmail explicitly, or acs_mail_lite::send explicitly. There are more places this happens in .LRN. Unfortunately there is also a "magic" place where email is sent that is totally unexpected. Inside the dotlrn_community::membership_approve procedure, there is a call to dotlrn_community::send_member_email, which will send an email to the user when the membership is approved, if 1) a parameter is set and 2) an administrator has created and enabled an email message to be sent.

 In addition there is a email sent using the "spam" package in dotlrn/www/admin/users-add-to-community where a dotlrn sitewide administrator can add users to a community, and the users are automatically emailed. In this case the administrator is not notified that an email is sent, or given an opportunity to customize or suppress the email. This causes problems when an administrator attempts to fix a problem by adding a user to a community automatically, and the user is sent a confusing message.

The dotlrn package allows for a custom email to get written by the community admin for each community, but the admin is not allowed to choose if the email is sent when an individual member is added. It is either on or off, always sent, or never sent.

dotlrn-ecommerce extends this by adding several more events for application submission, approval, rejection, etc. And admin can edit these emails on a sitewide or per community basis. In most cases the email is automatically sent, in one o r two cases the admin can edit the email, but not suppress it.

This leads to unwanted email. Often an admin must manually add or remove someone from a subsite or community or otherwise handle a problem. This can lead to welcome emails being sent at the wrong time, confusing the users. 

 A system wide solution would allow packages to create events where email is sent by the system (besides subscribed notifications). This solution would provide an includable interface for creating and editing a default email message for the events. It would also provide an interface to notify an admin that an email will be sent, giving options to suppress the email or edit the content of the email before sending it.

 The beginnings of this feature exist in the dotlrn_member_emails table, dotlrn_community::send_member_email procedure, and the dotlrn-ecommerce package which has a few pages that replicate this interface, but probably needs more generalization to allow working with subsites as well as dotlrn communities. There is also the reusable include for editing the default emails under dotlrn/lib/member-email

This proposal would provide a comprehensive solution for handling system level email events, allowing admins to know when an email is sent, and provide a consistent user interface to manage the emails.

MS:

You could achieve this in a general way by using the acs-lang interface at least for the subject and body. For each object_id you would create a new message key, e.g. acs-translations.welcome_email_subject_${object_id} and acs-translations.welcome_email_body_${object_id}. If you have multiple emails per community / subsite, you would rename them to acs-translations.confirmation_email_subject_${object_id} aso. A general interface would then be provided to look for all language key combinations of acs-translations.email_xxx, allowing you to edit the messages for all communities and subsites. If you want to edit them for only one, then you can look for all who have the same object_id. 

A default message would be given with acs-translations.email_subject_welcome, which the mail sending could default to in case no specific language key exists for the community / object_id. Furthermore, acs-translations.welcome_email_help message key is present describing what this email is about. A package like dotlrn would register the three default e-mail keys to start off with this and then the email-handling package can do the rest (e.g. with email-handler::send -to_party_ids -from_addr -email_type "welcome" -object_id).

This approach has the major advantage that you have internationalization by default.

Alternatively you could mimic the message handling done by contacts which allows you to have multiple message types, e.g. email, which you can fill in with default values, which has it's own I18N by having a locale stored in the DB table. But if you ask me, the acs-translations idea sounds better to me :). Though, you can obviously do this with your own tables as well, but you would loose on the nice features acs-lang has to offer.

 

 Files that currently could trigger email

add_user calls add_user_to_community

add_user_to_community calls membership_approve (if applicable). membership_approve calls send_member_email unconditionaly

so any call to add_user or add_user_to_community could result in a call to send_member_email

dotlrn_community::add_user
dotlrn/www/admin/add-instructor-3.tcl
                 users-add-to-community.tcl
                 member-add-3.tcl
                 members-chunk-table.tcl
                 members.tcl
                 register.tcl

dotlrn-ecommerce/www/admin/gwu-section-new.tcl
                     ecommerce/shopping-cart-add.tcl.backup
                     register/index.tcl                 
                tcl/implementation-procs.tcl
                
dotlrn_community::add_user_to_community
dotlrn/tcl/class-procs.tcl
           club-procs.tcl
           community-procs.tcl
           dotlrn-callback-procs.tcl
           
dotlrn_community::membership_approve
dotlrn/www/approve.tcl
dotlrn/www/admin/commmunity-members-add-to-community.tcl
                 users-add-to-community-email.tcl
dotlrn-ecommerce/www/admin/application-approve.tcl

Security: Cross Site Request Forgery (CSRF)

Created by Gustaf Neumann, last modified by Gustaf Neumann 24 Aug 2020, at 01:41 PM

Starting with OpenACS 5.9.1, OpenACS offers support for protecting against Cross Site Request Forgery (CSRF). In essence, this attack can cause a user’s web browser to perform an unwanted action on a trusted site for which the user is currently authenticated. The user gets a page presented, which looks harmless, but contains links or images that perform actions with the users credentials without the users consent. Note that the CSP does not protect a user against clicks on a malicious link.

CSRF protection works by ensuring that values for an action (e.g. by from a HTML form) are only accepted from a user that has received the form before. OpenACS generates by its security-procs a secure CSRF token value and provides it to a developer it in a global variable ::__csrf_token. When requests secured with the CSRF token are received, it can be validated on the server side. Note, that this mechanism is similar to "signing" values in OpenACS.

CSRF protection concerns of two parts: add the CSRF token to the form (POST requests) or to the href, and checking the received in the queries expecting input from CSRF protected resources. The first part works technically quite similar as securing CSP via nonces. Add code to the Tcl or ADP page that outputs the global variable (the test for the token is mostly for backwards compatibility)

    <form ...>
        ...
        <if @::__csrf_token@ defined>
           <input type="hidden" name="__csrf_token" value="@::__csrf_token;literal@">
        </if>
        ...
    </form>

Secondly, the page contract on the receiving side has to validate the csrf token. This can be achieved by adding a call to csrf::validate to the validation part of a page contract.

ad_page_contract {
    @author ...
    @creation-date ...
} -query {
  ...
} -validate {
   ...
   csrf { csrf::validate }
}

In the code base of OpenACS, CSRF protection was added on several places (e.g. public pages, the list template, etc.) such the checks of OpenACS sites on vulnerability scanners improve. Technically, it would be desirable to secure more places against CSRF attacks in the future. However, it depends on the requirements of a site whether or not e.g. the API browser or search should be CSRF protected. With protection turned on, one cannot share e.g. a link to a search with some other user (or a search engine). A site admin has to decide, how protected/public such links should be.

Security: Content Security Policy (CSP)

Created by Gustaf Neumann, last modified by Gustaf Neumann 24 Aug 2020, at 01:40 PM

Starting with version 5.9.1, OpenACS supports Content Security Policies (CSP), which is a means to secure websites against a range of Cross Side Scripting (XSS) attacks. In short, a CSP allows a developer to deactivate unneeded features in the browser of the client to provide there a sandbox with the minimum required capabilities. The CSP can allow e.g. just to retrieve .js files just form certain sites, or it can disallow script tags within the page, which might be injected by an attacker (for a more detailed introduction and tutorial, see CSP Reference, Google Developer Guide for CSP).

In general, a CSP defines the rules what should be allowed in a page. This could be done static for the whole page, but this means that the CSP rules must allow everything which is needed on a page with the highest requirements (e.g. a page with a richtext editor needs probably a script-src 'unsafe-eval' directive). This could render CSP pretty useless.

Therefore, OpenACS supports a CSP generator, which generates a CSP rule-set for every page dynamically based on the requirements of the page. A web developer can specify the requirements of a page/proc with the command security::csp::require. For example, the current OpenACS theme uses in its plain-master the following directives.

security::csp::require img-src ipv6-test.com

security::csp::require style-src maxcdn.bootstrapcdn.com
security::csp::require script-src maxcdn.bootstrapcdn.com

security::csp::require font-src 'self'
security::csp::require font-src maxcdn.bootstrapcdn.com

Based on the directives of the pages and the directives of the master templates, the security policy of the pages is built (typically in the blank-master). For example, the content security policy of the start page of OpenACS is

default-src 'self';
font-src 'self' maxcdn.bootstrapcdn.com data:;
img-src ipv6-test.com 'self';
report-uri /SYSTEM/csp-collector.tcl;
script-src maxcdn.bootstrapcdn.com 'self' 'nonce-49DBB4A924EA648C3025F7DD8C2553DC0EC700D1';
style-src maxcdn.bootstrapcdn.com 'self' 'unsafe-inline';

With this CSP, openacs.org gets an A+ rating from securityheaders.io.

Deactivating CSP for a Site

Per default, the content security policies are turned on. All packages of the oacs-5-9 branch can be used with the enabled content security policies. However, when a website contains legacy code using JavaScript, for which no content security policies are defined, this will result into non-functioning pages. Therefore, a website administrator can set the package parameter CSPEnabledP (in the package parameters of ACS Kernel in "security" section) to "0" to deactivate the CSP.

For Developers

In order to make old packages (not included in the oacs-5-9 branch) or newly developed packages CSP compliant, one should be aware that all inline code is considered harmful. This includes <script> elements, but also "javascript:" URIs or on* event handlers.

<script> Elements

The CSP guidelines recommend to replace the such elements in favor of JavaScript files obtained from the same source as the page itself. However, this is not always practical, especially, when JavaScript is generated dynamically. In such cases, two approaches are possible to make the script tag acceptable (without allowing all scripts on the page). CSP 2 offers the ability to add nonces or cryptographic hashes to secure this elements. OpenACS supports the first approach.

A nonce value is essentially a one-time value which can't be predicted by an attacker. OpenACS generates by its security-procs such as value and saves it in a global variable ::__csp_nonce. This can be used in the Tcl code or in an ADP page like in the following example:

<script language="JavaScript" 
   type="text/javascript"
   <if @::__csp_nonce@ not nil> nonce="@::__csp_nonce;literal@"</if>
>
...
</script>

Event handlers and "javascript:" URI

Most work are probably changes concerning event handlers (e.g. onclick, onblur, ...) and "javascript:" URIs (having "javascript" in the protocol part of the URI). In general, such code pieces must be refactored (see e.g. 1 or 2 for examples).

OpenACS 5.9.1 offers to ease this process the function template::add_event_listener, which can be used to register event handlers in a compliant fashion either per HTML ID or per CSS class (see cal-item-new.tcl or in forums/lib/message/row2.tcl for examples, how add_event_listener can be used).

How to handle "connection already closed" errors

Created by Gustaf Neumann, last modified by Gustaf Neumann 21 Aug 2020, at 12:47 PM

Newer versions of NaviServer produce by default an error, when a script tries to write a connection, which as already closed by the server (e.g. doing two ad_returndirect in a sequence). Such program code does not make sense and indicates an error that should be fixed.

In general, one can silence the behavior or fix it. To silence the behavior, on can alter the NaviServer config file to silently swallow these error conditions named "rejectalreadyclosedconn". See e.g. Sample configuration file

Recent version of OpenACS (oacs-5-10) have such error conditions already fixed in all the actively maintained packages in the oacs-5-10 branch. In order to fix custom packages, one has to watch out for the following situations:

• Request pages: all after every command that finishes a request a call to "ad_script_abort" is required. Such commands are essentially all "ns_return*" commands and the "ad_*" counterparts.
• Filters: in case, an application handles such commands in filters, the filter has to return with "filter_return" to stop the filter chain (i.e. not to call further filters). Notice that the documentation of ns_register states that the request page is as well handled to trigger e.g. the access log entry. So, in rare conditions, one has to check as well for additional closed cases by using "ns_conn isconnected".

Sometimes, it might be still tricky to find such occurrences. On recent versions of OpenACS one can use the xo* machinery to activate/deactivate Tcl command traces for such commands via ds/shell, like e.g. in:

xo::broadcast send {
  set traced_cmds [info commands ::ns_return*]
  foreach cmd $traced_cmds {trace add execution $cmd  enter {::tcltrace::before}}
}

"xo::broadcast" sends the command to every tcl based thread that executes the provided command. Tracing can be deactivated by replacing "trace add" by "trace remove" in this command.

Plain OpenACS (in oacs-5-10) has as well some support for this via acs-tcl/tcl/tcltrace-init.tcl.

Handling out of memory on "exec" calls

Created by Gustaf Neumann, last modified by Gustaf Neumann 13 Jul 2020, at 02:24 PM

On sites with high number of configured threads and high number of activated packages, the VM size of nsd might become large. One measures is to compile NaviServer and Tcl with the flag SYSTEM_MALLOC, and to run it with a memory efficient malloc library such as TCMalloc (using in the service file LD_PRELOAD=/usr/lib/libtcmalloc.so").

However, even then memory might become short, when "exec" is performed, even via the nsproxy module. The nsproxy module was developed to reduce memory consumption by running a separate process which communicates via pipes to nsd. The nsproxy module is designed to run multiple worker processes (the number can be configured), but when this number runs out (more such worker processes are needed) a new nsproxy worker process has to be created. This happens via a fork() system call under Linux, which can result in an out-of-memory message like the following:

 Error: exec: ns_fork() failed: Cannot allocate memory
 Error: exec failed: not enough memory

What should one do in such cases? In general, the first rule is to reduce the "exec" calls as far as possible, since these are relative slow and resource intensive, and NaviServer/Tcl provide a large number of built-ins or library functions, which should be used if possible.

Secondly, it is preferable to start many nsproxy workers rather soon in the live-time of nsd (e.g. at startup, when it has a small footprint) and ensure that the nsproxy module keeps these worker process alive a long time (by setting "idletimeout" to a high value)

ns_section ns/server/${server}/module/nsproxy {
     # ns_param recvtimeout        5000
     # ns_param waittimeout        1000
     # ns_param idletimeout        300000
     ns_param idletimeout  700000000
 }

When all configured nsproxy worker processes are running all the time, there is not need to fork later, and the error above will not occur anymore. The following snippet shows, how to start all nsproxy worker processes by creatin an ns_job queue with sufficient threads, and start a simple command executed by via nsproxy asynchronously (using the "-detached" flag) in parallel.

set concurrency [ns_proxy configure ExecPool -maxslaves]
if {"q1" ni [ns_job queues]} {
    ns_job create q1 $concurrency
}
#
# Queue a sufficient number of jobs to be executed in parallel
#
time {ns_job queue -detached q1 {exec sleep 1}} [expr {$concurrency * 2}]

Additionally, versions of NaviServer beyond 4.99.20 show via

ns_proxy stats ExecPool

the number of running worker processes of the ExecPool (also included in the process page of nsstats).

How to configure a Network Place under Windows XP to access file-storage via WebDAV

Created by Robert Taylor, last modified by Gustaf Neumann 01 May 2020, at 11:33 AM

Topic:

How to configure a Network Place under Windows XP to access a WebDAV folder in file storage. 

Overview: 

Creating a Network Place will allow you to drag and drop multiple files and even folders from your machine to your OpenACS or .LRN  website. Once you have done these eleven (arguably ten) simple (really!) steps to create a permanent connection on your computer to your file storage location. The Network Place will be accessible through My Network Places.

Once you have established a Network Place to your File Storage documents, you can transfer multiple files, create/delete/edit folders, delete files. Creating and modifying files and folders through the Network Place is the same as you would do using Explorer. You can also open, edit and save files without the hassle of downloading, saving and uploading them on your machine!

Instructions: 

	Step 1: To create a Network Place, open Windows Explorer. Then, right click on "My Network Places" and select Open.
	Step 2: Under Network Tasks, Select "Add a Network Place."
	Step 3: Click "Next" to start the Wizard.
	Step 4: Highlight "Choose Another Network Location" (if it is not already) and click "Next."
	Step 5: Copy the WebDAV URL listed in the file storage folder you want to access and paste it in as the "Internet or Network Address" for your connection.
	Step 6: Select "Yes" when you are prompted with the Security Alert dialog.
	Step 7: Enter the email address you use to log into the site (or username if your site uses usernames) and password in the "Connect to:" dialog box. This establishes your connection to the Network Place.
	Step 8: Create a name for the new Network Place or accept the default and then click "Next."
	Step 9: Leave "Open this network place when I click Finish" checked. Click "Finish" to complete the Wizard.
	Step 10: The first time you do this, you may be prompted again with the Security alert and a login screen. Please say "Yes" to the alert.
	Step 11: You may also need to login again using your site email address and password. (This is a Microsoft thing, not a bug and we can't do anything about it.)
	Voila! Your new Network Place will open and you will see the folders as they appear in file-storage on your site. You can now open Explorer and select, drag and drop multiple files or folders from your machine into SloanSpace. After you do this, refresh your browser to see the files/folders. The Network Place is permanent, so you only need to set it up once and reconnect to it when you need to (through My Network Places.

F. A. Q.

Created by Robert Taylor, last modified by Gustaf Neumann 17 Oct 2017, at 08:40 AM

• What is OpenACS? See en:docs-end-user
• What features does OpenACS have? See en:docs-eng
• How to get admin-level help en:docs-admin-help
• How to get help with OpenACS en:community-culture
• How do I format X in XoWiki? See the documentation: http://alice.wu-wien.ac.at:8000/xowiki-doc/
• How to create a patch to submit via bugtracker? //bugtracker/openacs/patch-submission-instructions

Previous version of FAQ (with many more Q and A):  /faq 

A cookbook of procedures is available at: OpenACS Cookbook 

Refactoring Recipes

Created by Michael Aram, last modified by Michael Aram 04 Jul 2017, at 05:31 PM

The intention of this page is to collect small refactoring snippets that one can/should apply to an existing OpenACS installation in order to improve and modernize its code base. Most of the recipes have already been applied to the official core packages.

Modernize Tcl

1. Refactor foreach {var1.. varN} $list {break} to lassign

   See http://wiki.tcl.tk/1530

   Deprecated code example

   foreach {dog cat} $animalslist break

   Recommended code example

   lassign $animalslist dog cat

   Command line for finding/replacing code

   todo

2. Refactor multiple lindex operations to lassign

   Deprecated code example

   set dog [lindex $animalslist 0]
set cat [lindex $animalslist 1]

   Recommended code example

   lassign $animalslist dog cat

   Command line for finding/replacing code

   todo

3. Brace expr expressions

   Deprecated code example

   [expr $money - 1]

   Recommended code example

   [expr {$money - 1}]

   Command line for finding/replacing code

   todo

4. Replace string equal with eq in expressions

   Deprecated code example

   if {[string equal "" $dog]} {error "I want a dog!"}

   Recommended code example

   if {$dog eq ""} {error "I want a dog!"}

   Command line for finding/replacing code

   todo

5. Replace lsearch with in or ni in expressions

   Deprecated code example

   if {[lsearch -exact $animalslist $dog] != -1 } {error "I dont want a dog!"}

   Recommended code example

   if {$dog in $animalslist} {error "I dont want a dog!"}

   Command line for finding/replacing code

   todo

6. Replace eval with {*} if possible

   Deprecated code example

   eval mycommand $args

   Recommended code example

   mycommand {*}$args

   Command line for finding/replacing code

   todo

Best Practices

1. Use bind variables in SQL statements

   Deprecated code example

   db_string check "SELECT * FROM animals WHERE color = $color;"

   Recommended code example

   db_string check "SELECT * FROM animals WHERE color = :color;"

   Command line for finding/replacing code

   todo

2. Use util::http instead of util_http*, ns_httpget, ::http, ::xo::HttpRequest

Substitute Deprecated Procedures

1. Replace empty_string_p with eq ""

   Deprecated code example

   if {[empty_string_p $dog]} {error "I want a dog!"}
if {![empty_string_p $cat]} {error "I dont want a cat!"}

   Recommended code example

   if {$dog eq ""} {error "I want a dog!"}
if {$cat ne ""} {error "I dont want a cat!"}

   Command line for finding/replacing code

   todo

   Rationale

   Byte-compiled comparisons are faster.

2. Replace exists_and_not_null with info exists and ne

   Deprecated code example

   if {[exists_and_not_null cat]} {error "I dont want a cat!"}

   Recommended code example

   if {[info exists cat] && $cat ne "" } {error "I dont want a cat!"}

   Command line for finding/replacing code

   todo

   Rationale

   Byte-compiled comparisons are faster.

3. Replace ad_parameter with parameter::get

   Deprecated code example

   ad_parameter -package_id 123 SystemURL ""

   Recommended code example

   parameter::get -package_id 123 SystemURL -default ""

   Command line for finding/replacing code

   todo

4. Replace ad_require_permission with permission::require_permission

   Deprecated code example

   ad_require_permission $oid "read"

   Recommended code example

   permission::require_permission -object_id $oid -privilege "read"

   Command line for finding/replacing code

   fgrep -rl "ad_require_permission" packages/ | xargs sed -i.sedbak -r 's/ad_require_permission\s+([^\s]*)\s+([^\s]*)/permission::require_permission -object_id \1 -privilege \2/g'

5. Replace util_unlist with lassign

   Deprecated code example

   util_unlist $animalslist dog cat

   Recommended code example

   lassign $animalslist dog cat

   Command line for finding/replacing code

   todo

Modifying the look of an installation

Created by Ryan Gallimore, last modified by Benjamin Brink 30 Jun 2017, at 05:23 PM

Please refer to OACS Theming for newer templating information & tutorial.

Template Files 

Almost all pages on an OpenACS site use ACS Templating, and so their appearance is driven by a layer of different files. Let's examine how this works:

• A templated page uses an ADP/Tcl pair. The first line in the ADP file is usually:

  <master>

  If it appears exactly like this, without any arguments, the template processor uses default-master for that subsite. For pages in /var/lib/aolserver/$OPENACS_SERVICE_NAME/www, this is /var/lib/aolserver/$OPENACS_SERVICE_NAME/www/default-master.adp and the associated .tcl file.

• The default-master is itself a normal ADP page. It draws the subsite navigation elements and invokes site-master (/var/lib/aolserver/$OPENACS_SERVICE_NAME/www/site-master.adp and .tcl)

• The site-master draws site-wide navigation elements and invokes blank-master (/var/lib/aolserver/$OPENACS_SERVICE_NAME/www/blank-master.adp and .tcl).

• Blank-master does HTML housekeeping and provides a framework for special sitewide navigation "meta" elements such as Translator widgets and Admin widgets.

Figure 1. Site Templates

CSS Files

 /packages/acs-subsite/www/resources/site-master.css contains styles for the following elements:

• Header
• User Bar
• Footer
• Navigation
• Widgets (Buttons)
• Text Styles
• Portlets
• acs-developer-support (along with /packages/acs-developer-support/www/resources/acs-developer-support.css)

 /packages/acs-subsite/www/resources/default-master.css defines styles for the following elements:

• Table, TR, TD
• Calendar package
• combo boxes (for date dropdowns)

/packages/acs-templating/www/resources contains many other css stylesheets that are appropriately named. Each package may also often have its own stylesheet.

Selva Theme 

Selva simplifies the work of customizing the look and feel of your OpenACS/dotLRN website. You will find Selva's documentation on each installation of openacs, once the package is installed, you will find its documentation as follows: http://yoursite/doc/theme-selva.

Since Selva is based purely on css you can have a better accessibility on designing your site.

Steps in placing logo using Selva 

• Go to http://yoursite/admin/site-map/
• Search on theme-selva package and Click on its parameter link
• Look for the parameter named " logoUrl" and place the the filename or URL of the Logo that you want to put in.
• Go to http://yoursite/admin/site-map/, look for "Main Site" and click on parameters, then set the DefaultMaster parameter to  /packages/theme-selva/www/selva-master

Note: This instruction assumed that you have already installed the package theme-selva in your openacs installation.

XoWiki: List of the available includelets

Created by Gustaf Neumann, last modified by Benjamin Brink 30 Jun 2017, at 07:11 AM

This is a list of the includelets, which can be included in a xowiki page: