Weblog

OpenACS Monitoring

Created by Malte Sussdorff, last modified by Gustaf Neumann 24 Nov 2023, at 06:48 PM

When running an OpenACS site, a couple of packages come in handy to monitor your site and see why and where bottlenecks are located. To use them,  you need to install NaviServer with libthread support. If you are lazy as I am, just download the installation script from naviserver-openacs and get started. This will make sure that you have all the necessary ingredients installed.

Then go to your systems' administration at /acs-admin and install two packages:

• xotcl-request-processor
• monitoring

The first one gives you an overview over the system performance, while the latter allows you to scan the error.log for errors and have them send to you via e-mail.  A little bit more detail:

Monitoring

The monitoring package has a couple of nice things which help you monitor your website. First and foremost, you can use it to get the error logs. This is HIGHLY recommended, as your users will not report errors to you. They will just complain internally and not use your site anymore. We learned this the hard way.

Once you have monitoring installed (and mounted at e.g. /monitoring) go there and edit the parameters. There you can define who should get the error reports send and how often they are sent out. NOTE: They will be sent out with every restart of the server, so this gives you a pretty good idea if your server has restarted.

You can also get the TOP reports (if you have a high load on the system), which is something I usually don't use, but maybe it is of importance for you.

Last but not least (for me at least) you can view which scheduled procs are running on your server and when they are executed next time. This is really helpful to understand what is happening on your site of the things you cannot see.

XOTcl Request Monitor

The request monitor (automatically mounted under /request-monitor) will give you a performance overview of your site, how many users are online, what are the average page load times ....

If you look at the overview page, you will first be delighted to see that the average times are so low. This is misleading, as the quick fetches to ".css" files and so on are counted into the equation as well.

More accurate is the aggregated stats view, which allows you to see which URLs are called most often on your site and how expensive they are to load. It is a good idea to have the most often called pages be *fast*.

One thing we do as well is to run the "last 100 request" page on a continuous basis in a browser window. It refreshes every 60 seconds and if you get the sorting right you can actually get a good overview of the activity on your site (e.g. we order by execution time, so we always see how long users are waiting for their pages and if this number goes up considerably, well, then we know we have to act!).

Chat

Created by Emmanuelle Raffenne, last modified by Gustaf Neumann 24 Nov 2023, at 06:41 PM

Package Specification Summary for Package: chat

Bug Tracker Summary for Package: chat

Code Metrics Summary for Package: chat

The chat package is currently using two different chat engines:

1. The Java applet and the Java chat server, or
2. the Ajax based chat class from xotcl-core

If no xotcl-core is installed, one can only use the version with Java applet. If xotcl-core is installed, one has to option to choose between both engines via parameter.

In essence, the Ajax based variant is the chat implementation from xowiki plus chat room management and a list of the currently connected users. Originally, the AJAX-based chat class (::xo::Chat) was developed for xowiki, but later moved to xotcl-core such that the chat package does not need a dependency on xotcl-core and xowiki.

Actually, the chat support in xowiki contains a different variety of AJAX based communication strategies:

1. Polling mode (the web browser queries via AJAX in definable intervals the server, and checks, if there is some new information available), and
2. Streaming mode (the client opens a connection to the server, the server pushes the information back to the client. xowiki chat supports two different sub-modes:
   1. streaming of JSON objects via asynchronous AJAX sockets, or
   2. scripted streaming, where the browser loads HTML with embedded script tags in the background from a hidden iframe

Both (2a) and (2b) require libthread installed and NaviServer. However, the advantage of the streaming approaches is that they are much faster in the look and feel and that they are better scalable on sites with a high load. 2a works only on Firefox (not in current versions of IE or Safari) and has the advantage over 2b that the browser does not show the background activity (no spinning wheel in Firefox). 2b is more robust and works with Safari, IE and Firefox (the information about the browser support is from 2006, and most probably different today). Both streaming modes implement, what is sometimes called COMET.

When the chat package uses the ::xo::Chat, it uses it only in currently only polling mode, which has the least requirements. This has been as well tested with (Firefox, IE and Safari).

XoWiki

Created by Gustaf Neumann, last modified by Gustaf Neumann 24 Nov 2023, at 06:36 PM

XoWiki is a wiki implementation for OpenACS implemented based on xotcl-core. Instead of trying to implement the full set of wiki markup commands in systems like MediaWiki, XoWiki is based on a rich text editor and focuses more on integration with OpenACS (e.g. categories, general comments, adp-includes, ad-substitution of template variables). XoWiki combines aspects of wikis (ease of page-creation) with aspects of a content management system (revisions, reusable content, multiple languages, page templates).

XoWiki provides functionalities of enterprise wiki systems (such as combining structured with unstructured information, providing security policies, etc.). XoWiki is reused in several other packages such as XoWiki Content flow, S5 or  the Learning Content Tool.

Documentation:

• Latest version of the XoWiki Documentation
• XoWiki FAQ
• XoWiki Tutorial (XoWiki as a framework)
• ::xowiki::Form examples (Import by XoWiki 'Import' feature in Admin UI)
• Solution Grove's xinha and xowiki howtos

Package Specification Summary for Package: xowiki

Bug Tracker Summary for Package: xowiki

Code Metrics Summary for Package: xowiki

Available Includelets

Available Formfield Classes

XOTcl Core

Created by Gustaf Neumann, last modified by Gustaf Neumann 24 Nov 2023, at 06:31 PM

What is xotcl-core

xotcl-core is an OpenACS package, which provides base functionality for OpenACS applications using XOTcl. The xotcl-core package is used by several other OpenACS packages such as the XOTcl request monitor, XoWiki, XoWiki Content Flow, S5 or the Learning Content Tool.

Install xotcl-core

First, get xotcl-core from the OpenACS repository:

Move xotcl-core to the directory, where your OpenACS packages are stored. We assume, this directory is /usr/local/openacs-4/packages
 

 Next, go to acs-admin/apm (http://YOURSERVER/acs-admin/apm/), choose "install-packages" and install XOTcl Core. Finally, restart the server.

Now you can browse  http://YOURSERVER/xotcl

Functionality of xotcl-core

xotcl-core defines several classes for

• library dependencies
• package management
• debugging
• security and policy management
• HTTP client and server functionality
• Object Relational Database Interface
• ical formatting
• cluster management
• chat handlers

and several utilities.

Tutorial xotcl-core

 See slides

Code metrics and Package Description

Package Specification Summary for Package: xotcl-core

Bug Tracker Summary for Package: xotcl-core

Code Metrics Summary for Package: xotcl-core

The OpenACS package xotcl-core requires in versions of OpenACS before 5.3 the following patches for the documentation procs and the bootstrap installer:

To apply these patches, go to the directory, where the OpenACS packages are stored. We assume, this directory is /usr/local/openacs-4/packages. 
 

Both patches do not harm in case XOTcl is not installed. Now reload these packages or restart the server.

for beginning developers

Created by OpenACS community, last modified by Antonio Pisano 03 Nov 2023, at 11:55 AM

Tutorials for OpenACS development

• Learn the pieces gradually and learn to read the docs so you can eventually Become Expert in the basics!
• Paper: OpenACS: robust web development framework (PDF)
• OpenACS Development Tutorial
• Emacs as an OpenACS IDE
• Vim as an OpenACS IDE
• Getting admin-level help
• OpenACS for administrators
• Prerequisites to installing OpenACS
• Coding Standards - Index
• en:ajax-accessibility
• Getting WAI-AA compliance from an unzenned package
• OpenACS "cookbook", how-to tutorials
• XOTcl for OpenACS - An Introduction to XOTcl and the basic infrastructure of xotcl-core
• Learning XoWiki - A Tutorial for the XoWiki Toolkit
• Starting an OpenACS application based on XoWiki
• Webinar - Part 1 - Basics
• Webinar - Part 2 - Site Map
• Webinar - Part 3 - Packages and ad_form
• Webinar - Part 4 - Survey
• Jade Rubick's Article Collection

Other related tutorials

• Tcl for Web Nerds
• A tDOM Tutorial
• SQL for Web Nerds
• Software Engineering for Internet Applications
• Philip and Alex's Guide to Web Publishing
• Anti-patterns, patterns to avoid in programming and managing systems

See also: OpenACS Mentorship Program

Other useful resources

Web Sites

• bash shell

• The Linux Documentation Project

• Introduction to Oracle SQL Course

Books

• UNIX Power Tools - An excellent introduction to the command line tools and basic programs of UNIX

• UNIX System Administration Handbook (formerly the "red book" - now the "purple" book)

• UNIX System Administrator's Bible - (LePage and Iarerra 1998; IDG)

• Running Linux

• Learning Gnu Emacs

• Linux in a Nutshell

Installing OpenACS

Created by OpenACS community, last modified by Gustaf Neumann 24 Oct 2023, at 01:45 PM

There are many ways to get OpenACS working for you quickly and/or easily. See Try OpenACS for demonstrations and hosting solutions. In general, OpenACS can be installed with NaviServer or with AOLserver, which are two closely related servers. While NaviServer sees regular development, the AOLserver community is very conservative. Some of the guides below refer to AOLserver, some to NaviServer.

Packaged installations

For some platforms, a packaged version of OpenACS is available:

• Windows - Installing OpenACS on Windows
• Arch Linux - Installing OpenACS on Arch Linux
• Docker - Docker installation

Generic installation scripts

For many Linux platforms (e.g. Ubuntu, Debian, Fedora), one can use the generic installer that compiles all base components (using NaviServer) and creates users/groups as needed; which works with PostgreSQL 9.2 or newer. These install scripts can also be used on Mac OS X, when MacPorts are installed. These installer scripts are regularly updated.

• See: naviserver-openacs

The following alternative script installs AOLserver and the contained modules from sources. Using AOLserver for new OpenACS installations is NOT RECOMMENDED (last release 2011), but could be useful or necessary in certain contexts. It assumes, that PostgreSQL is already installed:

1. Install AOLserver: https://openacs.org/storage/view/aolserver/install.tgz
2. Install OpenACS: en:openacs-subsystem-install

Manually installing OpenACS:

In a productive environment, one may need to customize the installation further manually according to the needs, since the configuration options are fairly large, and only the common parts are covered by these scripts. We recommend using the script as reference in this case and adapt the installation steps according to your needs.

OpenACS 5.10.1 Change Summary

Created by Gustaf Neumann, last modified by Gustaf Neumann 24 Oct 2023, at 10:04 AM

• Security:
  • Stronger password hashes for OpenACS
    • New password hashes in addition to the classical "salted-sha1"
      • "scram-sha-256": SCRAM (RFC 7677) with parameter recommendation from RFC 7677; requires OpenSSL 1.1.1 and NaviServer 4.99.23 or newer
      • "scrypt-16384-8-1": SCRYPT (RFC 7914) with parameter "-n 16384 -r 8 -p 1"; requires OpenSSL 3.0 and NaviServer 4.99.23 or newer
      • "argon2-argon2-12288-3-1": Argon2 (RFC9106), Parameterization recommendation from OWASP: m=12288 (12 MiB), t=3, p=1; requires OpenSSL 3.2 and NaviServer 5.0 or newer
      • "argon2-rfc9106-high-mem": Argon2 (RFC9106), first (memory intense) recommendation from RFC 9106; requires OpenSSL 3.2 and NaviServer 5.0 or newer
      • "argon2-rfc9106-low-mem": Argon2 (RFC9106), second recommendation from RFC 9106; requires OpenSSL 3.2 and NaviServer 5.0 or newer
    • Preferences of the password hash algorithms can be set via kernel package parameter "PasswordHashAlgorithm", the first available algorithm is taken from the preference list, hash re-coding happens automatically at the next login.
    • See https://openacs.org/forums/message-view?message_id=5537869
       
    • Added optional CSP rules based on MIME types. This is important for user-contributed content. When users upload e.g. SVG-files to the file storage, and the content is served from there, it poses a potential security hole. One can now define an additional parameter called "StaticCSP" in the section "ns/server/$server/acs" of the OpenACS configuration file to deactivate execution of script files from static content.
      
      ns_param StaticCSP {
    image/svg+xml "script-src 'none'"
}
       
    • Cookie-Namespace: When multiple OpenACS instances are served from the same domain name, the same cookies (e.g. ad_session_id, ad_login, ...) are set to all servers. For sensible cases, a cookie-namespace can be used, which can be used as a replacement of the traditional "ad_" prefix. This can be as well set in the section "ns/server/$server/acs" of the OpenACS configuration file
      
      # Provide optionally a different cookie namespace
# (used for prefixing OpenACS cookies)
ns_param CookieNamespace "ad_"
       
• Improved templating:
  • Client-side double click prevention
  • Support for generic icon names, which can be mapped differently depending on the installed packages and themes: The generic names are supported via <adp:icon name="NAME" title=...>. By using this feature, one can use font-based icons (like e.g. glyphicons of Bootstrap5, bootstrap-icons, fa-icons, ...) instead of the old-style .gif and .png images. This makes the appearance more uniform, has better resizing behavior, and works more efficiently (fewer requests for embedded resources). Most of the occurrences of the old-style images in standard core and non-core packages in oacs-5-10 are already replaced.
  • Support for listing registered URNs
     
• Require NaviServer (i.e. drop AOLserver support).
  Rationale: AOLserver cannot be compiled with the required modules with recent Tcl versions. Trying to backport NaviServer compatibility functions seems to be an overkill for the OpenACS project.
   
• Further reduce divergence between Oracle and Postgres SQL. Target version of Oracle could be 12.*, as Extended support ends in 2022 (see https://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf)
  • limit / rownum -> fetch first
  • use Postgres schemas for stored procedures so that they can be invoked with the same Oracle idiom
     
• Bootstrap 3 reached EOL in 2019, Bootstrap 4 had EOL 2022, so we should migrate to Bootstrap 5 (details: https://github.com/twbs/release)
   
• New Packages:
  • openacs-bootstrap5
  • bootstrap-icons
  • fa-icons
  • highcharts
     
• Potential incompatibility with OpenACS 5.10.0: "permission::permission_p" returns Boolean values as "t" and "f" and not "1" and "0". Avoid literal comparisons of the result and use boolean tests available in Tcl/OpenACS.
   
• Support for fresh installations on Oracle 19c (for details, see: oacs-5-10-on-oracle-19c)
• Require Tcl 8.6.2, XOTcl 2.1, PostgreSQL 12 (PostgreSQL 11 EOL: November 23), tdom 0.9

OpenACS 5.10.0 Change Summary

Created by Gustaf Neumann, last modified by Gustaf Neumann 07 Oct 2023, at 05:43 PM

• Functional improvements
  • Features:
    • Added additional page_contract filter: oneof(red|green|blue)
    • template::add_event_listener and template::add_confirm_handler now can target elements by CSS selector
    • Added text/markdown to the accepted text formats or rich-text widget
    • Support for range types in .xql files:

      PostgreSQL supports range types since 9.5. When using range types, square braces have to be used in SQL statements. Since OpenACS uses always Tcl substitution in .xql files, and OpenACS does NOT allow backslash substitution in these files, square brackets could not be escaped and therefore not be used in .xql files so far. This change allows now a developer to deactivate the substitution by passing e.g. -subst none to the db_* command using the .xql file. Valid values for -subst are all, none, vars, and commands, default is all which is exactly the behavior of previous releases. Therefore, this change is fully backward compatible.

    • Improved API browser: Visualization for code dependencies (which procs calls what, from where is a proc being called) and test-coverage
    • Warn site administrators about expiring certificates
    • Additional input types (and widgets) for ad_form:
      • checkbox_text
      • color
      • email
      • tel
      • url
      • number
      • file (multiple)
      • h5date and h5time: date and time fields using native HTML5 visualization and input normalization
    • Registry for .js and .css libraries: allow besides classical URLs symbolic names for loading external resources (e.g. jquery), this makes it easier to upgrade libraries in multiple packages (without running into problems with duplicate versions) and supports switching between CDN and local pathsURN. The existing implementation is based on URNs and extends the existing template-head API to support registration for URNs. A URN provides an abstraction and a single place for e.g. updating references to external resources when switching between a CDN and a locally stored resource, or when a resource should be updated. Instead of adding e.g. a CDN URL via template::head::add_script, one can add an URN and control its content from a single place. Use common namespaces for OpenACS such as urn:ad:css:* and urn:ad:js:*.

      • Register URNs:

        Example provider (e.g. in some theme):

           template::register_urn \ 
               -urn urn:ad:js:jquery \ 
               -resource /resources/xowiki/jquery/jquery.min.js
        

      • The registered URN can be used like classical URL after registration.

        Example consumer:

           template::head::add_javascript -src urn:ad:js:jquery

      • Declare composite files: Provide an interface to define that a .js file or a .css file contains multiple other .js/.css files in order to reduce the number of requests.

           template::head::includes -container urn:js::style.js -parts {urn:ad:js:jquery ...}

    • Improved support for streaming HTML: The new API function template::collect_body_scripts can be used to get the content of template::script or CSP calls (template::add_body_script, template::add_event_listener, template::add_body_handler, template::add_script) when streaming HTML (incremental HTML) is used. Before, these call could bot be used for streaming HTML.

  • Reforms:
    • Dynamic blueprint reloading:
      • When reloading apm packages, watched files, etc. the NaviServer blueprint is now updated correctly. This solves the long-standing problem that changes for scheduled procedures required a restart of the server. Furthermore, the old-style reloading was based on a ever-growing list of reload operations each time a new reload was requested, causing slow-downs in long running server instances, especially, when new threads are started.
      • Now changes are applied to all NaviServer threads, including threads for schedules procedures, ns_jobs and the like. Note that the current job has to be finished before the changes can be applied.
      • For the transition, it is possible to switch between the classical reloading style and blueprint reloading by changing a variable in acs-tcl/tcl/apm-procs.tcl
    • Login:
      • Get rid of bugging "login page expired" messages. The 17 years old construct was replaced by newer means to avoid caching of form values from the login form. Admins of existing sites should set the kernel parameter LoginPageExpirationTime to 0
    • Forums:
      • Removed hard-coded dependency with registered_users group when checking forum permissions
      • Don't rely so heavily on acs_permissions to model forum configuration, as this can have unexpected consequences in convoluted multi-group/multi-subsite scenarios. Prefer simpler table attributes instead
      • New style of attachments to the forums, allowing multiple attachments to a single message directly from the message post page, using the multiple file input widget. Retain compatibility with old style attachments, using the new 'AttachmentStyle' package instance parameter. Currently, this supports two values: 'simple' (new behavior) and 'complex' previous behavior.
    • Chat:
      • Revamping of the GUI
      • Responsiveness
      • Full screen mode
      • Skins support (minimal, classic and bubbles, included): Skins are located in the new /packages/xowiki/www/resources/chat-skins/ directory. New skins can be created by just adding the css and js files in the skins directory, and naming them accordingly (chat-$SKIN_NAME.{js|css}).
      • Avatars (can be enabled per room)
      • Number of active users in chat
      • Tab notifications of new messages
      • Web Notifications:
        • https://www.w3.org/TR/notifications/
        • https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API
    • acs-lang:
      • admin pages:
        • Added the option to unregister (delete permanently the message key from all locales) a message key that has been already marked as deleted. Useful for cleaning up old message keys.
        • Added the option to undelete, using the new ::message::undelete proc.
        • Made number and category (untranslated/deleted/...) of messages coherent in all pages.
        • Added the columns 'total' and 'deleted' to the index page.
      • object_id reference: it is now possible to associate a message key to an object_id in a way that e.g. when the object is deleted, so is the message key. This addresses cases such as the message keys generated by group creation or by the new XoWiki localized fields
    • Notifications:
      • Improved scalability for notifications: One of the most expensive operations in large site is the cleanup for notification_requests in situations, where the user has lost permissions on an object, on which the user wanted to receive notifications. This check was performed previously in notification::sweep::cleanup_notifications via a permission check over all notification requests, which can be very costly on large sites. This change moves this cleanup into the actual notification sending, where the permissions have to be sent anyhow.
      • When sending a notification on behalf of a person, if the system is not configured to process replies to notification, do not set the reply-to address to anything different than the sender
      • Notifications: proper cleanup of acs_objects resulting from the deletion of dynamic notification requests
    • User/Person/Party API: rework and rationalize caching of all party, person and user API, create separate caches for each of these types, make the API and return dicts. acs_user::get will not fail anymore with non-existing user.
    • User Portrait: created API to retrieve and create, store and delete the user's portrait. Also address leftover child relationships from the past and delete them properly.
• Non-functional Changes
  • Improved automated regression test infrastructure and test coverage
    • All packages in the oacs-5-10 branch pass regression test
    • Web testing was separated from non-maintained tcltest and was built on the standard OpenACS infrastructure
    • Include web testing per default in standard regression testing
    • Introduced new test authentication authority, allowing to run many user administration tests outside the context of a "real authority": in cases where the real authority depends on external services to proof identity, (e.g. Kerberos), those tests would just fail.
    • Introduce the display of warnings in the UI
    • Added test coverage information in the automated testing pages, using the new proc-coverage API and providing test coverage information for packages and system wide.
    • Increased overall coverage of public API
    • New tests checking various data-model properties and smells
       
  • Improved scalability:

    • Provided lock-free implementation of ad_page_contract_filters and ad_page_contract_filter_rules. This change improves parallel processing of requests and is primarily interesting for sites with a few mio page views per days. These locks were among the most frequent nsv locks

    • Reduced locks on util_memoize_cache my more invariants values into per-thread caching (acs_lookup_magic_object, ad_acs_version, .... ) and by avoiding specialized calls, which can be realized by already optimized ones (apm_package_installed_p_not_cached ref-timezones was most frequently used util_memoize_cache entry). These changes are necessary to avoid full still-stand on the unfortunate long-time locks on util_memoize_cache stemming from permission and user management with wild-card flush operations, which require to iterate over all cache entries (which might be on a busy server several hundred thousands)

    • Added new interface for cache partitioning to reduce lock latencies on high load websites

    • Added new interface for lock-free per-thread and per-request caching to avoid scattered ad-hoc implementations

    • Better reuse of DB handles (reduced expiring/reopen/etc.), faster access to handles

  • Improved startup time:
    • When the package acs-automated-testing is disabled, startup time is reduced by avoiding loading of support functions and tests; the size of the blueprint is reduced
    • xowf: loading of at-jobs is significantly improved.
       
  • Security improvements:
    • Strengthened page contracts
    • CSP support for application packages
    • CSP fine tuning
       
  • Better exception handling based on Tcl 8.6 exception handlers (try and throw, also available in Tcl 8.5)
    • Provided a new ad_try implementation based on Tcl's try replaces now the old ad_try, with_catch and with_finally, which are marked as deprecated
    • The new ad_try is in essence Tcl's try but with predefined handling of ad_script_abort and should be also used instead of catch, when the OpenACS API is used (which might use script aborts)
    • All core packages use the new ad_try instead of the deprecated versions.
       
  • Connection close reform:
    • NaviServer/AOLserver continue after connection closing commands to execute a script. This is in many situations not desired, especially, when for the page as well a .adp file exists, which will try to deliver this on the already closed connection. This can lead to errors in the error.log file, which are sometimes hard to analyze
    • Due to this cleanup, developers should use in most such cases cases ad_script_abort
    • Connection closing commands are e.g. ad_returnredirect, ad_redirect_for_registration, cr_write_content, ad_page_contract_handle_datasource_error, ad_return_string_as_file, ad_return_complaint, ad_return_error, ad_return_forbidden, ad_return_warning, ad_return_exception_page, ns_returnredirect, ns_return, ns_returnerror
    • The new version has made on most occasions explicit, when the script should abort.
       
  • API changes (new and extended API calls):
    • New API call category::get to obtain category description for a category_id and locale
    • New utility ad_pad emulating both lpad and rpad typically available in DBMSs
    • New proc lc_content_size_pretty, prettify data size given in bytes. It supports three different standards (SI base-10, IEC base-2 and the old JEDEC base-2), default is SI base-10.
    • New flag -export for ad_form: this flag uses export_vars under the hood and supports all of this API's features (e.g. :multiple, :sign, :array). This addresses a long standing TODO
    • util::pdfinfo: simple poppler-utils wrapper to extract pdf information
    • util::http: leverage new ns_http features such as request file spooling. Native implementation will now be used only on NaviServer >= 4.99.15.
    • Database API:
      • db_foreach: queries executed inside of a db_foreach will not be issued using a different handle and will therefore be safe to use in a transaction
      • db_list_of_lists: new -with_headers flag, which will make the first element of the returned list to be the column names as defined in the query
    • Groups API:
      • Logics to delete a group type have now been included in the API
      • Allow to filter group members by member_state in the API
         
  • Deprecated commands:
    • Many deprecated API calls were included in the code (and sometimes still in use) sometimes more than 10 years after these calls have been deprecated. In case a site modification still uses deprecated code, the user is warned about this. The OpenACS 5.10 code base does not depend on deprecated code.
    • Move deprecated code into separate files
    • Made loading of deprecated code optional (can be controlled via parameter "WithDeprecatedCode" in section "ns_section ns/server/${server}/acs" of the config file. By default, deprecated procs are still loaded
    • When deprecated code is not loaded, the blueprint of the interpreter is smaller. The following number of lines of code can be omitted when loading without the deprecated procs:
      • acs-tcl: 3178
      • acs-templating: 450
      • xotcl-core http-client-procs: 830
      • acs-content-repository: 1717 (including .xql files)
         
  • Bugfix and Code Maintenance:
    • Made sure all party emails are stored as lowercase through the API
    • Fixed long standing regression in template::list: we were looping through the list "elements", rather than the "display_elements". This prevents specifying different sets of columns to be returned depending on the -formats and -selected_format options in template::list::create.
    • acs-content-repository: New HEIC and HEIF mimetypes
    • acs-mail-lite: handle to_addr specified as "DisplayName <email>" without errors
    • Fixed invalidating of all existing user logins, (aka) "Logout from everywhere" feature, useful e.g. to make sure no device still holds a valid login when we change our password on a device
    • Don't lose the return URL when one tries to join a subsite before being logged in
    • Added doc(base_href) and doc(base_target) for setting <base> element via blank-baster (see issue #3435)
    • Groups:
      • When a new group is created, flush all the group::get_id caches with the same name so that the new group can be fetched correctly in case it replaces a previously deleted one
      • Cleanup message keys coming from groups in acs-translations when a group is deleted
    • acs-lang:
      • lang::util::convert_to_i18n: do not always register a en_US translation, which would be always overridden. Instead, let lang::message::register make sure that a en_US message exists and create one only as a fallback.
      • lc_time_fmt: leverage Tcl clock to address shortcomings such as handling of dates in Julian/Gregorian calendar and impossible dates such as 1999-02-29, implement missing formats, support previously undocumented formats explicitly
    • search: make sure objects in the search indexer queue still exist by the time they are swept by the indexer (e.g. items deleted before the indexer could sweep them)
    • attribute::delete: fix proc so it leverages stored procedure capability of dropping the database table as well
    • util::http: fix UTF-8 encoding issues for some cornercases
    • Localization: Complete Italian and Spanish localization for the whole .LRN set of packages (including themes). Message keys for new and previously localized packages have also been updated
       
  • General cleanup/maintenance
    • Improved handling of server implementation-specific code: server-specific code can be optionally loaded via specifying the server family in the filename. Provided *-procs-aolserver.tcl and *-procs-naviserver.tcl similar to *.postgresql.xql and *.oracle.xql where appropriate
    • Modernization of Tcl idioms.
    • Compliance of files, proc names, ... to the naming conventions.
    • White space cleanup, indentation changes.
    • Improvement of public API documentation
    • Adjustment of proc protection levels (public, private)
    • Adjustment of log severity
    • Cleanup of obsolete files
    • Replacement of handcrafted forms by ad_form
    • Typo fixing
    • Editor hints
    • Replacement of deprecated calls
    • Addition of missing contracts
    • ...
       
  • SQL cleanup:
    • Cleanup of obsolete nonportable SQL constructs in a way Oracle and PostgreSQL code base divergency is reduced:
      • "nvl" -> "coalesce"
      • "sysdate" / "now()" -> standard "current_date" or "current_timestamp"
      • Use standard-compliant "dual" table where appropriate (required by Oracle, supported by PostgreSQL)
      • Use non-dialectal cast idioms when appropriate
      • Adopt CTE idioms in Oracle codebase as well (e.g. connect -> with recursive)
      • ... (reference Oracle version will be 11gr2 as is oldest version officially supported by Oracle (See here and here)
    • Reduced superfluous .xql queries
      • acs-subsite: delete 21 files with un-referenced .xql queries
      • acs-tcl: delete 4 files
      • news: 3 files
      • file-storage: 1 file
      • dotlrn: 9 files
         
  • New Packages:
    • cookie-consent: alerting users about the use of cookies on a website
    • boomerang: performance of your website from your end user’s point of view
    • xooauth: OAuth implementation, including LTI (Learning Tools Interoperability)
    • dotlrn-bootstrap3-theme: Bootstrap 3 theme for DotLRN
    • xowf-monaco-plugin: Integration of Monaco editor with for code exercise types in xowf
    • proctoring-support: utilities and user interfaces to implement proctoring of the user session, mainly intended in the context of distance education and online exams. The main proctoring feature relies only on web technologies and does not require any plugin or additional software. Optional support for the Safe Exam Browser has also been introduced. The package is currently at the core of WU Online Exam infrastructure and is integrated in the inclass exam implementation for xowf.
       
• Require Tcl 8.6, XOTcl 2.1, PostgreSQL 9.6 (PostgreSQL 9.5 EOL: February 2021), tdom 0.9

Install OpenACS with NaviServer from Scratch

Created by Gustaf Neumann, last modified by Brian Fenton 18 Aug 2023, at 03:00 PM

This page describes how to install OpenACS with NaviServer on Unix-like systems (e.g. Linux, macOS, Solaris, OmniOS) by compiling all but PostgreSQL from scratch, guided by script that collects the components from various sources, compiles it, etc.

The installation is done in two steps:

• install-ns.sh: Install NaviServer and its components for a PostgreSQL installation from scratch by obtaining the relevant sources and compiling it. The script assumes PostgreSQL to be already installed (or obtainable via package managers), and installs all other components by obtaining it from the source repositories and compiling it from scratch (e.g. Tcl, tcllib, tDOM, libthread, nsf/XOTcl 2).
   
• install-oacs.sh: Install OpenACS from CVS/git. This script configures a (pre-installed) PostgreSQL installation for
  OpenACS, adds hstore, installs OpenACS core, basic OpenACS packages, xowiki, xowf and optionally dotlrn from CVS/git and generates a config file and startup files (for Ubuntu and Fedora Core). The script assumes a pre-existing NaviServer installation, installed e.g. via install-ns.sh

 These install scripts are frequently updated when new components are released or problems are detected (commit log).

If you open the links above, use save-as in the browser to save the files. Alternatively, download the files as .zip file or clone the repository via GitHub.

   cd /usr/local/src
   git clone https://github.com/gustafn/install-ns
   cd install-ns

The scripts work under a typical Linux installation (e.g. Ubuntu, Fedora Core) as well as on Mac OS X or on OmniOS,  OpenBSD 6.1, 6.3, 6.6, 6.8, 6.9, FreeBSD 12.2, 13.0, Ubuntu 12.04, 13.04, 14.04, 16.04, 18.04, 20.04, Raspbian GNU/Linux 9.4 (stretch), Fedora Core 18, 20, 32, 35, CentOS 7, Roxy Linux 8.4, ArchLinux.. The scripts are tested (over the years) with PostgreSQL 9.1 to 14.5.

On a fresh Ubuntu installation, you should be able to download the two scripts from this page and install OpenACS with NaviServer in the following steps:

   sudo bash

   bash install-ns.sh
   bash install-ns.sh build

   bash install-oacs.sh
   bash install-oacs.sh build

When running "install-ns.sh", you will see an output like the following (shortened here).

SETTINGS   build_dir              (Build directory)                 /usr/local/src
           ns_install_dir         (Installation directory)          /usr/local/ns
           version_ns             (Version of NaviServer)           4.99.24
           git_branch_ns          (Branch for git checkout of ns)   main
           version_modules        (Version of NaviServer Modules)   4.99.24
           version_tcllib         (Version of Tcllib)               1.20
           version_thread         (Version Tcl thread library)      
           version_xotcl          (Version of NSF/NX/XOTcl)         2.4.0
           version_tcl            (Version of Tcl)                  8.6.12
           version_tdom           (Version of tDOM)                 0.9.1
           ns_user                (NaviServer user)                 nsadmin
           ns_group               (NaviServer group)                nsadmin
                                  (Make command)                    make
                                  (Type command)                    type -a
           ns_modules             (NaviServer Modules)              nsdbpg
           with_mongo             (Add MongoDB client and server)   0
           with_postgres          (Install PostgreSQL DB server)    1
           with_postgres_driver   (Add PostgreSQL driver support)   1
           with_system_malloc     (Tcl compiled with system malloc) 0
           with_ns_doc            (NaviServer documentation)        1

The values in the first column can be used to tailor the system for your needs by setting same-named shell variables. One can use e.g.

   version_tdom=0.9.3 bash install-ns.sh 

to configure the compilation to use tDOM in version 0.9.3 instead of 0.9.1. As a more detailed example, if you want to use the newest version of NaviServer, and you want to use a different build directory and get also some extra modules compiled from their git versions, use e.g. the following command:

   version_ns=GIT \
      build_dir=/usr/local/src-test \
      ns_modules="nsdbpg nsdbi nsdbipg nsudp nssmtpd nsloopctl" \
      bash install-ns.sh 

For standard setups, the default settings should be OK.

After running both scripts in the default configuration, you will see e.g.

   Congratulations, you have installed OpenACS with NaviServer on your machine.
   You might start the server manually with
       sudo /usr/local/ns/bin/nsd -t /usr/local/ns/config-oacs-5-10-0.tcl -u nsadmin -g nsadmin

One can start the server manually with the mentioned command.

Using systemd

On Fedora/CentOS or on Ubuntu installations starting with 15.04, systemd is used. The generated startup file for RedHat/Fedora is in /lib/systemd/system/oacs-5-10-0.service. The startup commands for systemd are

    sudo systemctl status oacs-5-10-0
    sudo systemctl start oacs-5-10-0
    sudo systemctl stop oacs-5-10-0

Remember, when a new systemd service is installed, systemd requires the following command to re-scan its service files:

    sudo systemctl daemon-reload

To start OpenACS automatically on every new start of the machine, issue the following command:

    sudo systemctl enable oacs-5-10-0

Using upstart

Alternatively, some older Ubuntu versions (up to 15.04) use upstart. For upstart, the the generated startup file is in /etc/init/oacs-5-9-1.conf. The service can be started/managed with the following commands

    sudo initctl status oacs-5-10-0
    sudo initctl start oacs-5-10-0
    sudo initctl stop oacs-5-10-0

Configuration

When the service is running, one can use OpenACS by browsing to http://localhost:8000/ (when the browser and server is running on the same host). The relevant files are stored under the following locations:

Security Guidelines

Created by Gustaf Neumann, last modified by Gustaf Neumann 24 Jul 2023, at 05:19 PM

The maintained OpenACS packages are regularly tested with state-of-the-art web vulnerability scanners. The packages are developed using the following the guidelines.
 

Protecting against SQL injection (SQI):

• Use colon-prefixed bind variables in SQL statements when possible (for local variables)

  # use bindvars with local variables
  set x 0
  db_0or1row . {select * from acs_objects where object_id = :x}
  

  or pass bind variables explicitly, using the -bind option

  # use bind variables by passing a dictionary (attribute value list)
  set bindvars {object_id 0 security_inherit_p t}
  db_0or1row . {
     select * from acs_objects 
     where object_id = :object_id
     and security_inherit_p = :security_inherit_p
  } -bind $bindvars
  

• In case, bind-variables are not possible, use ns_dbquotevalue or ns_dbquotelist (see NaviServer documentation)

   set x 0
   db_0or1row . [subst {select * from acs_objects where object_id = [ns_dbquotevalue $x]}] 
  

  Manual page
• In general, avoid using double-quoted SQL statements whenever possible to avoid unescaped substitutions.
   

Protecting against cross side scripting (XSS):

• Validate all input values using page contracts with value checkers, and validate all form variables in the form specification.
• Make sure that Content Security Policies (CSP) are activated in the package parameters. When you have to relax the CSP roles, use "security::csp::require" as local as possible (e.g. on the page level) and not on the site level.