OpenACS 5.10.0 Change Summary

• Functional improvements
  • Features:
    • Added additional page_contract filter: oneof(red|green|blue)
    • template::add_event_listener and template::add_confirm_handler now can target elements by CSS selector
    • Added text/markdown to the accepted text formats or rich-text widget
    • Support for range types in .xql files:

      PostgreSQL supports range types since 9.5. When using range types, square braces have to be used in SQL statements. Since OpenACS uses always Tcl substitution in .xql files, and OpenACS does NOT allow backslash substitution in these files, square brackets could not be escaped and therefore not be used in .xql files so far. This change allows now a developer to deactivate the substitution by passing e.g. -subst none to the db_* command using the .xql file. Valid values for -subst are all, none, vars, and commands, default is all which is exactly the behavior of previous releases. Therefore, this change is fully backward compatible.

    • Improved API browser: Visualization for code dependencies (which procs calls what, from where is a proc being called) and test-coverage
    • Warn site administrators about expiring certificates
    • Additional input types (and widgets) for ad_form:
      • checkbox_text
      • color
      • email
      • tel
      • url
      • number
      • file (multiple)
      • h5date and h5time: date and time fields using native HTML5 visualization and input normalization
    • Registry for .js and .css libraries: allow besides classical URLs symbolic names for loading external resources (e.g. jquery), this makes it easier to upgrade libraries in multiple packages (without running into problems with duplicate versions) and supports switching between CDN and local pathsURN. The existing implementation is based on URNs and extends the existing template-head API to support registration for URNs. A URN provides an abstraction and a single place for e.g. updating references to external resources when switching between a CDN and a locally stored resource, or when a resource should be updated. Instead of adding e.g. a CDN URL via template::head::add_script, one can add an URN and control its content from a single place. Use common namespaces for OpenACS such as urn:ad:css:* and urn:ad:js:*.

      • Register URNs:

        Example provider (e.g. in some theme):

           template::register_urn \ 
               -urn urn:ad:js:jquery \ 
               -resource /resources/xowiki/jquery/jquery.min.js
        

      • The registered URN can be used like classical URL after registration.

        Example consumer:

           template::head::add_javascript -src urn:ad:js:jquery

      • Declare composite files: Provide an interface to define that a .js file or a .css file contains multiple other .js/.css files in order to reduce the number of requests.

           template::head::includes -container urn:js::style.js -parts {urn:ad:js:jquery ...}

    • Improved support for streaming HTML: The new API function template::collect_body_scripts can be used to get the content of template::script or CSP calls (template::add_body_script, template::add_event_listener, template::add_body_handler, template::add_script) when streaming HTML (incremental HTML) is used. Before, these call could bot be used for streaming HTML.

  • Reforms:
    • Dynamic blueprint reloading:
      • When reloading apm packages, watched files, etc. the NaviServer blueprint is now updated correctly. This solves the long-standing problem that changes for scheduled procedures required a restart of the server. Furthermore, the old-style reloading was based on a ever-growing list of reload operations each time a new reload was requested, causing slow-downs in long running server instances, especially, when new threads are started.
      • Now changes are applied to all NaviServer threads, including threads for schedules procedures, ns_jobs and the like. Note that the current job has to be finished before the changes can be applied.
      • For the transition, it is possible to switch between the classical reloading style and blueprint reloading by changing a variable in acs-tcl/tcl/apm-procs.tcl
    • Login:
      • Get rid of bugging "login page expired" messages. The 17 years old construct was replaced by newer means to avoid caching of form values from the login form. Admins of existing sites should set the kernel parameter LoginPageExpirationTime to 0
    • Forums:
      • Removed hard-coded dependency with registered_users group when checking forum permissions
      • Don't rely so heavily on acs_permissions to model forum configuration, as this can have unexpected consequences in convoluted multi-group/multi-subsite scenarios. Prefer simpler table attributes instead
      • New style of attachments to the forums, allowing multiple attachments to a single message directly from the message post page, using the multiple file input widget. Retain compatibility with old style attachments, using the new 'AttachmentStyle' package instance parameter. Currently, this supports two values: 'simple' (new behavior) and 'complex' previous behavior.
    • Chat:
      • Revamping of the GUI
      • Responsiveness
      • Full screen mode
      • Skins support (minimal, classic and bubbles, included): Skins are located in the new /packages/xowiki/www/resources/chat-skins/ directory. New skins can be created by just adding the css and js files in the skins directory, and naming them accordingly (chat-$SKIN_NAME.{js|css}).
      • Avatars (can be enabled per room)
      • Number of active users in chat
      • Tab notifications of new messages
      • Web Notifications:
        • https://www.w3.org/TR/notifications/
        • https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API
    • acs-lang:
      • admin pages:
        • Added the option to unregister (delete permanently the message key from all locales) a message key that has been already marked as deleted. Useful for cleaning up old message keys.
        • Added the option to undelete, using the new ::message::undelete proc.
        • Made number and category (untranslated/deleted/...) of messages coherent in all pages.
        • Added the columns 'total' and 'deleted' to the index page.
      • object_id reference: it is now possible to associate a message key to an object_id in a way that e.g. when the object is deleted, so is the message key. This addresses cases such as the message keys generated by group creation or by the new XoWiki localized fields
    • Notifications:
      • Improved scalability for notifications: One of the most expensive operations in large site is the cleanup for notification_requests in situations, where the user has lost permissions on an object, on which the user wanted to receive notifications. This check was performed previously in notification::sweep::cleanup_notifications via a permission check over all notification requests, which can be very costly on large sites. This change moves this cleanup into the actual notification sending, where the permissions have to be sent anyhow.
      • When sending a notification on behalf of a person, if the system is not configured to process replies to notification, do not set the reply-to address to anything different than the sender
      • Notifications: proper cleanup of acs_objects resulting from the deletion of dynamic notification requests
    • User/Person/Party API: rework and rationalize caching of all party, person and user API, create separate caches for each of these types, make the API and return dicts. acs_user::get will not fail anymore with non-existing user.
    • User Portrait: created API to retrieve and create, store and delete the user's portrait. Also address leftover child relationships from the past and delete them properly.
• Non-functional Changes
  • Improved automated regression test infrastructure and test coverage
    • All packages in the oacs-5-10 branch pass regression test
    • Web testing was separated from non-maintained tcltest and was built on the standard OpenACS infrastructure
    • Include web testing per default in standard regression testing
    • Introduced new test authentication authority, allowing to run many user administration tests outside the context of a "real authority": in cases where the real authority depends on external services to proof identity, (e.g. Kerberos), those tests would just fail.
    • Introduce the display of warnings in the UI
    • Added test coverage information in the automated testing pages, using the new proc-coverage API and providing test coverage information for packages and system wide.
    • Increased overall coverage of public API
    • New tests checking various data-model properties and smells
       
  • Improved scalability:

    • Provided lock-free implementation of ad_page_contract_filters and ad_page_contract_filter_rules. This change improves parallel processing of requests and is primarily interesting for sites with a few mio page views per days. These locks were among the most frequent nsv locks

    • Reduced locks on util_memoize_cache my more invariants values into per-thread caching (acs_lookup_magic_object, ad_acs_version, .... ) and by avoiding specialized calls, which can be realized by already optimized ones (apm_package_installed_p_not_cached ref-timezones was most frequently used util_memoize_cache entry). These changes are necessary to avoid full still-stand on the unfortunate long-time locks on util_memoize_cache stemming from permission and user management with wild-card flush operations, which require to iterate over all cache entries (which might be on a busy server several hundred thousands)

    • Added new interface for cache partitioning to reduce lock latencies on high load websites

    • Added new interface for lock-free per-thread and per-request caching to avoid scattered ad-hoc implementations

    • Better reuse of DB handles (reduced expiring/reopen/etc.), faster access to handles

  • Improved startup time:
    • When the package acs-automated-testing is disabled, startup time is reduced by avoiding loading of support functions and tests; the size of the blueprint is reduced
    • xowf: loading of at-jobs is significantly improved.
       
  • Security improvements:
    • Strengthened page contracts
    • CSP support for application packages
    • CSP fine tuning
       
  • Better exception handling based on Tcl 8.6 exception handlers (try and throw, also available in Tcl 8.5)
    • Provided a new ad_try implementation based on Tcl's try replaces now the old ad_try, with_catch and with_finally, which are marked as deprecated
    • The new ad_try is in essence Tcl's try but with predefined handling of ad_script_abort and should be also used instead of catch, when the OpenACS API is used (which might use script aborts)
    • All core packages use the new ad_try instead of the deprecated versions.
       
  • Connection close reform:
    • NaviServer/AOLserver continue after connection closing commands to execute a script. This is in many situations not desired, especially, when for the page as well a .adp file exists, which will try to deliver this on the already closed connection. This can lead to errors in the error.log file, which are sometimes hard to analyze
    • Due to this cleanup, developers should use in most such cases cases ad_script_abort
    • Connection closing commands are e.g. ad_returnredirect, ad_redirect_for_registration, cr_write_content, ad_page_contract_handle_datasource_error, ad_return_string_as_file, ad_return_complaint, ad_return_error, ad_return_forbidden, ad_return_warning, ad_return_exception_page, ns_returnredirect, ns_return, ns_returnerror
    • The new version has made on most occasions explicit, when the script should abort.
       
  • API changes (new and extended API calls):
    • New API call category::get to obtain category description for a category_id and locale
    • New utility ad_pad emulating both lpad and rpad typically available in DBMSs
    • New proc lc_content_size_pretty, prettify data size given in bytes. It supports three different standards (SI base-10, IEC base-2 and the old JEDEC base-2), default is SI base-10.
    • New flag -export for ad_form: this flag uses export_vars under the hood and supports all of this API's features (e.g. :multiple, :sign, :array). This addresses a long standing TODO
    • util::pdfinfo: simple poppler-utils wrapper to extract pdf information
    • util::http: leverage new ns_http features such as request file spooling. Native implementation will now be used only on NaviServer >= 4.99.15.
    • Database API:
      • db_foreach: queries executed inside of a db_foreach will not be issued using a different handle and will therefore be safe to use in a transaction
      • db_list_of_lists: new -with_headers flag, which will make the first element of the returned list to be the column names as defined in the query
    • Groups API:
      • Logics to delete a group type have now been included in the API
      • Allow to filter group members by member_state in the API
         
  • Deprecated commands:
    • Many deprecated API calls were included in the code (and sometimes still in use) sometimes more than 10 years after these calls have been deprecated. In case a site modification still uses deprecated code, the user is warned about this. The OpenACS 5.10 code base does not depend on deprecated code.
    • Move deprecated code into separate files
    • Made loading of deprecated code optional (can be controlled via parameter "WithDeprecatedCode" in section "ns_section ns/server/${server}/acs" of the config file. By default, deprecated procs are still loaded
    • When deprecated code is not loaded, the blueprint of the interpreter is smaller. The following number of lines of code can be omitted when loading without the deprecated procs:
      • acs-tcl: 3178
      • acs-templating: 450
      • xotcl-core http-client-procs: 830
      • acs-content-repository: 1717 (including .xql files)
         
  • Bugfix and Code Maintenance:
    • Made sure all party emails are stored as lowercase through the API
    • Fixed long standing regression in template::list: we were looping through the list "elements", rather than the "display_elements". This prevents specifying different sets of columns to be returned depending on the -formats and -selected_format options in template::list::create.
    • acs-content-repository: New HEIC and HEIF mimetypes
    • acs-mail-lite: handle to_addr specified as "DisplayName <email>" without errors
    • Fixed invalidating of all existing user logins, (aka) "Logout from everywhere" feature, useful e.g. to make sure no device still holds a valid login when we change our password on a device
    • Don't lose the return URL when one tries to join a subsite before being logged in
    • Added doc(base_href) and doc(base_target) for setting <base> element via blank-baster (see issue #3435)
    • Groups:
      • When a new group is created, flush all the group::get_id caches with the same name so that the new group can be fetched correctly in case it replaces a previously deleted one
      • Cleanup message keys coming from groups in acs-translations when a group is deleted
    • acs-lang:
      • lang::util::convert_to_i18n: do not always register a en_US translation, which would be always overridden. Instead, let lang::message::register make sure that a en_US message exists and create one only as a fallback.
      • lc_time_fmt: leverage Tcl clock to address shortcomings such as handling of dates in Julian/Gregorian calendar and impossible dates such as 1999-02-29, implement missing formats, support previously undocumented formats explicitly
    • search: make sure objects in the search indexer queue still exist by the time they are swept by the indexer (e.g. items deleted before the indexer could sweep them)
    • attribute::delete: fix proc so it leverages stored procedure capability of dropping the database table as well
    • util::http: fix UTF-8 encoding issues for some cornercases
    • Localization: Complete Italian and Spanish localization for the whole .LRN set of packages (including themes). Message keys for new and previously localized packages have also been updated
       
  • General cleanup/maintenance
    • Improved handling of server implementation-specific code: server-specific code can be optionally loaded via specifying the server family in the filename. Provided *-procs-aolserver.tcl and *-procs-naviserver.tcl similar to *.postgresql.xql and *.oracle.xql where appropriate
    • Modernization of Tcl idioms.
    • Compliance of files, proc names, ... to the naming conventions.
    • White space cleanup, indentation changes.
    • Improvement of public API documentation
    • Adjustment of proc protection levels (public, private)
    • Adjustment of log severity
    • Cleanup of obsolete files
    • Replacement of handcrafted forms by ad_form
    • Typo fixing
    • Editor hints
    • Replacement of deprecated calls
    • Addition of missing contracts
    • ...
       
  • SQL cleanup:
    • Cleanup of obsolete nonportable SQL constructs in a way Oracle and PostgreSQL code base divergency is reduced:
      • "nvl" -> "coalesce"
      • "sysdate" / "now()" -> standard "current_date" or "current_timestamp"
      • Use standard-compliant "dual" table where appropriate (required by Oracle, supported by PostgreSQL)
      • Use non-dialectal cast idioms when appropriate
      • Adopt CTE idioms in Oracle codebase as well (e.g. connect -> with recursive)
      • ... (reference Oracle version will be 11gr2 as is oldest version officially supported by Oracle (See here and here)
    • Reduced superfluous .xql queries
      • acs-subsite: delete 21 files with un-referenced .xql queries
      • acs-tcl: delete 4 files
      • news: 3 files
      • file-storage: 1 file
      • dotlrn: 9 files
         
  • New Packages:
    • cookie-consent: alerting users about the use of cookies on a website
    • boomerang: performance of your website from your end user’s point of view
    • xooauth: OAuth implementation, including LTI (Learning Tools Interoperability)
    • dotlrn-bootstrap3-theme: Bootstrap 3 theme for DotLRN
    • xowf-monaco-plugin: Integration of Monaco editor with for code exercise types in xowf
    • proctoring-support: utilities and user interfaces to implement proctoring of the user session, mainly intended in the context of distance education and online exams. The main proctoring feature relies only on web technologies and does not require any plugin or additional software. Optional support for the Safe Exam Browser has also been introduced. The package is currently at the core of WU Online Exam infrastructure and is integrated in the inclass exam implementation for xowf.
       
• Require Tcl 8.6, XOTcl 2.1, PostgreSQL 9.6 (PostgreSQL 9.5 EOL: February 2021), tdom 0.9