xowiki - Weblog Page

Postponed for later releases:
- Implement subsite-singleton (in addition to the classical singleton)
- Say farewell to CVS
- Work on Tcl9 compatibility
- Removal of JavaScript files from core packages (decide to drop and/or deal with local extensions)
  - openacs-4/packages/acs-templating/www/resources/tinymce
  - openacs-4/packages/acs-templating/www/resources/xinha-nightly
  - openacs-4/packages/acs-templating/www/scripts/xinha
  - openacs-4/packages/acs-templating/www/resources/htmlarea
- Get rid of ajaxhelper (e.g. move it to unsupported packages)
- Replace all error-swallowing catch operations
- Make thougrough cleanup after regression test (maybe provide a parameter to keep artefacts for further investigation)
- Make more parts of xotcl-core a first-class citizen of the OpenACS core (e.g. database interface)
- Add better markdown support to xowiki via markdown formfield + FormPage
- Data bloat hygiene:
  - Rethink package parameter and portlet parameter data models
  - Parameters: include subsite-parameters in parameter resolution (package->subsite->global)
Upgrade/migrate OpenACS website to Bootstrap5 (has many local modifications)

OpenACS Version 5.10.1 Agenda

OpenACS 5.10.1 Change Summary

Security:
- Stronger password hashes for OpenACS
  - New password hashes in addition to the classical "salted-sha1"
    - "scram-sha-256": SCRAM (RFC 7677) with parameter recommendation from RFC 7677; requires OpenSSL 1.1.1 and NaviServer 4.99.23 or newer
    - "scrypt-16384-8-1": SCRYPT (RFC 7914) with parameter "-n 16384 -r 8 -p 1"; requires OpenSSL 3.0 and NaviServer 4.99.23 or newer
    - "argon2-argon2-12288-3-1": Argon2 (RFC9106), Parameterization recommendation from OWASP: m=12288 (12 MiB), t=3, p=1; requires OpenSSL 3.2 and NaviServer 5.0 or newer
    - "argon2-rfc9106-high-mem": Argon2 (RFC9106), first (memory intense) recommendation from RFC 9106; requires OpenSSL 3.2 and NaviServer 5.0 or newer
    - "argon2-rfc9106-low-mem": Argon2 (RFC9106), second recommendation from RFC 9106; requires OpenSSL 3.2 and NaviServer 5.0 or newer
  - Preferences of the password hash algorithms can be set via kernel package parameter "PasswordHashAlgorithm", the first available algorithm is taken from the preference list, hash re-coding happens automatically at the next login.
  - See https://openacs.org/forums/message-view?message_id=5537869
  - Added optional CSP rules based on MIME types. This is important for user-contributed content. When users upload e.g. SVG-files to the file storage, and the content is served from there, it poses a potential security hole. One can now define an additional parameter called "StaticCSP" in the section "ns/server/$server/acs" of the OpenACS configuration file to deactivate execution of script files from static content.
    
    ns_param StaticCSP { image/svg+xml "script-src 'none'" }
  - Cookie-Namespace: When multiple OpenACS instances are served from the same domain name, the same cookies (e.g. ad_session_id, ad_login, ...) are set to all servers. For sensible cases, a cookie-namespace can be used, which can be used as a replacement of the traditional "ad_" prefix. This can be as well set in the section "ns/server/$server/acs" of the OpenACS configuration file
    
    # Provide optionally a different cookie namespace # (used for prefixing OpenACS cookies) ns_param CookieNamespace "ad_"
Improved templating:
- Client-side double click prevention
- Support for generic icon names, which can be mapped differently depending on the installed packages and themes: The generic names are supported via <adp:icon name="NAME" title=...>. By using this feature, one can use font-based icons (like e.g. glyphicons of Bootstrap5, bootstrap-icons, fa-icons, ...) instead of the old-style .gif and .png images. This makes the appearance more uniform, has better resizing behavior, and works more efficiently (fewer requests for embedded resources). Most of the occurrences of the old-style images in standard core and non-core packages in oacs-5-10 are already replaced.
- Support for listing registered URNs
Require NaviServer (i.e. drop AOLserver support).
Rationale: AOLserver cannot be compiled with the required modules with recent Tcl versions. Trying to backport NaviServer compatibility functions seems to be an overkill for the OpenACS project.
Further reduce divergence between Oracle and Postgres SQL. Target version of Oracle could be 12.*, as Extended support ends in 2022 (see https://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf)
- limit / rownum -> fetch first
- use Postgres schemas for stored procedures so that they can be invoked with the same Oracle idiom
Bootstrap 3 reached EOL in 2019, Bootstrap 4 had EOL 2022, so we should migrate to Bootstrap 5 (details: https://github.com/twbs/release)
New Packages:
- openacs-bootstrap5
- bootstrap-icons
- fa-icons
- highcharts
Potential incompatibility with OpenACS 5.10.0: "permission::permission_p" returns Boolean values as "t" and "f" and not "1" and "0". Avoid literal comparisons of the result and use boolean tests available in Tcl/OpenACS.
Support for fresh installations on Oracle 19c (for details, see: oacs-5-10-on-oracle-19c)
Require Tcl 8.6.2, XOTcl 2.1, PostgreSQL 12 (PostgreSQL 11 EOL: November 23), tdom 0.9

OpenACS Version 5.10.0 Agenda

OpenACS 5.10.0 Change Summary

Functional improvements
- Features:
  - Added additional page_contract filter: oneof(red|green|blue)
  - template::add_event_listener and template::add_confirm_handler now can target elements by CSS selector
  - Added text/markdown to the accepted text formats or rich-text widget
  - Support for range types in .xql files:
    PostgreSQL supports range types since 9.5. When using range types, square braces have to be used in SQL statements. Since OpenACS uses always Tcl substitution in .xql files, and OpenACS does NOT allow backslash substitution in these files, square brackets could not be escaped and therefore not be used in .xql files so far. This change allows now a developer to deactivate the substitution by passing e.g. -subst none to the db_* command using the .xql file. Valid values for -subst are all, none, vars, and commands, default is all which is exactly the behavior of previous releases. Therefore, this change is fully backward compatible.
  - Improved API browser: Visualization for code dependencies (which procs calls what, from where is a proc being called) and test-coverage
  - Warn site administrators about expiring certificates
  - Additional input types (and widgets) for ad_form:
    - checkbox_text
    - color
    - email
    - tel
    - url
    - number
    - file (multiple)
    - h5date and h5time: date and time fields using native HTML5 visualization and input normalization
  - Registry for .js and .css libraries: allow besides classical URLs symbolic names for loading external resources (e.g. jquery), this makes it easier to upgrade libraries in multiple packages (without running into problems with duplicate versions) and supports switching between CDN and local pathsURN. The existing implementation is based on URNs and extends the existing template-head API to support registration for URNs. A URN provides an abstraction and a single place for e.g. updating references to external resources when switching between a CDN and a locally stored resource, or when a resource should be updated. Instead of adding e.g. a CDN URL via template::head::add_script, one can add an URN and control its content from a single place. Use common namespaces for OpenACS such as urn:ad:css:* and urn:ad:js:*.
    - Register URNs:
      Example provider (e.g. in some theme):
```
   template::register_urn \ 
       -urn urn:ad:js:jquery \ 
       -resource /resources/xowiki/jquery/jquery.min.js
```
    - The registered URN can be used like classical URL after registration.
      
      Example consumer:
```
   template::head::add_javascript -src urn:ad:js:jquery
```
    - Declare composite files: Provide an interface to define that a .js file or a .css file contains multiple other .js/.css files in order to reduce the number of requests.
```
   template::head::includes -container urn:js::style.js -parts {urn:ad:js:jquery ...}
```
  - Improved support for streaming HTML: The new API function template::collect_body_scripts can be used to get the content of template::script or CSP calls (template::add_body_script, template::add_event_listener, template::add_body_handler, template::add_script) when streaming HTML (incremental HTML) is used. Before, these call could bot be used for streaming HTML.
- Reforms:
  - Dynamic blueprint reloading:
    - When reloading apm packages, watched files, etc. the NaviServer blueprint is now updated correctly. This solves the long-standing problem that changes for scheduled procedures required a restart of the server. Furthermore, the old-style reloading was based on a ever-growing list of reload operations each time a new reload was requested, causing slow-downs in long running server instances, especially, when new threads are started.
    - Now changes are applied to all NaviServer threads, including threads for schedules procedures, ns_jobs and the like. Note that the current job has to be finished before the changes can be applied.
    - For the transition, it is possible to switch between the classical reloading style and blueprint reloading by changing a variable in acs-tcl/tcl/apm-procs.tcl
  - Login:
    - Get rid of bugging "login page expired" messages. The 17 years old construct was replaced by newer means to avoid caching of form values from the login form. Admins of existing sites should set the kernel parameter LoginPageExpirationTime to 0
  - Forums:
    - Removed hard-coded dependency with registered_users group when checking forum permissions
    - Don't rely so heavily on acs_permissions to model forum configuration, as this can have unexpected consequences in convoluted multi-group/multi-subsite scenarios. Prefer simpler table attributes instead
    - New style of attachments to the forums, allowing multiple attachments to a single message directly from the message post page, using the multiple file input widget. Retain compatibility with old style attachments, using the new 'AttachmentStyle' package instance parameter. Currently, this supports two values: 'simple' (new behavior) and 'complex' previous behavior.
  - Chat:
    - Revamping of the GUI
    - Responsiveness
    - Full screen mode
    - Skins support (minimal, classic and bubbles, included): Skins are located in the new /packages/xowiki/www/resources/chat-skins/ directory. New skins can be created by just adding the css and js files in the skins directory, and naming them accordingly (chat-$SKIN_NAME.{js|css}).
    - Avatars (can be enabled per room)
    - Number of active users in chat
    - Tab notifications of new messages
    - Web Notifications:
      - https://www.w3.org/TR/notifications/
      - https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API
  - acs-lang:
    - admin pages:
      - Added the option to unregister (delete permanently the message key from all locales) a message key that has been already marked as deleted. Useful for cleaning up old message keys.
      - Added the option to undelete, using the new ::message::undelete proc.
      - Made number and category (untranslated/deleted/...) of messages coherent in all pages.
      - Added the columns 'total' and 'deleted' to the index page.
    - object_id reference: it is now possible to associate a message key to an object_id in a way that e.g. when the object is deleted, so is the message key. This addresses cases such as the message keys generated by group creation or by the new XoWiki localized fields
  - Notifications:
    - Improved scalability for notifications: One of the most expensive operations in large site is the cleanup for notification_requests in situations, where the user has lost permissions on an object, on which the user wanted to receive notifications. This check was performed previously in notification::sweep::cleanup_notifications via a permission check over all notification requests, which can be very costly on large sites. This change moves this cleanup into the actual notification sending, where the permissions have to be sent anyhow.
    - When sending a notification on behalf of a person, if the system is not configured to process replies to notification, do not set the reply-to address to anything different than the sender
    - Notifications: proper cleanup of acs_objects resulting from the deletion of dynamic notification requests
  - User/Person/Party API: rework and rationalize caching of all party, person and user API, create separate caches for each of these types, make the API and return dicts. acs_user::get will not fail anymore with non-existing user.
  - User Portrait: created API to retrieve and create, store and delete the user's portrait. Also address leftover child relationships from the past and delete them properly.
Non-functional Changes
- Improved automated regression test infrastructure and test coverage
  - All packages in the oacs-5-10 branch pass regression test
  - Web testing was separated from non-maintained tcltest and was built on the standard OpenACS infrastructure
  - Include web testing per default in standard regression testing
  - Introduced new test authentication authority, allowing to run many user administration tests outside the context of a "real authority": in cases where the real authority depends on external services to proof identity, (e.g. Kerberos), those tests would just fail.
  - Introduce the display of warnings in the UI
  - Added test coverage information in the automated testing pages, using the new proc-coverage API and providing test coverage information for packages and system wide.
  - Increased overall coverage of public API
  - New tests checking various data-model properties and smells
- Improved scalability:
  - Provided lock-free implementation of ad_page_contract_filters and ad_page_contract_filter_rules. This change improves parallel processing of requests and is primarily interesting for sites with a few mio page views per days. These locks were among the most frequent nsv locks
  - Reduced locks on util_memoize_cache my more invariants values into per-thread caching (acs_lookup_magic_object, ad_acs_version, .... ) and by avoiding specialized calls, which can be realized by already optimized ones (apm_package_installed_p_not_cached ref-timezones was most frequently used util_memoize_cache entry). These changes are necessary to avoid full still-stand on the unfortunate long-time locks on util_memoize_cache stemming from permission and user management with wild-card flush operations, which require to iterate over all cache entries (which might be on a busy server several hundred thousands)
  - Added new interface for cache partitioning to reduce lock latencies on high load websites
  - Added new interface for lock-free per-thread and per-request caching to avoid scattered ad-hoc implementations
  - Better reuse of DB handles (reduced expiring/reopen/etc.), faster access to handles
- Improved startup time:
  - When the package acs-automated-testing is disabled, startup time is reduced by avoiding loading of support functions and tests; the size of the blueprint is reduced
  - xowf: loading of at-jobs is significantly improved.
- Security improvements:
  - Strengthened page contracts
  - CSP support for application packages
  - CSP fine tuning
- Better exception handling based on Tcl 8.6 exception handlers (try and throw, also available in Tcl 8.5)
  - Provided a new ad_try implementation based on Tcl's try replaces now the old ad_try, with_catch and with_finally, which are marked as deprecated
  - The new ad_try is in essence Tcl's try but with predefined handling of ad_script_abort and should be also used instead of catch, when the OpenACS API is used (which might use script aborts)
  - All core packages use the new ad_try instead of the deprecated versions.
- Connection close reform:
  - NaviServer/AOLserver continue after connection closing commands to execute a script. This is in many situations not desired, especially, when for the page as well a .adp file exists, which will try to deliver this on the already closed connection. This can lead to errors in the error.log file, which are sometimes hard to analyze
  - Due to this cleanup, developers should use in most such cases cases ad_script_abort
  - Connection closing commands are e.g. ad_returnredirect, ad_redirect_for_registration, cr_write_content, ad_page_contract_handle_datasource_error, ad_return_string_as_file, ad_return_complaint, ad_return_error, ad_return_forbidden, ad_return_warning, ad_return_exception_page, ns_returnredirect, ns_return, ns_returnerror
  - The new version has made on most occasions explicit, when the script should abort.
- API changes (new and extended API calls):
  - New API call category::get to obtain category description for a category_id and locale
  - New utility ad_pad emulating both lpad and rpad typically available in DBMSs
  - New proc lc_content_size_pretty, prettify data size given in bytes. It supports three different standards (SI base-10, IEC base-2 and the old JEDEC base-2), default is SI base-10.
  - New flag -export for ad_form: this flag uses export_vars under the hood and supports all of this API's features (e.g. :multiple, :sign, :array). This addresses a long standing TODO
  - util::pdfinfo: simple poppler-utils wrapper to extract pdf information
  - util::http: leverage new ns_http features such as request file spooling. Native implementation will now be used only on NaviServer >= 4.99.15.
  - Database API:
    - db_foreach: queries executed inside of a db_foreach will not be issued using a different handle and will therefore be safe to use in a transaction
    - db_list_of_lists: new -with_headers flag, which will make the first element of the returned list to be the column names as defined in the query
  - Groups API:
    - Logics to delete a group type have now been included in the API
    - Allow to filter group members by member_state in the API
- Deprecated commands:
  - Many deprecated API calls were included in the code (and sometimes still in use) sometimes more than 10 years after these calls have been deprecated. In case a site modification still uses deprecated code, the user is warned about this. The OpenACS 5.10 code base does not depend on deprecated code.
  - Move deprecated code into separate files
  - Made loading of deprecated code optional (can be controlled via parameter "WithDeprecatedCode" in section "ns_section ns/server/${server}/acs" of the config file. By default, deprecated procs are still loaded
  - When deprecated code is not loaded, the blueprint of the interpreter is smaller. The following number of lines of code can be omitted when loading without the deprecated procs:
    - acs-tcl: 3178
    - acs-templating: 450
    - xotcl-core http-client-procs: 830
    - acs-content-repository: 1717 (including .xql files)
- Bugfix and Code Maintenance:
  - Made sure all party emails are stored as lowercase through the API
  - Fixed long standing regression in template::list: we were looping through the list "elements", rather than the "display_elements". This prevents specifying different sets of columns to be returned depending on the -formats and -selected_format options in template::list::create.
  - acs-content-repository: New HEIC and HEIF mimetypes
  - acs-mail-lite: handle to_addr specified as "DisplayName <email>" without errors
  - Fixed invalidating of all existing user logins, (aka) "Logout from everywhere" feature, useful e.g. to make sure no device still holds a valid login when we change our password on a device
  - Don't lose the return URL when one tries to join a subsite before being logged in
  - Added doc(base_href) and doc(base_target) for setting <base> element via blank-baster (see issue #3435)
  - Groups:
    - When a new group is created, flush all the group::get_id caches with the same name so that the new group can be fetched correctly in case it replaces a previously deleted one
    - Cleanup message keys coming from groups in acs-translations when a group is deleted
  - acs-lang:
    - lang::util::convert_to_i18n: do not always register a en_US translation, which would be always overridden. Instead, let lang::message::register make sure that a en_US message exists and create one only as a fallback.
    - lc_time_fmt: leverage Tcl clock to address shortcomings such as handling of dates in Julian/Gregorian calendar and impossible dates such as 1999-02-29, implement missing formats, support previously undocumented formats explicitly
  - search: make sure objects in the search indexer queue still exist by the time they are swept by the indexer (e.g. items deleted before the indexer could sweep them)
  - attribute::delete: fix proc so it leverages stored procedure capability of dropping the database table as well
  - util::http: fix UTF-8 encoding issues for some cornercases
  - Localization: Complete Italian and Spanish localization for the whole .LRN set of packages (including themes). Message keys for new and previously localized packages have also been updated
- General cleanup/maintenance
  - Improved handling of server implementation-specific code: server-specific code can be optionally loaded via specifying the server family in the filename. Provided *-procs-aolserver.tcl and *-procs-naviserver.tcl similar to *.postgresql.xql and *.oracle.xql where appropriate
  - Modernization of Tcl idioms.
  - Compliance of files, proc names, ... to the naming conventions.
  - White space cleanup, indentation changes.
  - Improvement of public API documentation
  - Adjustment of proc protection levels (public, private)
  - Adjustment of log severity
  - Cleanup of obsolete files
  - Replacement of handcrafted forms by ad_form
  - Typo fixing
  - Editor hints
  - Replacement of deprecated calls
  - Addition of missing contracts
  - ...
- SQL cleanup:
  - Cleanup of obsolete nonportable SQL constructs in a way Oracle and PostgreSQL code base divergency is reduced:
    - "nvl" -> "coalesce"
    - "sysdate" / "now()" -> standard "current_date" or "current_timestamp"
    - Use standard-compliant "dual" table where appropriate (required by Oracle, supported by PostgreSQL)
    - Use non-dialectal cast idioms when appropriate
    - Adopt CTE idioms in Oracle codebase as well (e.g. connect -> with recursive)
    - ... (reference Oracle version will be 11gr2 as is oldest version officially supported by Oracle (See here and here)
  - Reduced superfluous .xql queries
    - acs-subsite: delete 21 files with un-referenced .xql queries
    - acs-tcl: delete 4 files
    - news: 3 files
    - file-storage: 1 file
    - dotlrn: 9 files
- New Packages:
  - cookie-consent: alerting users about the use of cookies on a website
  - boomerang: performance of your website from your end user’s point of view
  - xooauth: OAuth implementation, including LTI (Learning Tools Interoperability)
  - dotlrn-bootstrap3-theme: Bootstrap 3 theme for DotLRN
  - xowf-monaco-plugin: Integration of Monaco editor with for code exercise types in xowf
  - proctoring-support: utilities and user interfaces to implement proctoring of the user session, mainly intended in the context of distance education and online exams. The main proctoring feature relies only on web technologies and does not require any plugin or additional software. Optional support for the Safe Exam Browser has also been introduced. The package is currently at the core of WU Online Exam infrastructure and is integrated in the inclass exam implementation for xowf.
Require Tcl 8.6, XOTcl 2.1, PostgreSQL 9.6 (PostgreSQL 9.5 EOL: February 2021), tdom 0.9

OpenACS Version 5.9.1 Agenda

Refactoring of rich-text editor integration

Driving force: Debian packaging (e.g. js minified code is not allowed)
Moved out code from acs-templating, provided interfaces to add many different rich-text editors as separate packages
New OpenACS packages:
- richtext-xinha
- richtext-tinymce
- richtext-ckeditor4 (has ability to choose between CDN and local installation via web interface)

Improving admin interface

New theme manager:
- Goals:
  - Make it easier to keep track of themes with local modifications
  - Make it easier to create local modifications of existing themes and to update these
  - Show differences between default theme parameter (in DB) and actual settings (in subsite parameters)
  - Allow to delete unused themes
  - Give site admin hints, which theme is used at which subsite
  - Ease theme switching
- Added a subsite::theme_changed callback to be able to handle theme changes in custom themes (was also necessary for proper integration with DotLRN theming)
- Added support for these features under subsite admin (/admin/)
- Improved support for themed templates via [template::themed_template]
Improved (broken) interface to define/manage groups over web interface
Allow to send as well mail, when membership was rejected
New functions [membership_rel::get_user_id], [membership_rel::get] and [membership_rel::get_group_id] to avoid code duplication
Added support to let user include %forgotten_password_url% in self-registration emails (e.g. in message key acs-subsite.email_body_Registration_password)
Improved subsite/www/members
- Make it possible to manage members of arbitrary groups
- Improved performance for large groups
- Improved configurability: when ShowMembersListTo is set to "3", show list to members only, when this is not the whole subsite
Improved user interface for /admin/applications for large number of applications
Various fixes for sitewide-admin pages (under /acs-admin)
Update blueprint in "install from repository" (currently just working in NaviServer)

SQL

Further cleanup of .xql files (like what as done for acs-subsite in OpenACS 5.9.0):
- 36 files deleted
- Removed more than 100 obsolete named queries
- Stripped misleading SQL statements
Marked redundant / uncalled SQL functions as deprecated
Replaced usages of obsolete view "all_object_party_privilege_map" by "acs_object_party_privilege_map"
Removed type discrepancy introduced in 2002:
- acs_object_types.object_type has type varchar(1000), while
- acs_object_types.supertype has type varchar(100)
- ... several more data types are involved, using acs_object_types.object_type as foreign key
Simplified core SQL functions by using defaults:
- Number of functions reduced by a factor of 2 compared to OpenACS 5.9.0 (while providing compatibility for clients using old versions),
- Reduced code redundancy
- Affected functions:
  - Reduced content_item__new from 12 versions to 6,
  - Reduce content_revision__new from 7 to 4
  - Similar in image__new, image__new_revision, content_item__copy, content_item__get_title, content_item__move
- PostgreSQL 9.5 supports named parameter in the same syntax as in Oracle. Further reduction of variants will be possible, once OpenACS requires at least PostgreSQL 9.5
Reduced usage of deprecated versions of SQL functions (mostly content repository calls)
Reduced generation of dead tuples by combining multiple DML statements to one (reduces costs of checkpoint cleanups in PostgreSQL)
Permission queries:
- Improved performance
- Support PACKAGE.FUNCTION notation for PostgreSQL to allow calls permission queries exactly the same way as in Oracle (e.g. "acs_permission.permission_p()"). This helps to reduce the number of PostgreSQL specific .xql files.
Modernized SQL:
- Use real Boolean types instead of character(1)
  (done for new-portal, forums, faq, attachments, categories, dotlrn, dotlrn-forums, evaluation)
- Use real enumeration types rather than check constraints (done for storage_type text/file/lob)

CR hygienics (reduce cr bloat)

Provided means to avoid insert/update/delete operations in the search queue:
OpenACS adds for every new revision often multiple entries to the search_queue, without providing any means to prevent this. This requires for busy sites very short intervals between queue sweeps (otherwise too many entries pile up). Another consequence is that this behavior keeps the PostgreSQL auto-vacuum daemons permanently active. Many of these operations are useless in cases where the content repository is used for content that should not be provided via search. The changed behavior should honors a publish-date set to the future, since it will not add any content with future publish dates to the search-queue.
Reduced number of insert cr_child_rels operations, just when needed:
cr_child_rels provide only little benefit (allow one to use roles in a child-rel), but the common operation is a well available in cr_items via the parent_id. cr_child_rels do not help for recursive queries either. One option would be to add an additional argument for content_item__new to omit child-rel creation (default is old behavior) and adapt the other cases.

Security improvements

Added support against CSRF (cross site request forgery)
- OpenACS maintains a per-request CSRF token that ensures that form replies are coming just from sites that received the form
- CSRF support is optional for packages where CSRF is less dangerous, and such requests are wanted (e.g. search and API-browser)
Added Support for W3C "Upgrade-Insecure-Headers" (see https://www.w3.org/TR/upgrade-insecure-requests/):
For standard compliant upgrade for requests from HTTP to HTTPS
Added support for W3C "Subresource Integrity" (SRI; see https://www.w3.org/TR/SRI/)
Added support for W3C "Content Security Policy" (CSP; see https://www.w3.org/TR/CSP/)
- Removed "javascript:*" links (all such urls are removed from the 90 packages in oacs-5-9, excluding js libraries (ajaxhelper) and richtext code)
- Removed "onclick", "onfocus", "onblur", "onchange" handlers from all .adp and .tcl files in the 90 packages in oacs-5-9 (excluding js libraries (ajaxhelper) and richtext code)
- Added optional nonces to all <script> elements with literal JavaScript content
Removed "generic downloader", which allowed to download arbitrary content items, when item_id was known (bug-fix)
Improved protection against XSS and SQL-injection (strengthen page contracts, add validators, added page_contract_filter "localurl", improve HTML escaping, and URI encoding)
Fixed for potential traversal attack (acs-api-documentation-procs)

Improvements for "host-node mapped" subsites

Fixed links from host-node mapped subsite pages to swa-functions (must be always on main subsite)
Made "util_current_directory" aware of host-node-mapped subsites
Added ability to pass "-cookie_domain" to make it possible to use the same cookie for different domains
Fixed result of affected commands "util_current_location", "ad_return_url", "ad_get_login_url" and "ad_get_logout_url" for HTTP and HTTPS, when UseHostnameDomainforReg is 0 or 1.
Improved UI for host-node maps when a large number of site nodes exists

Reform of acs-rels

Made acs-rels configurable to give the developer the option to specify, whether these are composable or not (default fully backward compatible). This is required to control transitivity in rel-segments
The code changes are based on a patch provided by Michael Steigman.
For details, see:
- https://openacs.org/forums/message-view?message_id=4031049
- https://openacs.org/forums/message-view?message_id=5330734

Improved status code handlers for AJAX scenarios

Don't report data source errors with status code 200 (use 422 instead)
Let "permission::require_permission" return forbidden (403) in AJAX calls (determined via [ad_conn ajaxp])

Improved Internationalization

Extended language catalogs for
- Russian (thanks to v v)
- Italian (thanks to Antonio Pisano)
- Spanish (thanks to Hector Romojaro)
- German (thanks to Markus Moser)
Added (missing) message keys
Improved wording of entries
Added message keys for member_state changes, provide API via group::get_member_state_pretty

Improved online documentation (/doc)

Fixed many broken links
Removed fully obsolete sections
Improved markup (modernize HTML)
Updated various sections

Misc code improvements:

18 issues from the OpenACS-bug-tracker fixed
Made code more robust against invalid/incorrect input (page_contracts, validators, values obtained from header fields such as Accept-Language)
Fixed quoting of message keys on many places
Improved exception handling (often, a "catch" swallows one to much, e.g. script_aborts), introducing "ad_exception".
Generalized handling of leading zeros:
- Fixed cases where leading zeros could lead to unwanted octal interpretations
- Switch to use of " util::trim_leading_zeros" instead of "template::util::leadingTrim", "dt_trim_leading_zeros" and "template::util::leadingTrim", marked the latter as deprecated
URL encoding
- "ad_urlencode_folder_path": new function to perform an urlencode operation on the segments of the provided folder path
- "export_vars": encode path always correctly, except -no_base_encode is specified
- Fixed encoding of the URL path in "ad_returnredirect"
Improvements for "ad_conn":
- Added [ad_conn behind_proxy_p] to check, whether the request is coming from behind a proxy server
- Added [ad_conn behind_secure_proxy_p] to check, whether the request is coming from behind a secure proxy server
- Added [ad_conn ajax_p] to check, whether the request is an AJAX requests (assumption: AJAX request sets header-field Requested-With: XMLHttpRequest")
- Added [ad_conn vhost_url] to obtain the url of host-node-mapped subsites
Added various missing upgrade scripts (missing since many years) of changes that were implemented for new installs to reduce differences between "new"-and "old" (upgraded) installations
Templating
- Get rid of various pesky "MISSING FORMWIDGET: ...formbutton:ok" messages
- Improved support for javascript event handlers in template::head
- New functions "template::add_event_listener" and "template::add_confirm_handler"
- Fix handling, when "page_size_variable_p" is set (was broken since ages)
Improved location and URL handling:
- Refactored and commented "util_current_location" to address security issues, handle IPv6 addresses, IP literal notation, multiple drivers, "
- Improved "security::get_secure_location" (align with documentation)
- New functions:
  - "util::configured_location"
  - "util::join_location", "util::split_location"
  for working on HTTP locations to reduce scattered regexps handling URL components
- Improved IPv6 support
- Use native "ns_parseurl" when available, provide backward compatible version for AOLserver
MIME types:
- Added more Open XML formats for MS-Office to allowed content types
- Modernized entries to IANA recommendations
- New function "cr_check_mime_type" centralizing the retrieval of the mime_type from uploaded content
Finalized cleanup of permissions (started in OpenACS 5.9.0):
- Get rid of "acs_object_context_index " (and therefore on "acs_object_party_privilege_map " as well) on PostgreSQL.
  Reasons:
  - huge table,
  - expensive maintenance, used only in a few places,
Misc new functions:
- "lang::util::message_key_regexp": factor out scattered regexp for detecting message keys
- "ns_md5" and "ns_parseurl": improve compatibility between AOLserver and NaviServer
- "ad_dom_sanitize_html": allow one to specify different sets of tags, attributes and protocols and "ad_dom_fix_html", which is a light weight tidy variant.
Improved HTML rendering (acs-api-browser), provide width and height to speed up rendering
Improved ADP files (e.g. missing doc(title))
Added usage of "ad_include_contract" on more occasions
Modernize Tcl and HTML coding
Reduced dependency on external programs (use Tcl functions instead)
Improved robustness of "file delete" operations all over the code
Improved documentation, fix demo pages
Aligned usages of log notification levels (distinction between "error", "warning" and "notice") with coding-standards
Cleaned up deprecated calls:
- Removed usage of deprecated API functions (e.g. "cc_lookup_email_user", "cc_email_from_party", "util_unlist", ...)
- Moved more deprecated procs to acs-outdated
- Marked remaining (and unused) "cc_*" functions as well as deprecated.
Improved Oracle and windows support
Fixed common spelling errors and standardize spelling of product names all over the code (comments, documentation, ...)
Many more small bug fixes

Version numbers:

require PG 9.2 (End Of Life of PostgreSQL 9.0 was Oct 2015)
require XOTcl 2.0 (presented at the Tcl conference in 2011).

Packages:

New Package Parameters
- acs-kernel:
  - MaxUrlLength: remove hard-coded constant in request processor for max accepted url paths
  - SecureSessionCookie: Let site admin determine, whether or not to use secured session cookies (useful, when not all requests are over HTTPS)
  - CSPEnabledP: activate/deactivate CSP
- acs-kernel (recommended to be set via config file in section "ns/server/${server}/>acs"
  - NsShutdownWithNonZeroExitCode: tell NaviServer to return with a nonzero return code to cause restart (important under windows)
  - LogIncludeUserId: include user_id in access log
- acs-api-browser:
  - ValidateCSRFP: make checking of CSRF optional (default 1)
- acs-content-repository:
  - AllowMimeTypeCreationP: Decides whether we allow unknown mime types to be automatically registered (default: 0}
- news-portlet:
  - display_item_lead_p: Should we display news leads in the portlet? (default 0)
- search:
  - ValidateCSRFP: make checking of CSRF optional (default 1)
- xotcl-request-monitor:
  - do_track_activity: turn activity monitoring on or off (default 0)

New OpenACS packages:
- richtext-xinha
- richtext-tinymce
- richtext-ckeditor4 (has ability to choose between CDN and local installation via GUI)
- openacs-bootstrap3-theme (as used on openacs.org)
- dotlrn-bootstrap3-theme

xotcl-core:
- Improved XOTcl 2.0 and NX support (e.g. api-browser)
- Added "-debug", "-deprecated" to ad_* defined methods (such as e.g. "ad_instproc")
- Make use of explicit "create" statements when creating XOTcl/NX objects (makes it easier to grab intentions and to detect typos)
- Added parameter to "get_instance_from_db" to specify, whether the loaded objects should be initialized
- Added support for PostgreSQL prepared statements of SQL interface in ::xo::dc (nsdb driver)
xowiki:
- Named all web-callable methods www-NAME (to make it clear, what is called, what has to be checked especially carefully)
- Moved templates from www into xowiki/resources to avoid naming conflicts
- Improved ckeditor support
- Added usage of prepared statements for common queries
- Improved error handling
- Better value checking for query parameter, error reporting via ad_return_complaint
- Added option "-path_encode" to methods "pretty_link" and "folder_path" to allow one to control, whether the result should be encoded or not (default true)
- Form fields:
  - Improved repeatable form fields (esp. composite cases), don't require preallocation (can be costly in composite cases)
  - Added signing of form-fields
  - Added HTML5 attributes such as "multiple" (for "file") or "autocomplete"
  - Fixed generation of "orderby" attribute based on form-field names
  - richtext: allow one to specify "extraAllowedContent" via options
  - Improved layout of horizontal check boxes
- Menu bar:
  - Added dropzone (requires bootstrap): drag and drop file upload
  - Added mode toggle (requires bootstrap)
  - Extended default policies for handling e.g. dropzone (file-upload method)
  - Distinguish between "startpage" (menu.Package.Startpage) and "table of contents" (menu.Package.Toc)
- Notifications:
  - Added support for better tailorable notifications: introduced method "notification_render" (similar to "search_render")
  - Added support for tailorable subject lines (method "notification_subject")
- Improved bootstrap support, use "bootstrap" as PreferredCSSToolkit
- Switched to ckeditor4 as PreferredRichtextEditor
- Improved handling of script-abort from within the payload of ::xowiki::Object payloads
- Added parameter to "get_all_children" to specify, whether the child objects should be initialized
xowf:
- Added property "payload" to "WorkflowConstruct" in order to simplify customized workflow "allocate" actions
- Internationalized more menu buttons
xotcl-request-monitor
- Added class "BanUser" (use. e.g. IP address to disallow requests from a user via request monitor)
- Added support for optional user tracking in database
- Added support for monitoring response-time for certain urls via munin
- Increased usage of XOTcl 2.0 variable resolver (potentially speed improvement 4x)
- Performed some refactoring of response-time handling to allow site-admin to make e.g. use of NaviServer's dynamic connection pool management (not included in CVS)
- Added support for partial times in long-calls.tcl to ease interpretation of unexpected slow calls
- last100.tcl: Don't report hrefs to URLs, except to SWAs
chat:
- Introduced new options to set chat rooms so login and/or logout message are not issued every time a user enters/exits a chat-room (important for chats with huge number of participants)
- Parameterized viewing of chat-logs
- Fixed cases of over-/under-quoting
- Fixed JavaScript for IE, where innerHTML can cause problems
file-storage:
- Don't show action keys, when user has no permissions
- Added support for copying of same-named files into a folder (adding suffix)
- Fixed old bugs in connection with "views" package

OpenACS Version 5.9.0 Agenda

Slimming pg SQL core:
- Part 1: improve performance of object deletion
  - remove manual delete operations from acs_object__delete()
- Part 2: content-repository - manual referential integrity management
  - handle referential integrity via pg's integrity constraints rather by functions cr_revision_del_ri_tr, cr_revision_ins_ri_tr, cr_revision_up_ri_tr, cr_revision_del_rev_ri_tr, and cr_revision_del_rev_ri_tr
  - fix broken/missing upgrade scripts from earlier updates
- Part 3: content-repository - manual deletions and nulling
  - Removed manual nulling of live_revision and latest_revision
  - Removed manual deletion of old_revision and new_revision in cr_item_publish_audit
  - Removed manual deletion of item_id in cr_item_publish_audit, cr_release_periods, cr_item_template_map, and cr_item_keyword_map
  - Removed manual deletion of direct permissions
  - Added missing index for child_id to cr_child_rels.
- Part 4: get rid of tree_sortkey in acs-objects
  - Check/fix dependencies in oacs-5-8 packages
  - Get rid of broken/uncalled functions using the column
  - Check/fix dependencies in other packages
  - Remove tree_sortkey and max_child_sortkey
Web interface:
- Improve client performance
  - moving core.js from head to body
  - provide kernel parameter to control expiration date for /resources/
- Protect against more XSS attacks
- Improved HTML validity (see oacs-5-9-html-validity for the checklist)
- Add lightweight support for ckeditor4 for templating::richtext widget (configurable via package parameter "RichTextEditor" of acs-templating. ckeditor4 supports mobile devices (such as iPad, ...).
- New kernel parameter ResourcesExpireInterval to control expiration dates of resources
Templating:
- Improve theme-ability
  - Move more information into theme packages in order to create responsive designs
  - Reduce hard-coding of paths, HTML etc.
- Dimensional slider reform (ad_dimensional):
  - Remove hard-coded table layout from dimensional slider
  - Add backwards compatible templates
  - Move hard-coded styles into theme styling
  - Remove obsolete comments from ad_dimensional
- Complete template variable controls (adding noi18n, addressing bug #2692):
  - @foo@: perform html quoting and internationalization
  - @foo;noquote@: perform internationalization
  - @foo;noi18n@: perform html quoting
  - @foo;literal@: perform neither html quoting nor internationalization
- Improved Russian nationalization
- Support of expiration dates and passwords for signed variables
Documentation:
- Use ACS templating for the (static) OpenACS documentation to provide a more consistent layout and user experience.
- Make pretty-naming of acs-core packages more consistent.
Misc improvements:
- Mark unused functions of acs-tcl/tcl/table-display-procs.tcl as deprecated
- Reduce number of muxtex locks by pre-request and per-thread caching
- Improved development und debugging aids:
  - use "ad_log error|warning .... " instead of "ns_log" to include information of request and callstack in error.log
  - ability to display ns_log entries caused by a request in ds-footer
  - ability to save delivered web pages in file-system for testing HTML validity (especially for admin pages, which are unaccessible for external validity testers)
- More bug fixes
Version numbers:
* require PG 9.0 (End Of Life of PostgreSQL 8.4 was July 2014)
* require XOTcl 2.0 (presented at the Tcl conference in 2011).

OpenACS Version 5.8 Agenda

PostgreSQL 9.2+:
- Get rid of nonstandard backslash escapes in function definitions
- Change quote syntax in SQL files (single quotes around the functions) to recommended PostgreSQL quoting using (recommended since pg8.0, jan 2005). li>Drop aliases in favor of named function arguments (recommended since pg8.0)
- Fix wrong function_args, add missing function_args, align default semantics with the defaults in pg (providing "null" as default means the argument is optional)
- Make OpenACS loadable without any tweaks in the pg config files
Use recursive queries for e.g. permission lookup to avoid performance problems in pg 8.4 and newer)
ADP: Use byte-compiled function wherever possible in compiled adp-code, support "@var;literal@" when neither quotes nor localization is needed in compiled adp-code
Improve support of NaviServer
Switch to Tcl 8.5 (TIP #143)
Improve scalability: Reduce mutex-stress on util-memoize cache and for cache maintenance in general
Code cleanup:
- Get rid of calls to deprecated code (e.g. ad_tables, ad_parameter, ... in acs-core and main packages)
- Improve awareness of usage of deprecated code (complain to error.log)
- Use Tcl 8.5 idioms
- cleanup of various http-client approaches and introduce a common implementation util::http::get and util::http::post; get rid of other usages, mark these as deprecated
- page-contracts: Perform checking of all ids in acs-core and main packages to improve error messages and to improve security
OpenACS 5.8.1 should be released with main packages

OpenACS Version 5.7 Agenda

Support for object management in core
Postgresql 9.0
TinyMCE update (fix for random JS injection issue, affecting Safari)
Fix for "remember me" issue
WCAG2-AA

OpenACS Version 5.6 Agenda

global parameters
package "embeds"
fix search by package_id
core works on Postgresql 8.4

OpenACS Version 5.5 Agenda

DONE: Postgresql 8.3 support: especially regarding tsearch2
DONE: acs-authentication:
- fix upgrade, add conditional logic into site wide tcl library so that you can login to perform the rest of the upgrade
DONE: tinymce:
- upgrade to 3.1.1 + language packs
- HTML Strict cleanup
- create appropriate parameters for its config in acs-templating
acs-mail-lite:
- DONE: cleanup duplicated procs (bounce)
- review the parsing of bouncing messages (case user_id 0)
- DONE: rollout support
Documentation improvements as discussed at the Guatemala conference:
- Make current openacs.org/test-doc source for static files included in the release and provide ease means to achieve this for the release manager
  - DONE (CVS HEAD): Provide in XoWiki an alternative table of contents by nested UL/LI (without JavaScript) for static output
  - DONE (CVS HEAD): Provide in XoWiki a prototype page similar to "book" without edit-buttons etc., using the new table of contents
- Update openacs.org/test-doc where necessary (incomplete list):
  - DONE: Fix the page ordering for the higher chapters (the original document had no 3rd. level numbering)
  - update pages in /test-doc which are more recent in openacs/xowiki
  - bump version numbers of OpenACS, where appropriate (some places talk about openacs-5-0, others about openacs-5-1, oacs-5-2-3rc1 or 5-3)
  - some version numbers of the required components are quite a mess. e.g. some parts say that Postgres 7.3 is required, some examples talks about postgres 7.4.7 and 8.2.4 in the same listing.
  - also the dotlrn version numbers are old dotrln-2.0
  - Tcl version numbers should be 8.4.19
  - The install section for XOTcl is missing in II.3.4
  - remove ChangeLog from documentation
  - find some other prominent place for the ChangeLog
  - Fix indenting in examples (e.g. in Rocael's robust web development framework)
  - overthink Win2000 guidelines. There are the native compiled packages from Maurizio, including everything from postgres, xotcl ....
- It is desired to find a single person responsible for overworking the documentation, however, funding is unclear.

OpenACS Version 5.4 Agenda

DONE: HTML Strict (openacs core)
DONE: finish template::head (daveb)
DONE: test acs-mail-lite (complex send)
DONE: test notifications (complex send)
DONE: new XinHA release, get rid of RTE & HTMLarea, test on Safari
DONE: Form builder: add the ID attribute to the form tag
DONE: acs-lang - keepLocalTranslationP to be removed
DONE search and intermedia-driver: move intermedia specific stuff to its package
DONE: acs-mail-lite - patch for mime::qp_encode bug

Future

Split Xinha and TinyMCE into separate packages see: https://openacs.org/forums/message-view?message_id=2750958
Usability ("my account" page)
XHTML ?
Testing and documentation for recording automated tests using the firefox plugin and the upload feature for it new in automated testing. Probably needs some polishing and should be talked to with Quest who are getting into this.
Parameter Scope Patch https://openacs.org/bugtracker/openacs/patch?patch%5fnumber=845
Remove obsolete master template stuff (default and site master template in openacs-4/www, acs-subsite's group-master, and related CSS and images). Probably in the version which follows 5.5 (probably 5.6). Also remove the compat master stuff at the same time.

Things to merge into this page

Old 5.0 Roadmap discussion

Roadmap discussion 1

Ideas for Boston 2006 Future of OpenACS discussion

A .LRN Roadmap

Another .LRN Roadmap discussion

What's on this page?

This page should include work that is planned on and has someone committed to working on it.

Release Status

See openacs-release-status

Development is taking place in the oacs-5-10 branch.

OpenACS Version 5.10.0 Agenda/wish list

Postponed for later releases:
- Implement subsite-singleton (in addition to the classical singleton)
- Say farewell to CVS
- Make more parts of xotcl-core a first-class citizen of the OpenACS core (e.g. database interface)
- Data bloat hygiene:
  - Rethink package parameter and portlet parameter data models
  - Parameters: include "subsite-parameters" in parameter resolution (package->subsite->global)
Functional improvements
- Features:
  - Added additional page_contract filter: oneof(red|green|blue)
  - template::add_event_listener and template::add_confirm_handler now can target elements by CSS selector
  - Added text/markdown to the accepted text formats or rich-text widget
  - Support for range types in .xql files:
    PostgreSQL supports range types since 9.5. When using range types, square braces have to be used in SQL statements. Since OpenACS uses always Tcl substitution in .xql files, and OpenACS does NOT allow backslash substitution in these files, square brackets could not be escaped and therefore not be used in .xql files so far. This change allows now a developer to deactivate the substitution by passing e.g. "-subst none" to the db_* command using the .xql file. Valid values for "-subst" are "all", "none", "vars", and "commands", default is "all" which is exactly the behavior of before. Therefore, this change is fully backward compatible.
  - New API call "category::get" to obtain category description for a category_id and locale
  - Added doc(base_href) and doc(base_target) for setting <base> element via blank-baster (see issue #3435)
  - Improved API browser: Visualization for code dependencies (which procs calls what, from where is a proc being called) and test-coverage
  - Warn site administrators about expiring certificates
  - Content repository: New HEIC and HEIF mimetypes
  - New utility ad_pad emulating both lpad and rpad typically available in DBMSs
  - New proc lc_content_size_pretty, prettify data size given in bytes. It supports three different standards (SI base-10, IEC base-2 and the old JEDEC base-2), default is SI base-10.
  - Database API:
    - db_foreach: queries executed inside of a db_foreach will not be issued using a different handle and will therefore be safe to use in a transaction
    - db_list_of_lists: new -with_headers flag, which will make the first element of the returned list to be the column names as defined in the query
  - Groups:
    - Logics to delete a group type have now been included in the API
    - Allow to filter group members by member_state in the API
  - ad_form:
    - new input types:
      - checkbox_text
      - color
      - email
      - tel
      - url
      - number
      - file (multiple)
      - h5date and h5time: date and time fields using native HTML5 visualization and input normalization
    - -export flag: this flag now uses export_vars under the hood and therefore support all of this API's features (e.g. :multiple, :sign, :array). This addresses a long standing TODO
  - Registry for .js and .css libraries: allow besides classical URLs symbolic names for loading external resources (e.g. jquery), this makes it easier to upgrade libraries in multiple packages (without running into problems with duplicate versions) and supports switching between CDN and local pathsURN.
    The existing implementation is based on URNs and extends the existing template-head API to support registration for URNs. A URN provides an abstraction and a single place for e.g. updating references to external resources when switching between a CDN and a locally stored resource, or when a resource should be updated. Instead of adding e.g. a CDN URL via template::head::add_script, one can add an URN and control its content from a single place. Use common namespaces for OpenACS such as urn:ad:css:* and urn:ad:js:*.
    - Register URNs:
      Example provider (e.g. in some theme):
```
   template::register_urn \ 
       -urn urn:ad:js:jquery \ 
       -resource /resources/xowiki/jquery/jquery.min.js
```
    - The registered URN can be used like classical URL after registration.
      Example consumer:
```
   template::head::add_javascript -src urn:ad:js:jquery
```
    - Declare composite files: Provide an interface to define that a .js file or a .css file contains multiple other .js/.css files in order to reduce the number of requests.
```
   template::head::includes -container urn:js::style.js -parts {urn:ad:js:jquery ...}
```
  - Improved support for streaming HTML: The new API function template::collect_body_scripts can be used to get the content of template::script or CSP calls (template::add_body_script, template::add_event_listener, template::add_body_handler, template::add_script) when streaming HTML (incremental HTML) is used. Before, these call could bot be used for streaming HTML
  - util::pdfinfo: simple poppler-utils wrapper to extract pdf information
- Reforms:
  - Dynamic blueprint reloading:
    - When reloading apm packages, watched files, etc. the NaviServer blueprint is now updated correctly. This solves the long-standing problem that changes for scheduled procedures required a restart of the server. Furthermore, the old-style reloading was based on a ever-growing list of reload operations each time a new reload was requested, causing slow-downs in long running server instances, especially, when new threads are started.
    - Now changes are applied to all NaviServer threads, including threads for
      schedules procedures, ns_jobs and the like. Note that the current job has to be finished before the changes can be applied.
    - For the transition, it is possible to switch between the classical reloading style and blueprint reloading by changing a variable in acs-tcl/tcl/apm-procs.tcl
  - Login:
    - Get rid of bugging "login page expired" messages. The 17 years old construct was replaced by newer means to avoid caching of form values from the login form. Admins of existing sites should set the kernel parameter "LoginPageExpirationTime" to 0
  - Forums:
    - Removed hard-coded dependency with registered_users group when checking forum permissions
    - Don't rely so heavily on acs_permissions to model forum configuration, as this can have unexpected consequences in convoluted multi-group/multi-subsite scenarios. Prefer simpler table attributes instead
    - New style of attachments to the forums, allowing multiple attachments to a single message directly from the message post page, using the multiple file input widget. Retain compatibility with old style attachments, using the new 'AttachmentStyle' package instance parameter. Currently, this supports two values: 'simple' (new behavior) and 'complex' previous behavior.
  - Chat:
    - Revamping of the GUI
    - Responsiveness
    - Full screen mode
    - Skins support (minimal, classic and bubbles, included): Skins are located in the new /packages/xowiki/www/resources/chat-skins/ directory. New skins can be created by just adding the css and js files in the skins directory, and naming them accordingly (chat-$SKIN_NAME.{js|css}).
    - Avatars (can be enabled per room)
    - Number of active users in chat
    - Tab notifications of new messages
    - Web Notifications:
      - https://www.w3.org/TR/notifications/
      - https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API
  - acs-lang:
    - admin pages:
      - Added the option to unregister (delete permanently the message key from all locales) a message key that has been already marked as deleted. Useful for cleaning up old message keys.
      - Added the option to undelete, using the new ::message::undelete proc.
      - Made number and category (untranslated/deleted/...) of messages coherent in all pages.
      - Added the columns 'total' and 'deleted' to the index page.
    - object_id reference: it is now possible to associate a message key to an object_id in a way that e.g. when the object is deleted, so is the message key. This addresses cases such as the message keys generated by group creation or by the new XoWiki localized fields
  - Notifications:
    - When sending a notification on behalf of a person, if the system is not configured to process replies to notification, do not set the reply-to address to anything different than the sender
  - User/Person/Party API: rework and rationalize caching of all party, person and user API, create separate caches for each of these types, make the API and return dicts. acs_user::get will not fail anymore with non-existing user.
  - User Portrait: created API to retrieve and create, store and delete the user's portrait. Also address leftover child relationships from the past and delete them properly.
  - util::http: leverage new ns_http features such as request file spooling. Native implementation will now be used only on Naviserver >= 4.99.15.
- Bugfixes:
  - Made sure all party emails are stored as lowercase through the API
  - Fixed long standing regression in template::list: we were looping through the list "elements", rather than the "display_elements". This prevents specifying different sets of columns to be returned depending on the "-formats" and "-selected_format" options in template::list::create.
  - acs-mail-lite: handle to_addr specified as "DisplayName <email>" without errors
  - Fixed invalidating of all existing user logins, (aka) "Logout from everywhere" feature, useful e.g. to make sure no device still holds a valid login when we change our password on a device
  - Don't lose the return URL when one tries to join a subsite before being logged in
  - Notifications: proper cleanup of acs_objects resulting from the deletion of dynamic notification requests
  - Groups:
    - When a new group is created, flush all the group::get_id caches with the same name so that the new group can be fetched correctly in case it replaces a previously deleted one
    - Cleanup message keys coming from groups in acs-translations when a group is deleted
  - acs-lang:
    - lang::util::convert_to_i18n: do not always register a en_US translation, which would be always overridden. Instead, let lang::message::register make sure that a en_US message exists and create one only as a fallback.
    - lc_time_fmt: laverage tcl clock to address shortcomings such as handling of dates in Julian/Gregorian calendar and impossible dates such as 1999-02-29, implement missing formats, support previously undocumented formats explicitly
  - search: make sure objects in the search indexer queue still exist by the time they are swept by the indexer (e.g. items deleted before the indexer could sweep them)
  - attribute::delete: fix proc so it leverages stored procedure capability of dropping the database table as well
  - util::http: fix UTF-8 encoding issues for some cornercases
- New Packages:
  - cookie-consent: alerting users about the use of cookies on a website
  - boomerang: performance of your website from your end user’s point of view
  - xooauth: OAuth implementation, including LTI
  - dotlrn-bootstrap3-theme: Bootstrap 3 theme for DotLRN
  - xowf-monaco-plugin: Integration of Monaco editor with for code exercise types in xowf
  - proctoring-support: utilities and user interfaces to implement proctoring of the user session, mainly intended in the context of distance education and online exams. The main proctoring feature is relies only on web technologies and does not require any plugin or additional software. Optional support for the Safe Exam Browser has also been introduced. The package is currently at the core of WU Online Exam infrastructure and is integrated in the inclass exam implementation for XoWf
Non-functional Changes
- Improved regression testing
  - Goal is that all packages in oacs-5-10 pass regression test
  - Include web testing in standard regression testing
  - Introduced new test authentication authority, allowing to run many user administration tests outside the context of a "real authority": in cases where the real authority depends on external services to proof identity, (e.g. Kerberos), those tests would just fail.
  - Introduce the display of warnings in the UI
  - Added test coverage information in the automated testing pages, using the new proc-coverage API and providing test coverage information for packages and system wide.
  - Increased overall coverage of public API
  - New tests checking various data-model properties and smells
- Improved scalability:
  - Provided lock-free implementation of ad_page_contract_filters and ad_page_contract_filter_rules. This change improves parallel processing of requests and is primarily interesting for sites with a few mio page views per days. These locks were among the most frequent nsv locks
  - Reduced locks on util_memoize_cache my more invariants values into per-thread caching (acs_lookup_magic_object, ad_acs_version, .... ) and by avoiding specialized calls, which can be realized by already optimized ones ("apm_package_installed_p_not_cached ref-timezones" was most frequently used util_memoize_cache entry). These changes are necessary to avoid full still-stand on the unfortunate long-time locks on util_memoize_cache stemming from permission and user management with wild-card flush operations, which require to iterate over all cache entries (which might be on a busy server several hundred thousands)
  - Added new interface for cache partitioning to reduce lock latencies on high load websites
  - Added new interface for lock-free per-thread and per-request caching to avoid scattered ad-hoc implementations
  - Better reuse of DB handles (reduced expiring/reopen/etc.), faster access to handles
- Improved startup time:
  - When the package acs-automated-testing is disabled, startup time is reduced by avoiding loading of support functions and tests; the size of the blueprint is reduced
  - xowf: loading of at-jobs is significantly improved.
- Security improvements:
  - Strengthened page contracts
  - CSP support for application packages
  - CSP fine tuing
- Better exception handling based on Tcl 8.6 exception handlers ("try" and "throw", also available in Tcl 8.5)
  - Provided a new "ad_try" implementation based on Tcl's "try" replaces now the old "ad_try", "with_catch" and "with_finally", which are marked as deprecated
  - The new "ad_try" is in essence Tcl's "try" but with predefined handling of "ad_script_abort" and should be also used instead of "catch", when the OpenACS API is used (which might use script aborts)
  - All core packages use the new "ad_try" instead of the deprecated versions
- Connection close reform:
  - NaviServer/AOLserver continue after connection closing commands to execute a script. This is in many situations not desired, especially, when for the page as well a .adp file exists, which will try to deliver this on the already closed connection. This can lead to errors in the error.log file, which are sometimes hard to analyze
  - Due to this cleanup, developers should use in most such cases cases "ad_script_abort"
  - Connection closing commands are e.g. ad_returnredirect, ad_redirect_for_registration, cr_write_content, ad_page_contract_handle_datasource_error, ad_return_string_as_file, ad_return_complaint, ad_return_error,
    ad_return_forbidden, ad_return_warning, ad_return_exception_page, ns_returnredirect, ns_return, ns_returnerror
  - The new version has made on most occasions explicit, when the script should abort.
- SQL cleanup:
  - Cleanup of obsolete nonportable SQL constructs in a way Oracle and PostgreSQL code base divergency is reduced:
    - "nvl" -> "coalesce"
    - "sysdate" / "now()" -> standard "current_date" or "current_timestamp"
    - Use standard-compliant "dual" table where appropriate (required by Oracle, supported by PostgreSQL)
    - Use non-dialectal cast idioms when appropriate
    - Adopt CTE idioms in Oracle codebase as well (e.g. connect -> with recursive)
    - ... (reference Oracle version will be 11gr2 as is oldest version officially supported by Oracle (See here and here)
  - Reduced superfluous .xql queries
    - acs-subsite: delete 21 files with un-referenced .xql queries
    - acs-tcl: delete 4 files
    - news: 3 files
    - file-storage: 1 file
    - dotlrn: 9 files
- Deprecated commands:
  - Move deprecated code into separate files
  - Made loading of deprecated code optional (can be controlled via parameter "WithDeprecatedCode" in section "ns_section ns/server/${server}/acs" of the config file. By default, deprecated procs are still loaded
  - When deprecated code is not loaded, the blueprint of the interpreter is smaller. The following number of lines of code can be omitted when loading without the deprecated procs:
    - acs-tcl: 3178
    - acs-templating: 450
    - xotcl-core http-client-procs: 830
    - acs-content-repository: 1717 (including .xql files)
- Improved handling of server implementation-specific code
  - Provided *-procs-aolserver.tcl and *-procs-naviserver.tcl similar to *.postgresql.xql and *.oracle.xql where appropriate
- General cleanup/mainteneance
  - Modernization of Tcl idioms.
  - Compliance of files, proc names, ... to the naming conventions.
  - White space cleanup, indentation changes.
  - Improvement of public API documentation
  - Adjustment of proc protection levels (public, private)
  - Adjustment of log severity
  - Cleanup of obsolete files
  - Replacement of handcrafted forms by ad_form
  - Typo fixing
  - Editor hints
  - Replacement of deprecated calls
  - Addition of missing contracts
  - ...
Require Tcl 8.6, XOTcl 2.1, PostgreSQL 9.6 (PostgreSQL 9.5 EOL: February 2021), tdom 0.9

OpenACS Version 5.9.1 Agenda

Refactoring of rich-text editor integration

Driving force: Debian packaging (e.g. js minified code is not allowed)
Moved out code from acs-templating, provided interfaces to add many different rich-text editors as separate packages
New OpenACS packages:
- richtext-xinha
- richtext-tinymce
- richtext-ckeditor4 (has ability to choose between CDN and local installation via web interface)

Improving admin interface

New theme manager:
- Goals:
  - Make it easier to keep track of themes with local modifications
  - Make it easier to create local modifications of existing themes and to update these
  - Show differences between default theme parameter (in DB) and actual settings (in subsite parameters)
  - Allow to delete unused themes
  - Give site admin hints, which theme is used at which subsite
  - Ease theme switching
- Added a subsite::theme_changed callback to be able to handle theme changes in custom themes (was also necessary for proper integration with DotLRN theming)
- Added support for these features under subsite admin (/admin/)
- Improved support for themed templates via [template::themed_template]
Improved (broken) interface to define/manage groups over web interface
Allow to send as well mail, when membership was rejected
New functions [membership_rel::get_user_id], [membership_rel::get] and [membership_rel::get_group_id] to avoid code duplication
Added support to let user include %forgotten_password_url% in self-registration emails (e.g. in message key acs-subsite.email_body_Registration_password)
Improved subsite/www/members
- Make it possible to manage members of arbitrary groups
- Improved performance for large groups
- Improved configurability: when ShowMembersListTo is set to "3", show list to members only, when this is not the whole subsite
Improved user interface for /admin/applications for large number of applications
Various fixes for sitewide-admin pages (under /acs-admin)
Update blueprint in "install from repository" (currently just working in NaviServer)

SQL

Further cleanup of .xql files (like what as done for acs-subsite in OpenACS 5.9.0):
- 36 files deleted
- Removed more than 100 obsolete named queries
- Stripped misleading SQL statements
Marked redundant / uncalled SQL functions as deprecated
Replaced usages of obsolete view "all_object_party_privilege_map" by "acs_object_party_privilege_map"
Removed type discrepancy introduced in 2002:
- acs_object_types.object_type has type varchar(1000), while
- acs_object_types.supertype has type varchar(100)
- ... several more data types are involved, using acs_object_types.object_type as foreign key
Simplified core SQL functions by using defaults:
- Number of functions reduced by a factor of 2 compared to OpenACS 5.9.0 (while providing compatibility for clients using old versions),
- Reduced code redundancy
- Affected functions:
  - Reduced content_item__new from 12 versions to 6,
  - Reduce content_revision__new from 7 to 4
  - Similar in image__new, image__new_revision, content_item__copy, content_item__get_title, content_item__move
- PostgreSQL 9.5 supports named parameter in the same syntax as in Oracle. Further reduction of variants will be possible, once OpenACS requires at least PostgreSQL 9.5
Reduced usage of deprecated versions of SQL functions (mostly content repository calls)
Reduced generation of dead tuples by combining multiple DML statements to one (reduces costs of checkpoint cleanups in PostgreSQL)
Permission queries:
- Improved performance
- Support PACKAGE.FUNCTION notation for PostgreSQL to allow calls permission queries exactly the same way as in Oracle (e.g. "acs_permission.permission_p()"). This helps to reduce the number of PostgreSQL specific .xql files.
Modernized SQL:
- Use real Boolean types instead of character(1)
  (done for new-portal, forums, faq, attachments, categories, dotlrn, dotlrn-forums, evaluation)
- Use real enumeration types rather than check constraints (done for storage_type text/file/lob)

CR hygienics (reduce cr bloat)

Provided means to avoid insert/update/delete operations in the search queue:
OpenACS adds for every new revision often multiple entries to the search_queue, without providing any means to prevent this. This requires for busy sites very short intervals between queue sweeps (otherwise too many entries pile up). Another consequence is that this behavior keeps the PostgreSQL auto-vacuum daemons permanently active. Many of these operations are useless in cases where the content repository is used for content that should not be provided via search. The changed behavior should honors a publish-date set to the future, since it will not add any content with future publish dates to the search-queue.
Reduced number of insert cr_child_rels operations, just when needed:
cr_child_rels provide only little benefit (allow one to use roles in a child-rel), but the common operation is a well available in cr_items via the parent_id. cr_child_rels do not help for recursive queries either. One option would be to add an additional argument for content_item__new to omit child-rel creation (default is old behavior) and adapt the other cases.

Security improvements

Added support against CSRF (cross site request forgery)
- OpenACS maintains a per-request CSRF token that ensures that form replies are coming just from sites that received the form
- CSRF support is optional for packages where CSRF is less dangerous, and such requests are wanted (e.g. search and API-browser)
Added Support for W3C "Upgrade-Insecure-Headers" (see https://www.w3.org/TR/upgrade-insecure-requests/):
For standard compliant upgrade for requests from HTTP to HTTPS
Added support for W3C "Subresource Integrity" (SRI; see https://www.w3.org/TR/SRI/)
Added support for W3C "Content Security Policy" (CSP; see https://www.w3.org/TR/CSP/)
- Removed "javascript:*" links (all such urls are removed from the 90 packages in oacs-5-9, excluding js libraries (ajaxhelper) and richtext code)
- Removed "onclick", "onfocus", "onblur", "onchange" handlers from all .adp and .tcl files in the 90 packages in oacs-5-9 (excluding js libraries (ajaxhelper) and richtext code)
- Added optional nonces to all <script> elements with literal JavaScript content
Removed "generic downloader", which allowed to download arbitrary content items, when item_id was known (bug-fix)
Improved protection against XSS and SQL-injection (strengthen page contracts, add validators, added page_contract_filter "localurl", improve HTML escaping, and URI encoding)
Fixed for potential traversal attack (acs-api-documentation-procs)

Improvements for "host-node mapped" subsites

Fixed links from host-node mapped subsite pages to swa-functions (must be always on main subsite)
Made "util_current_directory" aware of host-node-mapped subsites
Added ability to pass "-cookie_domain" to make it possible to use the same cookie for different domains
Fixed result of affected commands "util_current_location", "ad_return_url", "ad_get_login_url" and "ad_get_logout_url" for HTTP and HTTPS, when UseHostnameDomainforReg is 0 or 1.
Improved UI for host-node maps when a large number of site nodes exists

Reform of acs-rels

Made acs-rels configurable to give the developer the option to specify, whether these are composable or not (default fully backward compatible). This is required to control transitivity in rel-segments
The code changes are based on a patch provided by Michael Steigman.
For details, see:
- https://openacs.org/forums/message-view?message_id=4031049
- https://openacs.org/forums/message-view?message_id=5330734

Improved status code handlers for AJAX scenarios

Don't report data source errors with status code 200 (use 422 instead)
Let "permission::require_permission" return forbidden (403) in AJAX calls (determined via [ad_conn ajaxp])

Improved Internationalization

Extended language catalogs for
- Russian (thanks to v v)
- Italian (thanks to Antonio Pisano)
- Spanish (thanks to Hector Romojaro)
- German (thanks to Markus Moser)
Added (missing) message keys
Improved wording of entries
Added message keys for member_state changes, provide API via group::get_member_state_pretty

Improved online documentation (/doc)

Fixed many broken links
Removed fully obsolete sections
Improved markup (modernize HTML)
Updated various sections

Misc code improvements:

18 issues from the OpenACS-bug-tracker fixed
Made code more robust against invalid/incorrect input (page_contracts, validators, values obtained from header fields such as Accept-Language)
Fixed quoting of message keys on many places
Improved exception handling (often, a "catch" swallows one to much, e.g. script_aborts), introducing "ad_exception".
Generalized handling of leading zeros:
- Fixed cases where leading zeros could lead to unwanted octal interpretations
- Switch to use of " util::trim_leading_zeros" instead of "template::util::leadingTrim", "dt_trim_leading_zeros" and "template::util::leadingTrim", marked the latter as deprecated
URL encoding
- "ad_urlencode_folder_path": new function to perform an urlencode operation on the segments of the provided folder path
- "export_vars": encode path always correctly, except -no_base_encode is specified
- Fixed encoding of the URL path in "ad_returnredirect"
Improvements for "ad_conn":
- Added [ad_conn behind_proxy_p] to check, whether the request is coming from behind a proxy server
- Added [ad_conn behind_secure_proxy_p] to check, whether the request is coming from behind a secure proxy server
- Added [ad_conn ajax_p] to check, whether the request is an AJAX requests (assumption: AJAX request sets header-field Requested-With: XMLHttpRequest")
- Added [ad_conn vhost_url] to obtain the url of host-node-mapped subsites
Added various missing upgrade scripts (missing since many years) of changes that were implemented for new installs to reduce differences between "new"-and "old" (upgraded) installations
Templating
- Get rid of various pesky "MISSING FORMWIDGET: ...formbutton:ok" messages
- Improved support for javascript event handlers in template::head
- New functions "template::add_event_listener" and "template::add_confirm_handler"
- Fix handling, when "page_size_variable_p" is set (was broken since ages)
Improved location and URL handling:
- Refactored and commented "util_current_location" to address security issues, handle IPv6 addresses, IP literal notation, multiple drivers, "
- Improved "security::get_secure_location" (align with documentation)
- New functions:
  - "util::configured_location"
  - "util::join_location", "util::split_location"
  for working on HTTP locations to reduce scattered regexps handling URL components
- Improved IPv6 support
- Use native "ns_parseurl" when available, provide backward compatible version for AOLserver
MIME types:
- Added more Open XML formats for MS-Office to allowed content types
- Modernized entries to IANA recommendations
- New function "cr_check_mime_type" centralizing the retrieval of the mime_type from uploaded content
Finalized cleanup of permissions (started in OpenACS 5.9.0):
- Get rid of "acs_object_context_index " (and therefore on "acs_object_party_privilege_map " as well) on PostgreSQL.
  Reasons:
  - huge table,
  - expensive maintenance, used only in a few places,
Misc new functions:
- "lang::util::message_key_regexp": factor out scattered regexp for detecting message keys
- "ns_md5" and "ns_parseurl": improve compatibility between AOLserver and NaviServer
- "ad_dom_sanitize_html": allow one to specify different sets of tags, attributes and protocols and "ad_dom_fix_html", which is a light weight tidy variant.
Improved HTML rendering (acs-api-browser), provide width and height to speed up rendering
Improved ADP files (e.g. missing doc(title))
Added usage of "ad_include_contract" on more occasions
Modernize Tcl and HTML coding
Reduced dependency on external programs (use Tcl functions instead)
Improved robustness of "file delete" operations all over the code
Improved documentation, fix demo pages
Aligned usages of log notification levels (distinction between "error", "warning" and "notice") with coding-standards
Cleaned up deprecated calls:
- Removed usage of deprecated API functions (e.g. "cc_lookup_email_user", "cc_email_from_party", "util_unlist", ...)
- Moved more deprecated procs to acs-outdated
- Marked remaining (and unused) "cc_*" functions as well as deprecated.
Improved Oracle and windows support
Fixed common spelling errors and standardize spelling of product names all over the code (comments, documentation, ...)
Many more small bug fixes

Version numbers:

require PG 9.2 (End Of Life of PostgreSQL 9.0 was Oct 2015)
require XOTcl 2.0 (presented at the Tcl conference in 2011).

Packages:

New Package Parameters
- acs-kernel:
  - MaxUrlLength: remove hard-coded constant in request processor for max accepted url paths
  - SecureSessionCookie: Let site admin determine, whether or not to use secured session cookies (useful, when not all requests are over HTTPS)
  - CSPEnabledP: activate/deactivate CSP
- acs-kernel (recommended to be set via config file in section "ns/server/${server}/>acs"
  - NsShutdownWithNonZeroExitCode: tell NaviServer to return with a nonzero return code to cause restart (important under windows)
  - LogIncludeUserId: include user_id in access log
- acs-api-browser:
  - ValidateCSRFP: make checking of CSRF optional (default 1)
- acs-content-repository:
  - AllowMimeTypeCreationP: Decides whether we allow unknown mime types to be automatically registered (default: 0}
- news-portlet:
  - display_item_lead_p: Should we display news leads in the portlet? (default 0)
- search:
  - ValidateCSRFP: make checking of CSRF optional (default 1)
- xotcl-request-monitor:
  - do_track_activity: turn activity monitoring on or off (default 0)

New OpenACS packages:
- richtext-xinha
- richtext-tinymce
- richtext-ckeditor4 (has ability to choose between CDN and local installation via GUI)
- openacs-bootstrap3-theme (as used on openacs.org)
- dotlrn-bootstrap3-theme

xotcl-core:
- Improved XOTcl 2.0 and NX support (e.g. api-browser)
- Added "-debug", "-deprecated" to ad_* defined methods (such as e.g. "ad_instproc")
- Make use of explicit "create" statements when creating XOTcl/NX objects (makes it easier to grab intentions and to detect typos)
- Added parameter to "get_instance_from_db" to specify, whether the loaded objects should be initialized
- Added support for PostgreSQL prepared statements of SQL interface in ::xo::dc (nsdb driver)
xowiki:
- Named all web-callable methods www-NAME (to make it clear, what is called, what has to be checked especially carefully)
- Moved templates from www into xowiki/resources to avoid naming conflicts
- Improved ckeditor support
- Added usage of prepared statements for common queries
- Improved error handling
- Better value checking for query parameter, error reporting via ad_return_complaint
- Added option "-path_encode" to methods "pretty_link" and "folder_path" to allow one to control, whether the result should be encoded or not (default true)
- Form fields:
  - Improved repeatable form fields (esp. composite cases), don't require preallocation (can be costly in composite cases)
  - Added signing of form-fields
  - Added HTML5 attributes such as "multiple" (for "file") or "autocomplete"
  - Fixed generation of "orderby" attribute based on form-field names
  - richtext: allow one to specify "extraAllowedContent" via options
  - Improved layout of horizontal check boxes
- Menu bar:
  - Added dropzone (requires bootstrap): drag and drop file upload
  - Added mode toggle (requires bootstrap)
  - Extended default policies for handling e.g. dropzone (file-upload method)
  - Distinguish between "startpage" (menu.Package.Startpage) and "table of contents" (menu.Package.Toc)
- Notifications:
  - Added support for better tailorable notifications: introduced method "notification_render" (similar to "search_render")
  - Added support for tailorable subject lines (method "notification_subject")
- Improved bootstrap support, use "bootstrap" as PreferredCSSToolkit
- Switched to ckeditor4 as PreferredRichtextEditor
- Improved handling of script-abort from within the payload of ::xowiki::Object payloads
- Added parameter to "get_all_children" to specify, whether the child objects should be initialized
xowf:
- Added property "payload" to "WorkflowConstruct" in order to simplify customized workflow "allocate" actions
- Internationalized more menu buttons
xotcl-request-monitor
- Added class "BanUser" (use. e.g. IP address to disallow requests from a user via request monitor)
- Added support for optional user tracking in database
- Added support for monitoring response-time for certain urls via munin
- Increased usage of XOTcl 2.0 variable resolver (potentially speed improvement 4x)
- Performed some refactoring of response-time handling to allow site-admin to make e.g. use of NaviServer's dynamic connection pool management (not included in CVS)
- Added support for partial times in long-calls.tcl to ease interpretation of unexpected slow calls
- last100.tcl: Don't report hrefs to URLs, except to SWAs
chat:
- Introduced new options to set chat rooms so login and/or logout message are not issued every time a user enters/exits a chat-room (important for chats with huge number of participants)
- Parameterized viewing of chat-logs
- Fixed cases of over-/under-quoting
- Fixed JavaScript for IE, where innerHTML can cause problems
file-storage:
- Don't show action keys, when user has no permissions
- Added support for copying of same-named files into a folder (adding suffix)
- Fixed old bugs in connection with "views" package

OpenACS Version 5.9.0 Agenda

Slimming pg SQL core:
- Part 1: improve performance of object deletion
  - remove manual delete operations from acs_object__delete()
- Part 2: content-repository - manual referential integrity management
  - handle referential integrity via pg's integrity constraints rather by functions cr_revision_del_ri_tr, cr_revision_ins_ri_tr, cr_revision_up_ri_tr, cr_revision_del_rev_ri_tr, and cr_revision_del_rev_ri_tr
  - fix broken/missing upgrade scripts from earlier updates
- Part 3: content-repository - manual deletions and nulling
  - Removed manual nulling of live_revision and latest_revision
  - Removed manual deletion of old_revision and new_revision in cr_item_publish_audit
  - Removed manual deletion of item_id in cr_item_publish_audit, cr_release_periods, cr_item_template_map, and cr_item_keyword_map
  - Removed manual deletion of direct permissions
  - Added missing index for child_id to cr_child_rels.
- Part 4: get rid of tree_sortkey in acs-objects
  - Check/fix dependencies in oacs-5-8 packages
  - Get rid of broken/uncalled functions using the column
  - Check/fix dependencies in other packages
  - Remove tree_sortkey and max_child_sortkey
Web interface:
- Improve client performance
  - moving core.js from head to body
  - provide kernel parameter to control expiration date for /resources/
- Protect against more XSS attacks
- Improved HTML validity (see oacs-5-9-html-validity for the checklist)
- Add lightweight support for ckeditor4 for templating::richtext widget (configurable via package parameter "RichTextEditor" of acs-templating. ckeditor4 supports mobile devices (such as iPad, ...).
- New kernel parameter ResourcesExpireInterval to control expiration dates of resources
Templating:
- Improve theme-ability
  - Move more information into theme packages in order to create responsive designs
  - Reduce hard-coding of paths, HTML etc.
- Dimensional slider reform (ad_dimensional):
  - Remove hard-coded table layout from dimensional slider
  - Add backwards compatible templates
  - Move hard-coded styles into theme styling
  - Remove obsolete comments from ad_dimensional
- Complete template variable controls (adding noi18n, addressing bug #2692):
  - @foo@: perform html quoting and internationalization
  - @foo;noquote@: perform internationalization
  - @foo;noi18n@: perform html quoting
  - @foo;literal@: perform neither html quoting nor internationalization
- Improved Russian nationalization
- Support of expiration dates and passwords for signed variables
Documentation:
- Use ACS templating for the (static) OpenACS documentation to provide a more consistent layout and user experience.
- Make pretty-naming of acs-core packages more consistent.
Misc improvements:
- Mark unused functions of acs-tcl/tcl/table-display-procs.tcl as deprecated
- Reduce number of muxtex locks by pre-request and per-thread caching
- Improved development und debugging aids:
  - use "ad_log error|warning .... " instead of "ns_log" to include information of request and callstack in error.log
  - ability to display ns_log entries caused by a request in ds-footer
  - ability to save delivered web pages in file-system for testing HTML validity (especially for admin pages, which are unaccessible for external validity testers)
- More bug fixes
Version numbers:
* require PG 9.0 (End Of Life of PostgreSQL 8.4 was July 2014)
* require XOTcl 2.0 (presented at the Tcl conference in 2011).

OpenACS Version 5.8 Agenda

PostgreSQL 9.2+:
- Get rid of nonstandard backslash escapes in function definitions
- Change quote syntax in SQL files (single quotes around the functions) to recommended PostgreSQL quoting using (recommended since pg8.0, jan 2005). li>Drop aliases in favor of named function arguments (recommended since pg8.0)
- Fix wrong function_args, add missing function_args, align default semantics with the defaults in pg (providing "null" as default means the argument is optional)
- Make OpenACS loadable without any tweaks in the pg config files
Use recursive queries for e.g. permission lookup to avoid performance problems in pg 8.4 and newer)
ADP: Use byte-compiled function wherever possible in compiled adp-code, support "@var;literal@" when neither quotes nor localization is needed in compiled adp-code
Improve support of NaviServer
Switch to Tcl 8.5 (TIP #143)
Improve scalability: Reduce mutex-stress on util-memoize cache and for cache maintenance in general
Code cleanup:
- Get rid of calls to deprecated code (e.g. ad_tables, ad_parameter, ... in acs-core and main packages)
- Improve awareness of usage of deprecated code (complain to error.log)
- Use Tcl 8.5 idioms
- cleanup of various http-client approaches and introduce a common implementation util::http::get and util::http::post; get rid of other usages, mark these as deprecated
- page-contracts: Perform checking of all ids in acs-core and main packages to improve error messages and to improve security
OpenACS 5.8.1 should be released with main packages

OpenACS Version 5.7 Agenda

Support for object management in core
Postgresql 9.0
TinyMCE update (fix for random JS injection issue, affecting Safari)
Fix for "remember me" issue
WCAG2-AA

OpenACS Version 5.6 Agenda

global parameters
package "embeds"
fix search by package_id
core works on Postgresql 8.4

OpenACS Version 5.5 Agenda

DONE: Postgresql 8.3 support: especially regarding tsearch2
DONE: acs-authentication:
- fix upgrade, add conditional logic into site wide tcl library so that you can login to perform the rest of the upgrade
DONE: tinymce:
- upgrade to 3.1.1 + language packs
- HTML Strict cleanup
- create appropriate parameters for its config in acs-templating
acs-mail-lite:
- DONE: cleanup duplicated procs (bounce)
- review the parsing of bouncing messages (case user_id 0)
- DONE: rollout support
Documentation improvements as discussed at the Guatemala conference:
- Make current openacs.org/test-doc source for static files included in the release and provide ease means to achieve this for the release manager
  - DONE (CVS HEAD): Provide in XoWiki an alternative table of contents by nested UL/LI (without JavaScript) for static output
  - DONE (CVS HEAD): Provide in XoWiki a prototype page similar to "book" without edit-buttons etc., using the new table of contents
- Update openacs.org/test-doc where necessary (incomplete list):
  - DONE: Fix the page ordering for the higher chapters (the original document had no 3rd. level numbering)
  - update pages in /test-doc which are more recent in openacs/xowiki
  - bump version numbers of OpenACS, where appropriate (some places talk about openacs-5-0, others about openacs-5-1, oacs-5-2-3rc1 or 5-3)
  - some version numbers of the required components are quite a mess. e.g. some parts say that Postgres 7.3 is required, some examples talks about postgres 7.4.7 and 8.2.4 in the same listing.
  - also the dotlrn version numbers are old dotrln-2.0
  - Tcl version numbers should be 8.4.19
  - The install section for XOTcl is missing in II.3.4
  - remove ChangeLog from documentation
  - find some other prominent place for the ChangeLog
  - Fix indenting in examples (e.g. in Rocael's robust web development framework)
  - overthink Win2000 guidelines. There are the native compiled packages from Maurizio, including everything from postgres, xotcl ....
- It is desired to find a single person responsible for overworking the documentation, however, funding is unclear.

OpenACS Version 5.4 Agenda

DONE: HTML Strict (openacs core)
DONE: finish template::head (daveb)
DONE: test acs-mail-lite (complex send)
DONE: test notifications (complex send)
DONE: new XinHA release, get rid of RTE & HTMLarea, test on Safari
DONE: Form builder: add the ID attribute to the form tag
DONE: acs-lang - keepLocalTranslationP to be removed
DONE search and intermedia-driver: move intermedia specific stuff to its package
DONE: acs-mail-lite - patch for mime::qp_encode bug

Future

Split Xinha and TinyMCE into separate packages see: https://openacs.org/forums/message-view?message_id=2750958
Usability ("my account" page)
XHTML ?
Testing and documentation for recording automated tests using the firefox plugin and the upload feature for it new in automated testing. Probably needs some polishing and should be talked to with Quest who are getting into this.
Parameter Scope Patch https://openacs.org/bugtracker/openacs/patch?patch%5fnumber=845
Remove obsolete master template stuff (default and site master template in openacs-4/www, acs-subsite's group-master, and related CSS and images). Probably in the version which follows 5.5 (probably 5.6). Also remove the compat master stuff at the same time.

Things to merge into this page

Old 5.0 Roadmap discussion

Roadmap discussion 1

Ideas for Boston 2006 Future of OpenACS discussion

A .LRN Roadmap

Another .LRN Roadmap discussion

What's on this page?

This page should include work that is planned on and has someone committed to working on it.

References to this Page: en:Interface___CSS_Coding_Guidelines, en:openacs-release-status, en:OCT-Meetings Categories: OpenACS Projects (OpenACS.org)

OpenACS 5.10.1 Change Summary

Created by Gustaf Neumann, last modified by Gustaf Neumann 24 Oct 2023, at 10:04 AM

Security:
- Stronger password hashes for OpenACS
  - New password hashes in addition to the classical "salted-sha1"
    - "scram-sha-256": SCRAM (RFC 7677) with parameter recommendation from RFC 7677; requires OpenSSL 1.1.1 and NaviServer 4.99.23 or newer
    - "scrypt-16384-8-1": SCRYPT (RFC 7914) with parameter "-n 16384 -r 8 -p 1"; requires OpenSSL 3.0 and NaviServer 4.99.23 or newer
    - "argon2-argon2-12288-3-1": Argon2 (RFC9106), Parameterization recommendation from OWASP: m=12288 (12 MiB), t=3, p=1; requires OpenSSL 3.2 and NaviServer 5.0 or newer
    - "argon2-rfc9106-high-mem": Argon2 (RFC9106), first (memory intense) recommendation from RFC 9106; requires OpenSSL 3.2 and NaviServer 5.0 or newer
    - "argon2-rfc9106-low-mem": Argon2 (RFC9106), second recommendation from RFC 9106; requires OpenSSL 3.2 and NaviServer 5.0 or newer
  - Preferences of the password hash algorithms can be set via kernel package parameter "PasswordHashAlgorithm", the first available algorithm is taken from the preference list, hash re-coding happens automatically at the next login.
  - See https://openacs.org/forums/message-view?message_id=5537869
  - Added optional CSP rules based on MIME types. This is important for user-contributed content. When users upload e.g. SVG-files to the file storage, and the content is served from there, it poses a potential security hole. One can now define an additional parameter called "StaticCSP" in the section "ns/server/$server/acs" of the OpenACS configuration file to deactivate execution of script files from static content.
    
    ns_param StaticCSP { image/svg+xml "script-src 'none'" }
  - Cookie-Namespace: When multiple OpenACS instances are served from the same domain name, the same cookies (e.g. ad_session_id, ad_login, ...) are set to all servers. For sensible cases, a cookie-namespace can be used, which can be used as a replacement of the traditional "ad_" prefix. This can be as well set in the section "ns/server/$server/acs" of the OpenACS configuration file
    
    # Provide optionally a different cookie namespace # (used for prefixing OpenACS cookies) ns_param CookieNamespace "ad_"
Improved templating:
- Client-side double click prevention
- Support for generic icon names, which can be mapped differently depending on the installed packages and themes: The generic names are supported via <adp:icon name="NAME" title=...>. By using this feature, one can use font-based icons (like e.g. glyphicons of Bootstrap5, bootstrap-icons, fa-icons, ...) instead of the old-style .gif and .png images. This makes the appearance more uniform, has better resizing behavior, and works more efficiently (fewer requests for embedded resources). Most of the occurrences of the old-style images in standard core and non-core packages in oacs-5-10 are already replaced.
- Support for listing registered URNs
Require NaviServer (i.e. drop AOLserver support).
Rationale: AOLserver cannot be compiled with the required modules with recent Tcl versions. Trying to backport NaviServer compatibility functions seems to be an overkill for the OpenACS project.
Further reduce divergence between Oracle and Postgres SQL. Target version of Oracle could be 12.*, as Extended support ends in 2022 (see https://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf)
- limit / rownum -> fetch first
- use Postgres schemas for stored procedures so that they can be invoked with the same Oracle idiom
Bootstrap 3 reached EOL in 2019, Bootstrap 4 had EOL 2022, so we should migrate to Bootstrap 5 (details: https://github.com/twbs/release)
New Packages:
- openacs-bootstrap5
- bootstrap-icons
- fa-icons
- highcharts
Potential incompatibility with OpenACS 5.10.0: "permission::permission_p" returns Boolean values as "t" and "f" and not "1" and "0". Avoid literal comparisons of the result and use boolean tests available in Tcl/OpenACS.
Support for fresh installations on Oracle 19c (for details, see: oacs-5-10-on-oracle-19c)
Require Tcl 8.6.2, XOTcl 2.1, PostgreSQL 12 (PostgreSQL 11 EOL: November 23), tdom 0.9

Categories: OpenACS Projects (OpenACS.org)

OpenACS 5.10.0 Change Summary

Created by Gustaf Neumann, last modified by Gustaf Neumann 07 Oct 2023, at 05:43 PM

Functional improvements
- Features:
  - Added additional page_contract filter: oneof(red|green|blue)
  - template::add_event_listener and template::add_confirm_handler now can target elements by CSS selector
  - Added text/markdown to the accepted text formats or rich-text widget
  - Support for range types in .xql files:
    PostgreSQL supports range types since 9.5. When using range types, square braces have to be used in SQL statements. Since OpenACS uses always Tcl substitution in .xql files, and OpenACS does NOT allow backslash substitution in these files, square brackets could not be escaped and therefore not be used in .xql files so far. This change allows now a developer to deactivate the substitution by passing e.g. -subst none to the db_* command using the .xql file. Valid values for -subst are all, none, vars, and commands, default is all which is exactly the behavior of previous releases. Therefore, this change is fully backward compatible.
  - Improved API browser: Visualization for code dependencies (which procs calls what, from where is a proc being called) and test-coverage
  - Warn site administrators about expiring certificates
  - Additional input types (and widgets) for ad_form:
    - checkbox_text
    - color
    - email
    - tel
    - url
    - number
    - file (multiple)
    - h5date and h5time: date and time fields using native HTML5 visualization and input normalization
  - Registry for .js and .css libraries: allow besides classical URLs symbolic names for loading external resources (e.g. jquery), this makes it easier to upgrade libraries in multiple packages (without running into problems with duplicate versions) and supports switching between CDN and local pathsURN. The existing implementation is based on URNs and extends the existing template-head API to support registration for URNs. A URN provides an abstraction and a single place for e.g. updating references to external resources when switching between a CDN and a locally stored resource, or when a resource should be updated. Instead of adding e.g. a CDN URL via template::head::add_script, one can add an URN and control its content from a single place. Use common namespaces for OpenACS such as urn:ad:css:* and urn:ad:js:*.
    - Register URNs:
      Example provider (e.g. in some theme):
```
   template::register_urn \ 
       -urn urn:ad:js:jquery \ 
       -resource /resources/xowiki/jquery/jquery.min.js
```
    - The registered URN can be used like classical URL after registration.
      
      Example consumer:
```
   template::head::add_javascript -src urn:ad:js:jquery
```
    - Declare composite files: Provide an interface to define that a .js file or a .css file contains multiple other .js/.css files in order to reduce the number of requests.
```
   template::head::includes -container urn:js::style.js -parts {urn:ad:js:jquery ...}
```
  - Improved support for streaming HTML: The new API function template::collect_body_scripts can be used to get the content of template::script or CSP calls (template::add_body_script, template::add_event_listener, template::add_body_handler, template::add_script) when streaming HTML (incremental HTML) is used. Before, these call could bot be used for streaming HTML.
- Reforms:
  - Dynamic blueprint reloading:
    - When reloading apm packages, watched files, etc. the NaviServer blueprint is now updated correctly. This solves the long-standing problem that changes for scheduled procedures required a restart of the server. Furthermore, the old-style reloading was based on a ever-growing list of reload operations each time a new reload was requested, causing slow-downs in long running server instances, especially, when new threads are started.
    - Now changes are applied to all NaviServer threads, including threads for schedules procedures, ns_jobs and the like. Note that the current job has to be finished before the changes can be applied.
    - For the transition, it is possible to switch between the classical reloading style and blueprint reloading by changing a variable in acs-tcl/tcl/apm-procs.tcl
  - Login:
    - Get rid of bugging "login page expired" messages. The 17 years old construct was replaced by newer means to avoid caching of form values from the login form. Admins of existing sites should set the kernel parameter LoginPageExpirationTime to 0
  - Forums:
    - Removed hard-coded dependency with registered_users group when checking forum permissions
    - Don't rely so heavily on acs_permissions to model forum configuration, as this can have unexpected consequences in convoluted multi-group/multi-subsite scenarios. Prefer simpler table attributes instead
    - New style of attachments to the forums, allowing multiple attachments to a single message directly from the message post page, using the multiple file input widget. Retain compatibility with old style attachments, using the new 'AttachmentStyle' package instance parameter. Currently, this supports two values: 'simple' (new behavior) and 'complex' previous behavior.
  - Chat:
    - Revamping of the GUI
    - Responsiveness
    - Full screen mode
    - Skins support (minimal, classic and bubbles, included): Skins are located in the new /packages/xowiki/www/resources/chat-skins/ directory. New skins can be created by just adding the css and js files in the skins directory, and naming them accordingly (chat-$SKIN_NAME.{js|css}).
    - Avatars (can be enabled per room)
    - Number of active users in chat
    - Tab notifications of new messages
    - Web Notifications:
      - https://www.w3.org/TR/notifications/
      - https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API/Using_the_Notifications_API
  - acs-lang:
    - admin pages:
      - Added the option to unregister (delete permanently the message key from all locales) a message key that has been already marked as deleted. Useful for cleaning up old message keys.
      - Added the option to undelete, using the new ::message::undelete proc.
      - Made number and category (untranslated/deleted/...) of messages coherent in all pages.
      - Added the columns 'total' and 'deleted' to the index page.
    - object_id reference: it is now possible to associate a message key to an object_id in a way that e.g. when the object is deleted, so is the message key. This addresses cases such as the message keys generated by group creation or by the new XoWiki localized fields
  - Notifications:
    - Improved scalability for notifications: One of the most expensive operations in large site is the cleanup for notification_requests in situations, where the user has lost permissions on an object, on which the user wanted to receive notifications. This check was performed previously in notification::sweep::cleanup_notifications via a permission check over all notification requests, which can be very costly on large sites. This change moves this cleanup into the actual notification sending, where the permissions have to be sent anyhow.
    - When sending a notification on behalf of a person, if the system is not configured to process replies to notification, do not set the reply-to address to anything different than the sender
    - Notifications: proper cleanup of acs_objects resulting from the deletion of dynamic notification requests
  - User/Person/Party API: rework and rationalize caching of all party, person and user API, create separate caches for each of these types, make the API and return dicts. acs_user::get will not fail anymore with non-existing user.
  - User Portrait: created API to retrieve and create, store and delete the user's portrait. Also address leftover child relationships from the past and delete them properly.
Non-functional Changes
- Improved automated regression test infrastructure and test coverage
  - All packages in the oacs-5-10 branch pass regression test
  - Web testing was separated from non-maintained tcltest and was built on the standard OpenACS infrastructure
  - Include web testing per default in standard regression testing
  - Introduced new test authentication authority, allowing to run many user administration tests outside the context of a "real authority": in cases where the real authority depends on external services to proof identity, (e.g. Kerberos), those tests would just fail.
  - Introduce the display of warnings in the UI
  - Added test coverage information in the automated testing pages, using the new proc-coverage API and providing test coverage information for packages and system wide.
  - Increased overall coverage of public API
  - New tests checking various data-model properties and smells
- Improved scalability:
  - Provided lock-free implementation of ad_page_contract_filters and ad_page_contract_filter_rules. This change improves parallel processing of requests and is primarily interesting for sites with a few mio page views per days. These locks were among the most frequent nsv locks
  - Reduced locks on util_memoize_cache my more invariants values into per-thread caching (acs_lookup_magic_object, ad_acs_version, .... ) and by avoiding specialized calls, which can be realized by already optimized ones (apm_package_installed_p_not_cached ref-timezones was most frequently used util_memoize_cache entry). These changes are necessary to avoid full still-stand on the unfortunate long-time locks on util_memoize_cache stemming from permission and user management with wild-card flush operations, which require to iterate over all cache entries (which might be on a busy server several hundred thousands)
  - Added new interface for cache partitioning to reduce lock latencies on high load websites
  - Added new interface for lock-free per-thread and per-request caching to avoid scattered ad-hoc implementations
  - Better reuse of DB handles (reduced expiring/reopen/etc.), faster access to handles
- Improved startup time:
  - When the package acs-automated-testing is disabled, startup time is reduced by avoiding loading of support functions and tests; the size of the blueprint is reduced
  - xowf: loading of at-jobs is significantly improved.
- Security improvements:
  - Strengthened page contracts
  - CSP support for application packages
  - CSP fine tuning
- Better exception handling based on Tcl 8.6 exception handlers (try and throw, also available in Tcl 8.5)
  - Provided a new ad_try implementation based on Tcl's try replaces now the old ad_try, with_catch and with_finally, which are marked as deprecated
  - The new ad_try is in essence Tcl's try but with predefined handling of ad_script_abort and should be also used instead of catch, when the OpenACS API is used (which might use script aborts)
  - All core packages use the new ad_try instead of the deprecated versions.
- Connection close reform:
  - NaviServer/AOLserver continue after connection closing commands to execute a script. This is in many situations not desired, especially, when for the page as well a .adp file exists, which will try to deliver this on the already closed connection. This can lead to errors in the error.log file, which are sometimes hard to analyze
  - Due to this cleanup, developers should use in most such cases cases ad_script_abort
  - Connection closing commands are e.g. ad_returnredirect, ad_redirect_for_registration, cr_write_content, ad_page_contract_handle_datasource_error, ad_return_string_as_file, ad_return_complaint, ad_return_error, ad_return_forbidden, ad_return_warning, ad_return_exception_page, ns_returnredirect, ns_return, ns_returnerror
  - The new version has made on most occasions explicit, when the script should abort.
- API changes (new and extended API calls):
  - New API call category::get to obtain category description for a category_id and locale
  - New utility ad_pad emulating both lpad and rpad typically available in DBMSs
  - New proc lc_content_size_pretty, prettify data size given in bytes. It supports three different standards (SI base-10, IEC base-2 and the old JEDEC base-2), default is SI base-10.
  - New flag -export for ad_form: this flag uses export_vars under the hood and supports all of this API's features (e.g. :multiple, :sign, :array). This addresses a long standing TODO
  - util::pdfinfo: simple poppler-utils wrapper to extract pdf information
  - util::http: leverage new ns_http features such as request file spooling. Native implementation will now be used only on NaviServer >= 4.99.15.
  - Database API:
    - db_foreach: queries executed inside of a db_foreach will not be issued using a different handle and will therefore be safe to use in a transaction
    - db_list_of_lists: new -with_headers flag, which will make the first element of the returned list to be the column names as defined in the query
  - Groups API:
    - Logics to delete a group type have now been included in the API
    - Allow to filter group members by member_state in the API
- Deprecated commands:
  - Many deprecated API calls were included in the code (and sometimes still in use) sometimes more than 10 years after these calls have been deprecated. In case a site modification still uses deprecated code, the user is warned about this. The OpenACS 5.10 code base does not depend on deprecated code.
  - Move deprecated code into separate files
  - Made loading of deprecated code optional (can be controlled via parameter "WithDeprecatedCode" in section "ns_section ns/server/${server}/acs" of the config file. By default, deprecated procs are still loaded
  - When deprecated code is not loaded, the blueprint of the interpreter is smaller. The following number of lines of code can be omitted when loading without the deprecated procs:
    - acs-tcl: 3178
    - acs-templating: 450
    - xotcl-core http-client-procs: 830
    - acs-content-repository: 1717 (including .xql files)
- Bugfix and Code Maintenance:
  - Made sure all party emails are stored as lowercase through the API
  - Fixed long standing regression in template::list: we were looping through the list "elements", rather than the "display_elements". This prevents specifying different sets of columns to be returned depending on the -formats and -selected_format options in template::list::create.
  - acs-content-repository: New HEIC and HEIF mimetypes
  - acs-mail-lite: handle to_addr specified as "DisplayName <email>" without errors
  - Fixed invalidating of all existing user logins, (aka) "Logout from everywhere" feature, useful e.g. to make sure no device still holds a valid login when we change our password on a device
  - Don't lose the return URL when one tries to join a subsite before being logged in
  - Added doc(base_href) and doc(base_target) for setting <base> element via blank-baster (see issue #3435)
  - Groups:
    - When a new group is created, flush all the group::get_id caches with the same name so that the new group can be fetched correctly in case it replaces a previously deleted one
    - Cleanup message keys coming from groups in acs-translations when a group is deleted
  - acs-lang:
    - lang::util::convert_to_i18n: do not always register a en_US translation, which would be always overridden. Instead, let lang::message::register make sure that a en_US message exists and create one only as a fallback.
    - lc_time_fmt: leverage Tcl clock to address shortcomings such as handling of dates in Julian/Gregorian calendar and impossible dates such as 1999-02-29, implement missing formats, support previously undocumented formats explicitly
  - search: make sure objects in the search indexer queue still exist by the time they are swept by the indexer (e.g. items deleted before the indexer could sweep them)
  - attribute::delete: fix proc so it leverages stored procedure capability of dropping the database table as well
  - util::http: fix UTF-8 encoding issues for some cornercases
  - Localization: Complete Italian and Spanish localization for the whole .LRN set of packages (including themes). Message keys for new and previously localized packages have also been updated
- General cleanup/maintenance
  - Improved handling of server implementation-specific code: server-specific code can be optionally loaded via specifying the server family in the filename. Provided *-procs-aolserver.tcl and *-procs-naviserver.tcl similar to *.postgresql.xql and *.oracle.xql where appropriate
  - Modernization of Tcl idioms.
  - Compliance of files, proc names, ... to the naming conventions.
  - White space cleanup, indentation changes.
  - Improvement of public API documentation
  - Adjustment of proc protection levels (public, private)
  - Adjustment of log severity
  - Cleanup of obsolete files
  - Replacement of handcrafted forms by ad_form
  - Typo fixing
  - Editor hints
  - Replacement of deprecated calls
  - Addition of missing contracts
  - ...
- SQL cleanup:
  - Cleanup of obsolete nonportable SQL constructs in a way Oracle and PostgreSQL code base divergency is reduced:
    - "nvl" -> "coalesce"
    - "sysdate" / "now()" -> standard "current_date" or "current_timestamp"
    - Use standard-compliant "dual" table where appropriate (required by Oracle, supported by PostgreSQL)
    - Use non-dialectal cast idioms when appropriate
    - Adopt CTE idioms in Oracle codebase as well (e.g. connect -> with recursive)
    - ... (reference Oracle version will be 11gr2 as is oldest version officially supported by Oracle (See here and here)
  - Reduced superfluous .xql queries
    - acs-subsite: delete 21 files with un-referenced .xql queries
    - acs-tcl: delete 4 files
    - news: 3 files
    - file-storage: 1 file
    - dotlrn: 9 files
- New Packages:
  - cookie-consent: alerting users about the use of cookies on a website
  - boomerang: performance of your website from your end user’s point of view
  - xooauth: OAuth implementation, including LTI (Learning Tools Interoperability)
  - dotlrn-bootstrap3-theme: Bootstrap 3 theme for DotLRN
  - xowf-monaco-plugin: Integration of Monaco editor with for code exercise types in xowf
  - proctoring-support: utilities and user interfaces to implement proctoring of the user session, mainly intended in the context of distance education and online exams. The main proctoring feature relies only on web technologies and does not require any plugin or additional software. Optional support for the Safe Exam Browser has also been introduced. The package is currently at the core of WU Online Exam infrastructure and is integrated in the inclass exam implementation for xowf.
Require Tcl 8.6, XOTcl 2.1, PostgreSQL 9.6 (PostgreSQL 9.5 EOL: February 2021), tdom 0.9

Categories: OpenACS Projects (OpenACS.org)

E-Mail: Outgoing E-Mail

Created by Malte Sussdorff, last modified by Gustaf Neumann 06 Feb 2023, at 04:48 PM

Current documentation for ACS Mail Lite, the standard API for inbound and outbound email is at https://openacs.org/doc/acs-mail-lite/.

Recommendation

The current recommendation for sending outgoing mail is via the NaviServer module nssmtpd, since this handles well large outgoing mails and works as well with >1000 file descriptors open. In OpenACS 5.10, set the package parameter "EmailDeliveryMode" in the acs-mail-lite package to "nssmtpd".

Older Discussion

Outgoing E-Mail at the moment in acs-mail-lite is split in a multitude of procedures, some doubling the efforts of the other. To clean this up, I propose the following (in general):

Replace acs-mail-lite::send with acs-mail-lite::complex_send, making it a wrapper for complex_send or the other way round (rename complex_send to send and so on).
Only support sending of e-mails via SMTP. Use smtp::sendmessage from TCLLIB for it (as does complex_send).
Daveb suggested splitting up complex_send to make it easier to test parts of it. Here are some ideas:
- Sub procedure to generate "to/cc/bcc/_lists" which are used in the respective TO/CC/BCC header. This will also clean up the sending of individual emails.
- Have only one call to smtp::sendmessage and one hook for the complex_send callback.
- Sub-procedure to append file tokens. Not sure if this is useful as we need to do upvar for the tokens and could not do automated tests on them anyway. So I'd not do it.
Have only one sweeper (the complex_sweeper) with support for multiple mail sending servers (so you can have multiple mail senders in the cluster instead of only one).
Split of incoming email handing into a separate file
Delete acs-mail-lite-procs.tcl :-). Just kidding, but deprecate most of the procedures.

Sadly, it is not a straightforward approach, as there is a catch. Using smtp::sendmessage forces us to figure out a new way for setting the bounce-email address header, as the old approach will not work any more (using the SMTP command "FROM" set to the bounce address). Another option is to use the non-standard "Errors-To" Header instead.

Categories: OpenACS Projects (OpenACS.org), Cookbook (OpenACS.org)

Dynamic Object Types and Attributes

Created by Malte Sussdorff, last modified by Gustaf Neumann 26 May 2021, at 12:17 PM

The goal of this page is to come up with a generic specification for an OpenACS core package that allows us to create object types using a WebUI and assign attributes to it. Furthermore, using this interface it should be possible to extend any other existing object type and seamlessly have it work in the respective pages without many changes needed.

At the moment three different approaches exist to tackle (part) of this problem. This needs to be modified so we only get one version out of it. The three approaches are:

AMS

ams is used by contacts as its primary storage for attributes. It's primary focus is to easily extend existing object types with attributes.

AMS stores the additional attributes in a meta table system from which it retrieves the values again. This allows flexibility as you can reuse attributes easily and you do not have to create database statements when you want to extend and object type. Additionally you do not need one table per object type and you are not limited by the number of rows a table can have as a maximum.

Dynamic Types

Dynamic Types is used by project manager to enhance the tasks and the project attributes. It is furthermore used in custom applications by cognovís, Solution Grove and xarg Ltd.

(DAVEB) In general, Lee and I also want to support,

Multiple storage models (generic, type-specific table)
Defining forms for multiple objects on one "page"
Defining forms that are not associated with an object

So I think we have the same ideas. This is where I think the form definition framework needs to be separate from the object type, so we can define a form and where the data goes, whether it is one object, multiple objects, or no object. We definitely should extend, and improve the existing acs_attributes table and features to support what we need.

Dynfields

Dynfields (formerly known as flexbase) is the dynamic attribute system used by ]po[ (formerly known as Project/Open) and based of AMS while trying to get the good things from dynamic types in it as well.

XoDynfields

This package is based on the xotcl object relational mapping and adds the the ability to create objects with dynamic slots. This also figures out multi-values and uses a relational database table for the storage of the multi-values (so not serialized in the id column).

The reason for this is that we need to run queries against the multi values on a regular basis like "show me all persons that are have the following car", where car is a multivalue (as persons can have more than one car, albeit with todays gazoline prices ....).

If we instantiate all persons from the database upon server start, how quick is XoTCL in searching for objects where a slot has a certain value, even if multi-valued. Additionally the question is, if I can quickly search for objects whose slot is a specific other object (assume car was not a multiple choice value, but multi-valued reference to the car class).

Storage

The attribute definition should be strored in AMS (AMS or ACS? DAVEB) attributes and the object types should be acs object types to make it easier and reuse existing functionality. Once we agree on a single system, the need for extension of ams_attributes tables done by the exisiting packages won't be needed anymore, as we could generically extend ams_attributes.

As for the attributes itself, we have two storage options. BOTH should be implemented and defined in the Attribute generation where it is actually stored.

Metadata Storage

This is the way AMS stores it's data. You have ams tables that hold the attributes and depending on the settings of the attribute the options (if multiple choice) or the plain value. Additionally support for address and telephone number object type exists. As you can see this actually allows a hierarchy of object types (an object type can exist out of multiple other object types and not only the ones provided by the database).

Plus

On the plus side is it's flexibility (no need to make changes to the database tables) and the ability to use subtypes.
Additionally you can quickly add and remove options from a multiple choice attribute.

Minus

A major drawback of this approach is the fact that the metadata tables can grow very large if you store all attributes in them (although in our CRM settings this so far was not an issue, knock on wood).
Additionally adding the attributes into database selects can take it's time (as the stored PL/SQL functions to retrieve an attribute are slower than just querying a field in a database table). On the other hand, if used with the caching mechanisms provided in AMS and using template::multirow::sort instead of having listbuilder sort in the database, things get considerably sped up.

Table( extension)s

Dynamic Types stores the attributes in the object_types table as defined in acs_object_types. Alternatively you could specify a different table to use for the storing of the attribute, although this will delute one of the main benefits of DT, the fast access to the data.

Plus

See AMS minus

Minus

See AMS plus :-).

API

The api should provide you with procedures to

Create a new object type
Extend an existing object type with attributes
For each attribute let you define where you want to store it and of what type the attribute is
Choose the widget for displaying the attribute
Retrieve the attributes value both in Tcl and pl/sql
Extend ad_from and template::list so that it automatically allows you to display all attributes
Support pages / ams_lists, to limit the display in ad_form and template::list to the attributes in the list
Save all attributes of an object_type at once (needed when we deal with subtypes)
Support for alternative storage areas for subtypes. E.g. postal-addresses is stored in the postal_address table. Same for ]po[ attributes.
Support for external storage areas. Especially in the corporate world they have their own applications storing data. Instead of synchronizing it manually, have acs-attributes use the external storage.

Pages

A page in the context here (and please change it if you have a better idea) is what AMS calls lists. It allows you to group multiple attributes per object type so they can be entered in one go. As an example in contacts you can use pages for each group a person (object_type) belongs in, so depending on the group membership, only certain attributes can be edited or displayed. Additionally permissions can be granted on a page basis, therefore allowing only admins to enter admin relevant data. Though in contacts we do have need to provide permissions on a per page/object level (e.g all users can edit their own data except for attributes xyz stored in the pages abc), I don't think this is useful in a general way.

A second use case are projects, where you can have additional attributes depending on the project type. Combined with workflow this gives you a powerful tool where you can have a project undergo various stages and in each stage other attributes are relevant to be filled out, though the project itself is never losing the value of one of it's attributes in the process.

Furthermore, pages should support headings and allow the setting of required and default values for attributes (so the required tag and the default value can differ for an attribute, depending on which page it is used).

Additionally we should think about support for using display, template::list and form templates for each page, so we could modify how a page of attributes is displayed to a user.

Last but not least, we might want to have pages support display and editing of attributes from multiple object_types on the same page.

User Interface

A user interface needs to exist so you can easily create a new object type and add attributes to existing object types. Furthermore support for creating pages needs to be there so you can limit the number of attributes displayed in a form/list and define the order in which they are displayed (form_entry_order, form_display_order, list_display_order, default_list_sort_order).

Template::list

Listbuilder would need to change so we could (optional setting in page) allow the user to define which elements of a page he wants to have in his list. Furthermore, the sort order needs to take into account that we might have to sort by an attribute that is stored in metadata storage, so we would have to sort the list e.g.with template::list::sort. Last but not least, to make it easy for users, we should not have to rewrite every template::list statement to allow the display of additional attributes, so dynamic attribute support should hopefully come out of the box. Okay, this might just be too much of a goal, especially taking into account that lists usually combine multiple object types into one.

ad_form

ad_form would need to be amended so it supports pages, meaning if called with a page name it will display all the attribute of the page. This is in addition to any elements already defined on the page, though for obvious reasons it should not display the same page name twice. If called with multiple pages, display the attributes of each page, beginning with the first in the list and making sure that you are not displaying a page twice. This is useful if you have in contacts a person in multiple groups (which relates to multiple pages) but you want to edit the attributes of all the pages of the groups he is a member in at the same time

Categories: OpenACS Projects (OpenACS.org)

E-Mail: Incoming E-Mail

Created by Malte Sussdorff, last modified by Gustaf Neumann 24 Aug 2020, at 01:43 PM

Incoming E-Mail in OpenACS works with the latest version of acs-mail-lite in a general fashion using callbacks.

The original version of this documentation is found via archive.org at: http://www.cognovis.de/developer/en/incoming_email

We will take a look on what needs to be done to get incoming e-mail working and then continue on to see how packages can benefit.

Project notes: ACS Mail Lite sends via SMTP which permits the use of an external server to handle email. For scalability, consider expanding the incoming E-mail paradigm to likewise use Tcllib's imap4 or NaviServer's nsimap so that most all email can be handled on separate servers.

Install incoming E-Mail

First, one must have an understanding of postfix basics. See http://www.postfix.org/BASIC_CONFIGURATION_README.html.

These instructions use the following example values:

hostname: www.yourserver.com
oacs user: service0
OS: Linux
email user: service0
email's home dir: /home/service0
email user's mail dir: /home/service0/mail

Important: The email user service0 does not have a ".forward" file. This user is only used for running the OpenACS website. Follow careful use of email rules by following strict guidelines to avoid email looping back unchecked.

For postfix, the email user and oacs user do not have to be the same. Furthermore, postfix makes distinctions between virtual users and user aliases. Future versions of this documentation should use examples with different names to help distinguish between standard configuration examples and the requirements of ACS Mail Lite package.

Postfix configuration parameters:

myhostname=www.yourserver.com

myorigin=$myhostname

inet_interfaces=$myhostname, localhost

mynetworks_style=host

virtual_alias_domains = www.yourserver.com

virtual_maps=regexp:/etc/postfix/virtual

home_mailbox=mail/

Here is the sequence to follow if installing email service on system for first time. If your system already has email service, adapt these steps accordingly:

Install postfix
Install smtp (for postfix)
Install metamail (for acs-mail-lite)
Edit /etc/postfix/main.cf
Edit /etc/postfix/virtual Add a regular expression to filter relevant incoming emails for processing by OpenACS.
@www.yourserver.com service0
Edit /etc/postfix/master.cf - uncomment this line so postfix listens to emails from internet
smtp inet n - n - - smtpd
Create a mail directory as service0
mkdir /home/service0/mail
Configure ACS Mail Lite parameters
BounceDomain: www.yourserver.com
BounceMailDir: /home/service0/mail
EnvelopePrefix: bounce

The EnvelopePrefix is for bounce e-mails only.

NOTE: Parameters should be renamed:
BounceDomain to IncomingDomain
BounceMailDir to IncomingMaildir
EnvelopePrefix to BouncePrefix
..to reflect that acs-mail-lite is capable of dealing with other types of incoming e-mail.

Furthermore, setting IncomingMaildir parameter clarifies that incoming email handling is setup. This is useful for other packages to determine if they can rely on incoming e-mail working (e.g. to set the reply-to email to an e-mail address which actually works through a callback if the IncomingMaildir parameter is enabled).
Configure Notifications parameters
EmailReplyAddressPrefix: notification
EmailQmailQueueScanP: 0

We want acs-mail-lite incoming handle the Email Scanning, not each package separately.
Configure other packages likewise
Invoke postmap in OS shell to recompile virtual db:
postmap /etc/postfix/virtual
Restart Postfix.
/etc/init.d/postfix restart
Restart OpenACS

Processing incoming e-mail

A sweeper procedure like acs_mail_lite::load_mails should:

scan the e-mails which are in the IncomingMaildir directory on a regular basis.
check if any email came from an auto mailer.
Parse new ones, and
process them by firing off callbacks.

Vinod has made a check for auto mailers by using procmail as follows. Maybe we could get this dragged into Tcl code (using regexp or a Procmail recipe parser) instead, thereby removing the need for setting up procmail in the first place.

Revised procmail filters:

:0 w * ^subject:.*Out of Office AutoReply /dev/null 
:0 w * ^subject:.*Out of Office /dev/null :0 w * ^subject:.*out of the office /dev/null 
:0 w * ^subject:.*NDN /dev/null :0 w * ^subject:.*[QuickML] Error: /dev/null 
:0 w * ^subject:.*autoreply /dev/null :0 w * ^from.*mailer.*daemon /dev/null

To make things granular a separate parsing procedure should deal with loading the e-mail into the Tcl interpreter and setting variables in an array for further processing.

ad_proc parse_email { 
    -file:required
    -array:required
} { 
   ...
}

An email is split into several parts: headers, bodies and files.

The headers consists of a list with header names as keys and their corresponding values. All keys are lower case.

The bodies consists of a list with two elements: content-type and content.

The files consists of a list with three elements: content-type, filename and content.

An array with all the above data is upvarred to the caller environment.

Processing an email should result in an array like this:

HEADERS

message_id
subject
from
to
date
received
references
in-reply-to
return-path
.....

X-Headers:

X-Mozilla-Status
X-Virus Scanned
.....

We do not know which headers are going to be available in the e-mail. We set all headers found in the array. The callback implementation then checks if a certain header is present or not.

        #get all available headers
        set keys [mime::getheader $mime -names]
 
        set headers [list]

        # create both the headers array and all headers directly for the email array
        foreach header $keys {
            set value [mime::getheader $mime $header]
            set email([string tolower $header]) $value
            lappend headers [list $header $value]
        }
        set email(headers) $headers

Bodies

An e-mail usually consists of one or more bodies. With the advent of complex_send, OpenACS supports sending of multi-part e-mails which are needed if you want to send out and e-mail in text/html and text/plain (for old mail readers).

switch [mime::getproperty $part content] {
     "text/plain" {
          lappend bodies [list "text/plain" [mime::getbody $part]]
     }
     "text/html" {
          lappend bodies [list "text/html" [mime::getbody $part]]
     }
}

Files

OpenACS supports tcllib mime functions. Getting incoming files to work is a matter of looking for a part where there exists a "Content-disposition" part. All these parts are file parts. Together with scanning for email bodies, code looks something like this:

        set bodies [list]
        set files [list]
 
        #now extract all parts (bodies/files) and fill the email array
        foreach part $all_parts {

            # Attachments have a "Content-disposition" part
            # Therefore we filter out if it is an attachment here
            if {[catch {mime::getheader $part Content-disposition}]} {
                switch [mime::getproperty $part content] {
                    "text/plain" {
                        lappend bodies [list "text/plain" [mime::getbody $part]]
                    }
                    "text/html" {
                        lappend bodies [list "text/html" [mime::getbody $part]]
                    }
                }
            } else {
                set encoding [mime::getproperty $part encoding]
                set body [mime::getbody $part -decode]
                set content  $body
                set params [mime::getproperty $part params]
                if {[lindex $params 0] == "name"} {
                    set filename [lindex $params 1]
                } else {
                    set filename ""
                }

                # Determine the content_type
                set content_type [mime::getproperty $part content]
                if {$content_type eq "application/octet-stream"} {
                    set content_type [ns_guesstype $filename]
                }

                lappend files [list $content_type $encoding $filename $content]
            }
        }
        set email(bodies) $bodies
        set email(files) $files

Note that the files ie attachments are actually stored in the /tmp directory from where they can be processed further. It is up to the callback to decide if to import the file into OpenACS or not. Once all callbacks have been fired files in /tmp will have to be deleted again though.

Firing off callbacks

Now that we have the e-mail parsed and have an array with all the information, we can fire off the callbacks. The firing should happen in two stages.

The first stage is where we support a syntax like "object_id@yoursite.com".

Second, incoming e-mail could look up the object_type, and then call the callback implementation specific to this object_type. If object_type = 'content_item', use content_type instead.

ad_proc -public -callback acs_mail_lite::incoming_object_email { -array:required -object_id:required } { }

callback acs_mail_lite::incoming_object_email -impl $object_type -array email -object_id $object_id

ad_proc -public -callback acs_mail_lite::incoming_object_email -impl user {

    -array:required

    -object_id:required

} {

    Implementation of mail through support for incoming emails

} {

    # get a reference to the email array

    upvar $array email

 # make the bodies an array

        template::util::list_of_lists_to_array $email(bodies) email_body

        if {[exists_and_not_null email_body(text/html)]} {

            set body $email_body(text/html)

        } else {

            set body $email_body(text/plain)

        }

        set reply_to_addr "[party::get_by_email $email(from)]@[ad_url]"

        acs_mail_lite::complex_send \

            -from_addr $from_addr \

            -reply_to $reply_to_addr \

            -to_addr $to_addr \

            -subject $email(subject) \

            -body $body \

            -single_email \

            -send_immediately

}

Object id based implementations are useful for automatically generating "reply-to" addresses. With ProjectManager and Contacts object_id is also handy, because Project / TaskID is prominently placed on the website. If you are working on a task and you get an e-mail by your client that is related to the task, just forward the email to "$task_id@server.com" and it will be stored along with the task. Highly useful :).

Obviously you could have implementations for:

forums_forum_id: Start a new topic
forums_message_id: Reply to an existing topic
group_id: Send an e-mail to all group members
pm_project_id: add a comment to a project
pm_task_id: add a comment to a task and store the files in the projects folder (done)

Once the e-mail is dealt with in an object oriented approach we are either done with the message (an object_id was found in the to address) or we need to process it further.

ad_proc -public -callback acs_mail_lite::incoming_email {
    -array:required
    -package_id
} {
}

array set email {}
            
parse_email -file $msg -array email
set email(to) [parse_email_address -email $email(to)]
set email(from) [parse_email_address -email $email(from)]

# We execute all callbacks now
callback acs_mail_lite::incoming_email -array email

For this a general callback should exist which can deal with every leftover e-mail and each implementation will check if it wants to deal with this e-mail. How is this check going to happen? As an example, a package could have a prefix, as is the case with bounce e-mails as handled in acs_mail_lite::parse_bounce_address (see below):

ad_proc -public -callback acs_mail_lite::incoming_email -impl acs-mail-lite {
    -array:required
    -package_id:required
} {
    @param array        An array with all headers, files and bodies. To access the array you need to use upvar.
    @param package_id   The package instance that registered the prefix
    @return             nothing
    @error
} {
    upvar $array email

    set to [acs_mail_lite::parse_email_address -email $email(to)]
    ns_log Debug "acs_mail_lite::incoming_email -impl acs-mail-lite called. Recepient $to"

    util_unlist [acs_mail_lite::parse_bounce_address -bounce_address $to] user_id package_id signature
    
    # If no user_id found or signature invalid, ignore message
    # Here we decide not to deal with the message anymore



    if {[empty_string_p $user_id]} {
        if {[empty_string_p $user_id]} {
            ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: No equivalent user found for $to"
        } else {
            ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Invalid mail signature $signature"
        }
    } else {
        ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Bounce checking $to, $user_id"
        
        if { ![acs_mail_lite::bouncing_user_p -user_id $user_id] } {
            ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Bouncing email from user $user_id"
            # record the bounce in the database
            db_dml record_bounce {}
            
            if {![db_resultrows]} {
                db_dml insert_bounce {}
            }
        }
    }
}

Alternatively we could just check the whole to address for other things, e.g. if the to address belongs to a group (party)

ad_proc -public -callback acs_mail_lite::incoming_email -impl contacts_group_mail {
    -array:required
    {-package_id ""}
} {
    Implementation of group support for incoming emails
    
    If the to address matches an address stored with a group then send out the email to all group members

     @author Malte Sussdorff (malte.sussdorff@cognovis.de)
     @creation-date 2005-12-18

     @param array        An array with all headers, files and bodies. To access the array you need to use upvar.
     @return             nothing
     @error
} {

    # get a reference to the email array
    upvar $array email

    # Now run the simplest mailing list of all
    set to_party_id [party::get_by_email -email $email(to)]
    
    if {[db_string group_p "select 1 from groups where group_id = :to_party_id" -default 0]} {
        # make the bodies an array
        template::util::list_of_lists_to_array $email(bodies) email_body
        
        if {[exists_and_not_null email_body(text/html)]} {
            set body $email_body(text/html)
        } else {
            set body $email_body(text/plain)
        }
        
        acs_mail_lite::complex_send \
            -from_addr [lindex $email(from) 0] \
            -to_party_ids [group::get_members -group_id $to_party_id] \
            -subject $email(subject) \
            -body $body \
            -single_email \
            -send_immediately

    }
}

Or check if the to address follows a certain format.

ad_proc -public -callback acs_mail_lite::incoming_email -impl contacts_mail_through {
    -array:required
    {-package_id ""}
} {
    Implementation of mail through support for incoming emails
 
    You can send an e-amil through the system by sending it to user#target.com@yoursite.com
    The email will be send from your system and if mail tracking is installed the e-mail will be tracked.

    This allows you to go in direct communication with a customer using you standard e-mail program instead of having to go to the website.

    @author Malte Sussdorff (malte.sussdorff@cognovis.de)
    @creation-date 2005-12-18
 
    @param array        An array with all headers, files and bodies. To access the array you need to use upvar.
    @return             nothing
    @error
} {
    # get a reference to the email array
    upvar $array email

    # Take a look if the email contains an email with a "#"
    set pot_email [lindex [split $email(to) "@"] 0]
    if {[string last "#" $pot_email] > -1} {
       ....
   }
}

Alternatives to this are:

${component_name}-bugs@openacs.org (where component_name could be openacs or dotlrn or contacts or whatever), to store a new bug in bug-tracker
username@openacs.org (to do mail-through using the user name, which allows you to hide the actual e-mail of the user whom you are contacting).

Cleanup

Once all callbacks have been fired off, e-mails need to be deleted from the Maildir directory and files which have been extracted need to be deleted as well from the /tmp directory.

References to this Page: en:openacs-quick-install Categories: OpenACS Projects (OpenACS.org), Cookbook (OpenACS.org)

E-Mail: Event Handling

Created by Dave Bauer, last modified by Gustaf Neumann 24 Aug 2020, at 01:43 PM

Sending email on certain events in OpenACS/.LRN is done very haphazardly. This needs to be rewritten so there is a simple way to figure out when an email will be sent, and allow proper handling of user preferences, administrative parameters, and customization.

There are cases where the system (OpenACS or .LRN) needs to send out email, for example, when a new user joins, requests a password reset, or is added to a subsite or .LRN community.

Right now there isn't any system-wide way to mange this email. In some cases, the administrator is notified an email will be sent and is given the option to edit the email before it is sent, but there is no one way this is done. There are several pages that call ns_sendmail explicitly, or acs_mail_lite::send explicitly. There are more places this happens in .LRN. Unfortunately there is also a "magic" place where email is sent that is totally unexpected. Inside the dotlrn_community::membership_approve procedure, there is a call to dotlrn_community::send_member_email, which will send an email to the user when the membership is approved, if 1) a parameter is set and 2) an administrator has created and enabled an email message to be sent.

In addition there is a email sent using the "spam" package in dotlrn/www/admin/users-add-to-community where a dotlrn sitewide administrator can add users to a community, and the users are automatically emailed. In this case the administrator is not notified that an email is sent, or given an opportunity to customize or suppress the email. This causes problems when an administrator attempts to fix a problem by adding a user to a community automatically, and the user is sent a confusing message.

The dotlrn package allows for a custom email to get written by the community admin for each community, but the admin is not allowed to choose if the email is sent when an individual member is added. It is either on or off, always sent, or never sent.

dotlrn-ecommerce extends this by adding several more events for application submission, approval, rejection, etc. And admin can edit these emails on a sitewide or per community basis. In most cases the email is automatically sent, in one o r two cases the admin can edit the email, but not suppress it.

This leads to unwanted email. Often an admin must manually add or remove someone from a subsite or community or otherwise handle a problem. This can lead to welcome emails being sent at the wrong time, confusing the users.

A system wide solution would allow packages to create events where email is sent by the system (besides subscribed notifications). This solution would provide an includable interface for creating and editing a default email message for the events. It would also provide an interface to notify an admin that an email will be sent, giving options to suppress the email or edit the content of the email before sending it.

The beginnings of this feature exist in the dotlrn_member_emails table, dotlrn_community::send_member_email procedure, and the dotlrn-ecommerce package which has a few pages that replicate this interface, but probably needs more generalization to allow working with subsites as well as dotlrn communities. There is also the reusable include for editing the default emails under dotlrn/lib/member-email

This proposal would provide a comprehensive solution for handling system level email events, allowing admins to know when an email is sent, and provide a consistent user interface to manage the emails.

MS:

You could achieve this in a general way by using the acs-lang interface at least for the subject and body. For each object_id you would create a new message key, e.g. acs-translations.welcome_email_subject_${object_id} and acs-translations.welcome_email_body_${object_id}. If you have multiple emails per community / subsite, you would rename them to acs-translations.confirmation_email_subject_${object_id} aso. A general interface would then be provided to look for all language key combinations of acs-translations.email_xxx, allowing you to edit the messages for all communities and subsites. If you want to edit them for only one, then you can look for all who have the same object_id.

A default message would be given with acs-translations.email_subject_welcome, which the mail sending could default to in case no specific language key exists for the community / object_id. Furthermore, acs-translations.welcome_email_help message key is present describing what this email is about. A package like dotlrn would register the three default e-mail keys to start off with this and then the email-handling package can do the rest (e.g. with email-handler::send -to_party_ids -from_addr -email_type "welcome" -object_id).

This approach has the major advantage that you have internationalization by default.

Alternatively you could mimic the message handling done by contacts which allows you to have multiple message types, e.g. email, which you can fill in with default values, which has it's own I18N by having a locale stored in the DB table. But if you ask me, the acs-translations idea sounds better to me :). Though, you can obviously do this with your own tables as well, but you would loose on the nice features acs-lang has to offer.

Files that currently could trigger email

add_user calls add_user_to_community

add_user_to_community calls membership_approve (if applicable). membership_approve calls send_member_email unconditionaly

so any call to add_user or add_user_to_community could result in a call to send_member_email

dotlrn_community::add_user
dotlrn/www/admin/add-instructor-3.tcl
                 users-add-to-community.tcl
                 member-add-3.tcl
                 members-chunk-table.tcl
                 members.tcl
                 register.tcl

dotlrn-ecommerce/www/admin/gwu-section-new.tcl
                     ecommerce/shopping-cart-add.tcl.backup
                     register/index.tcl                 
                tcl/implementation-procs.tcl
                
dotlrn_community::add_user_to_community
dotlrn/tcl/class-procs.tcl
           club-procs.tcl
           community-procs.tcl
           dotlrn-callback-procs.tcl
           
dotlrn_community::membership_approve
dotlrn/www/approve.tcl
dotlrn/www/admin/commmunity-members-add-to-community.tcl
                 users-add-to-community-email.tcl
dotlrn-ecommerce/www/admin/application-approve.tcl

References to this Page: en:dotlrn-ecommerce, en:dotlrn Categories: OpenACS Projects (OpenACS.org), Cookbook (OpenACS.org)

Site Wide Image Upload Widget

Created by Dave Bauer, last modified by Gustaf Neumann 01 May 2020, at 04:50 PM

MGH and Solution Grove are collaborating on a new image upload widget that will hopefully be easier to use and more easily integrated into any Xinha text element.

Here is how it will work. The user will be editing the content of an object using a rich text editor (XINHA). If they decide an image or reference to a file is needed in the content, they can click the attach-file or attach-image icon. A popup window appears allowing the user to choose a file to upload. The file is uploaded and stored in the content repository. If it is an image a thumbnail will be generated and linked to the full size image. When the content of the object is saved, the content is scanned for references to files or images, and a link is stored in the database that the particular object is "using" a file or image. This is the simplest case. Some more complex cases include allowing the user to search or browse images they have previously uploaded in a popup window to choose and image to insert.

The use case we are trying to solve:

In assessment, right now an admin can associate one file or image with a question. The admin can already enter HTML for the question description. We need to allow image upload from Xinha so the admin can add an arbitraty number of images to one question description. We need this functionality in any richtext widget within the assessment creation process.

Requirements:

upload image without thinking about where it will go, just entry optional title, description and choose a file from your Desktop
image will have parent_id=package_id where its created. if an object is used in multiple pages, we will keep track of that and let the owner see where it is used. We'll need to write a parser to extract /image/ url links. (DONE uses application_data_link feature from acs-tcl)
image will inherit permissions depedning on the setting of the user. In general it'll inherit from the package_id or the subsite where its created.
- To avoid namespace conflicts we can do someting like this for the cr_items.name
  - - cr_items.name = '{$item_id} {$filename}' or automatically rename the item filename(1) (2)... (DONE, we strip the item id on display)
- Image may be "private" which means it does not inherit permissions from any other object. This means the only person who can see it is the owner. If the image is uploaded within the context of another object, ie Xowiki page, a link is created between the page and the image. If someone has permission to view he page, they can also see the image. We make this work by creating a special URL /image/${image_id}/private/${xowiki_page_id}/${filename} This generated a unique URL that includes the "viewing context". When this URL is accessed 1) the system is checked to make sure a data link exists, that is, that the image is used "in" the viewing context object, and that the viewing user has permission to see the viewing context object.
User uploading image will be granted direct "admin" permission over the image DONE
images will have a centralized URL for delivery, ie: an acs-subsite/www/images.vuh similar to o.vuh so urls will appear to be /images/${item_id}/image-filename.jpg (for example) (Thanks to gustaf to notice the obvious namespace conflict with just using filename) DONE
images are stored as image type in the content repository (stored with "file" storage type) DONE
thumbnails will be generated (propose to use the image-magick package, I have some code to contribute that will process an image upload and generate a thumbnail (this will be disabled if image-magick package is not installed)
- A new image thumbnail API is DONE

Once we have this in place, we have a simple, site-wide solution to upload image attachments to any object. This can be used in places like forums etc, and solves the problem of figuring out how to configure each Xinha widget in every package to find a place to store images. By attaching the image directly to the object we solve the issue of finding a place to store images.

Future Ideas

These ideas are not planned for an initial implementation, but definitely are on our TODO list

Add a UI to browse images you uploaded
Add a UI to allow users to browse any images they can read?
Add a way to search metadata on images when choosing an image so when you choose the add image button in Xinha, it will allow you to search existing images you have uploaded
Add a recently viewed/uploaded feature to the image picker, so you can choose an image you recently viewed or uploaded without searching for it
- Here is an example of what that could look like (actual working code exists for this now)
Add a clipboard feature to the image picker, so you can use the optional clipboard package to mark an image in your clipboard while viewing it, then choose it from the clipboard to insert a link into another object.

References to this Page: en:Forums Categories: OpenACS Projects (OpenACS.org)

Documentation Project Discussion

Created by OpenACS community, last modified by Gustaf Neumann 06 Oct 2019, at 12:54 PM

Current topic: What approach should we use to upgrade the documentation?

Here are some recent approaches expressed in one form or another for managing the documentation in the context of "the plan":

Approach 0. Why not use docbook, which was the previous way documentation was being handled?

docbook is open-source.
A growing community surrounds DocBook (has mailing lists)
A number of free and commercial tools are available for editing and publishing DocBook documents.
docbook enables us to publish in a variety of formats.
XML separates content from presentation: It relieves each contributor of the burden of presentation, freeing each writer to focus on content and sharing knowledge.
docbook is well tested technology. It has been in development since the early 1990's).

problems: In 2002, Docbook still was not fully capable of representing online books as practiced by book publishers and expected from readers with regards to usability on the web. That meant DocBook did not entirely meet OpenACS publishing requirements at that time.

In 2004, Docbook released version 4.2, which complies with all the OpenACS publishing requirements. Producing a web friendly book hierarchy arguably remains DocBooks' weakest point. For example, a dynamically built document should be able to extract details of a specific reference from a bibliographic (table) and present a footnote at the point where referenced. DocBook 4.2 allows for this with bibliocoverage, bibliorelation, and bibliosource. Yet, OpenACS documentation does not follow a standard book hierarchy since most of the documentation was written before version 4.2, and re-organizing it in docbook source would be challenging.

Approach 1. from en:Proposed_project_goals

Robert writes:

- First: ..attempt to take the rest of Documentation over to XoWiki..

- Secondly: ..try to setup an automated versioning system. We should end up with categories such as 5.2 Documentation, 5.3 Documentation, HEAD Documentation, etc. My current thinking is that we can work on HEAD category of documentation, once 5.3 is release it becomes categorized as such and a copy of the docs gets created and re-categorized as HEAD once again. This should allow versioning and easy upgrades/editing of docs (well easy may not be the right word, docs are a lot of work)
Robert, "Secondly" is how versioning has been accomplished using docbook. This method seems to work fine, and we can do it with xowiki docs by creating a set of static pages from the xowiki ones. --Torben

Approach 2

Robert writes:

Documentation: [move] ..the rest of the documents to XoWiki. The idea would be to have categorized documents. We would start by moving the 5.2 docs over and expanding on them. When 5.3 is release we would do an automated copy/paste of the 5.2 docs, re-categorize as 5.3 and start the editing process. This is just preliminary thinking at this point..
Robert, everyone seems to have their own way of slicing and dicing docs into categories. We ought to use the existing documentation requirements to guide how the documents are organized, and then they can be categorized any number of ways since multiple categories can be applied to each page. --Torben

Approach 3 (and previously 5). Refactoring original docbook docs en:New_Documentation_Process

STAGE A: CONVERT DOCUMENTATION TO XOWIKI (note: all api docs remain the same) Step #1. Catalog the current documentation.

(Malte writes) ..modify the script Gustaf provided to import the whole documentation in one go into a new XoWiki instance with the structure (page_order) that has been added in XoWiki 0.42 taken from the chapters of the documentation so that we do have an exact mirror of the documentation as it stands now. [Done, see https://openacs.org/test-doc].

Malte, why would we want an exact mirror of something that is not organized well.. too many topics per page and pages inconsistently presented.. requires a new reader to jump around to get familiarized with material. etc etc? Why not copy the docbook contents into a cleaner outline and work from there (as in approach 3 above)? --Torben

Torben, the /doc section is organized. That you do not like it is obvious and we could rework it later, but until we have the resource to rewrite the whole documentation, it makes much more sense to improve the documentation we have instead of putting it into the graveyard. And we need to come to terms. This discussion does not yield any results at the moment but keep us from doing the actual work: Improving the documentation -- Malte

[Then] ..assign categories to the documentation, allowing for an alternative view on the documents (so you could say "instead of showing the whole documentation only show the documents for a specific category"). Probably this needs some more detailed discussion with Gustaf finding out how this could be achieved in XoWIKI and what would make most sense. Ideally we could provide a different structure based on the target group (e.g. category) but this is probably shooting too far. Getting categorization and page ordering in a decent shape should provide us a lot of possibilities..

Malte, have you seen docs-admin-toc , docs-end-user-toc and docs-eng? These are outlines of existing pages in XoWiki that represent a revised version of the Table of Contents (TOC) in the docbook version. Feel free to propose new pages there for us to fill in content. --Torben

Yes I have seen them. Do they resemble a book in any way to you? They are alternative structures, indeed, but you can impose them on a book view as well, any time. A book is what we need, something people can go to and start reading. If the book starts with four different pages, each outlining a different reading path, even the better. But a book it should be nevertheless, because this is how people still learn. If you do not like the book approach, that is fine, then we should open this question up for a TIP. My main goal at the moment is to finally get this done and start working. And I want to get rid of the myriads of confusing advise given at openacs.org. I have someone to work with me on that in the next two months and I want to have a clear way to go forward. So I will just TIP this. -- Malte

Approach 4 Mental Maps en:Documentation_Project_Plan

This approach was originally posted at en:Documentation_Project. It was moved here as the topic expanded.

..port docbook pages to xowiki manually.. look at each part in detail.. separate to subsystems and how they are used (context). Why?

"..the human mind can only deal with a relatively small number of independent pieces of data at one time, but if data are chunked together in appropriate ways, the mind can perform higher order abstractions, and these in turn can be chunked together, with successive abstractions, until an entire complex situation is encompassed. The systems approach addresses this property of the human mind by providing strategies for the data gathering, chunking, and abstracting process." George G. Lendaris, On Systemness and the Problem Solver: Tutorial comments 1983.

A short video on how the mind deal best with large amounts of information by "chunking": The Science of Thinking

This work is in progress, with root documentation page here: en:openacs-handbook

A systems strategy of multiple perspectives has these rules:

Each xowiki page discusses a single topic.
Topics are linked together by any number of other xowiki pages to present an ordered presentation of the topics with a common thread/topic connecting them. For example, en:openacs-system-install is a page that links together the topics of installing the component software of OpenACS. Similarly, each component software, such as en:aolserver, has its own view of some overlapping topics.

Multiple perspectives meets these significant documentation requirements:

helps identify subsystems and how OpenACS works --becomes a natural tutorial without more words.
reduces the burden of keeping documentation up to date since there is only one place to put relevant information for a particular topic --no redundancy
pages are not organized by a dominant category morphology that tends to address the perspectives of just a few people. Most any perspective can be represented.
Readers do not have to filter out a bunch of information that is irrelevant to them or their task at hand.

Move all but maybe the first and last 2 items from https://openacs.org/doc/dev-guide.html to https://openacs.org/doc/acs-kernel/ (and what ever else is relevant to kernel only); and move the first item to https://openacs.org/doc/acs-admin etc. That way the core docs are presented in a consistent context with the other packages. Also, do not migrate these docs around as a package is designated part of the core (or subsequently removed from it). This would help developers see appropriate context (and meets one of the documentation requirements).

Allow documentation to link directly to the api-docs, to reduce redundancy and links go to current, local API docs. In other words, https://openacs.org/api-doc/package-view?version_id=358136 becomes: /api-doc/index?about_package_key=acs-datetime The feature has been added to OpenACS 5.3 so will be released soon.[DONE]

Move Administrator's Guide to the xowiki [in progress, see en:docs-admin and en:docs-admin-toc ], because this section:

has the most duplicated work (topics overlap on various pages). For example, more than one page explains how to add a package, how to restart the server, how to start the server etc.
needs to be updated most frequently because of changing installation requirements. For example, PostgreSQL requires different instructions for different revisions, external links change sporadically etc.

Incorporate the work already done in the first wiki ( https://openacs.org/wiki ), where volunteers have already added a wealth of new documentation. Note that some of this will already exist in xowiki from previous importing of docs etc. [TO DO]

We need to get rid of the myriads of different installation instructions. First of all they are not kept up to date (all of them). -- Malte

References to this Page: en:Proposed_project_goals, en:Documentation_Project, en:Documentation_Project_Plan, en:New_Documentation_Process Categories: Documentation (OpenACS.org), OpenACS Projects (OpenACS.org)