Forum OpenACS Development: ANNOUNCE: NaviServer 4.99.16 available

Dear all,
I am pleased to announce the availability of NaviServer 4.99.16 (see [1] and [2]). NaviServer 4.99.16 was tested with the newest version of OpenACS and it runs already on openacs.org.
Below are the Release Notes describing the changes since 4.99.15:
best regards
-gustaf
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.16/
[2] https://bitbucket.org/naviserver/naviserver/overview
=======================================
NaviServer 4.99.16, released 2017-12-29
=======================================

 345 files changed, 16488 insertions(+), 10401 deletions(-)

New Features:
-------------

  - ns_cache improvements:

    * Allow runtime reaction and re-configuration of caches
      (e.g. grow/shrink a cache via new cmd ns_cache_configure)

    * Added cache transaction semantics

      Background: When ns_cache_* commands are used within a database
      transaction (e.g. in OpenACS), it might occur, that partial
      results of the transaction are cached before the transaction is
      committed.  When the transaction is rolled back, invalid values
      might be kept in the stack leading to erroneous and hard to
      debug behavior.  Furthermore, information about changes might
      leak into other concurrent threads via the cache, even before
      the transaction is committed.

      The new cache transaction semantics is implemented via the three
      commands

	 - ns_cache_transaction_begin
	 - ns_cache_transaction_commit
	 - ns_cache_transaction_rollback

	When no ns_cache_transaction* commands are used, the behavior is
	exactly as before.

	When ns_cache_transaction* commands are in use, the following
	functionalities become available

	 - The ability to rollback of the values since the matching
	   ns_cache_transaction_begin

	 - Isolation of behavior: cached values from incomplete cache
	   transactions are just visible from the current thread, but
	   not from other threads.

	 - Nesting: transactions can be nested
	   (up to a compile time constant, per default: 16)

	 - Introspection: the statistics about cache commits and
	   rollbacks are included in the cache statistics.

    * Support caches sizes larger than 2GB

    * Summary of new ns_cache* commands
      . ns_cache_configure to change cache parameters at runtime
      .	ns_cache_exists: This command is is more than an order of
	magnitude faster than [expr {"foo" in [ns_cache_names]]
      . ns_cache_transaction_begin
      . ns_cache_transaction_commit
      . ns_cache_transaction_rollback


  - Virtual server improvements

    Make mapping of host entries in the virtual server definition more
    intelligent to avoid newbie gotchas and hard to find
    mis-configurations (entries in e.g. nssock/servers)
     a) add automatically an entry with the port, when non is given
     b) complain, when driver is not listening on the specified port
     c) add automatically an entry without the port, when the driver
       listening on the default port.
    Old configurations (doing a-z manually) should continue to work

  - Improved integration of NaviServer with e.g service files (systemd):
    When the server is starting in a forking mode, return a non-zero
    return code, when initialization of the server went wrong (e.g.
    error in the config file)


  - nsdb:
    provide interface for obtaining session_ids. When implementing
    e.g. prepared statements over the nsdbpg driver, the prepared
    statement is just valid for one session. by providing the
    session_id on the Tcl layer, prepared statements can be provided
    selectively on the Tcl level.


  - https:
    * Support OpenSSL 1.1.0 (configuration and runtime)
    * Support LibreSSL 2.6.3 and 2.6.4
    * Improved support for ancient versions of OpenSSL (e.g. in CentOS 5.0)
    * Added support for for client side Server Name Indication (SNI)
    * Report attempted TLS handshakes on plain connections to error.log


  - nscgi: new config parameter "allowstaticresources"

    New parameter "allowstaticresources": provide control on whether
    static resources can be delivered from a cgi-bin directory.

    Background: When CGI scripts are called from a cgi-bin directory
    without an explicit extension mapping to a script interpreter
    (e.g.  Perl, Bash, ...) the script has to be set executable. If
    this is not the case, the CGI script is delivered in source code,
    which might reveal unwanted information. The reason for this
    behavior was, that one can so deliver also static resources
    (e.g. images) from a cgi-bin directory. Now this behavior con be
    controlled with the configure parameter "allowstaticresources",
    which is by default off.  If an application depends on this
    behavior, please turn it on in the config file (in the nscgi
    section).


  - nsproxy:
    New subcommand "ns_proxy stats" to provide usage and runtime
    statistics per nsproxy pool.


  - logging:

    * Better error.log entries: Include the connection id in the
      thread name to ease debugging, when multiple requests of the
      same connection thread write to the error.log and it is not
      clear, where the first ends and the next starts.

    * Write notice message to error.log when entities are too large
      to provide information about possible misconfigurations or
      attacks in the error log

    * Added new config parameter for nslog: "logthreadname"

      When the parameter is specified, the thread name is placed as
      second field in the access.log. This eases debugging, since one
      can now link the lines in error.log caused by an request with
      the line in the access.log without the need of guessing via
      time-stamps. The thread name contains the the name of the
      connection pool and a connection id.

   *  When "mutexlocktrace" is activated, use thread name instead
      of id to ease debugging


  - Added parameter "defaultextension" for adp section in config file.
    Can be set e.g. to ".adp" to test for FILENAME.adp, when adp is
    mapped to FILENAME



Performance Improvements:
-------------------------

  * Improved performance of ns_urlencode/ns_urldecode in common
    cases by about a factor of 2

  * Improved performance by replacing the slow "snprintf(... %d ...)"
    function by a the new functions ns_uint32toa() and ns_uint64toa().
    This change improved the performance of EnterSet() by nearly a
    factor of 2.5. Before, snprintf() took in this function 3x (!) as
    long as Tcl_CreateHashEntry(); the new ns_u32toa() is about 6x
    faster than Tcl_CreateHashEntry().

  * Improved performance of ns_quotehtml:

    In real-world applications (with e.g. OpenACS) this function is
    one the the most frequently used functions of NaviServer. This
    change improves the performance of "ns_quotehtml" by about 40-60% by
    avoiding per-character calls to *DStringAppend and some more
    improvements.

  * Improved performance of server internal "ns_set" operations: remove
    the need of 3 malloc()/free() pairs per request

  * Use Tcl_CreateHashEntry() in frequently called functions instead
    of Tcl_FindHashEntry() since the second one uses the first.

  * Avoid strlen() operations on several occasions
  * Improved speed of line-counting in .adp files by a factor of 2
  * Reduced scope of mutex lock when registering filters


Bug Fixes:
----------

 - Allow fully qualified domain names in "Host" header field
   (as allowed in RFC 2976)

 - Avoid potential hangs of nxproxy in cases the evaltimeout is very
   small and the client is writing much data causing blocking write
   operations

 - urlencode reform:

   * The new code conforms to RFC3986 (2005) rather than RFC1738
     (1994) and RFC1808 (1995) vs. RFC2396 (1998).  The newer RFC3986
     has a more precise and differently structured definition (e.g. no
     'unwise' characters) of characters encoded in URLs. The coding of
     the query part is actually defined in the HTML 4.01
     definitions. Legacy sites can use the old encoding, when
     compiling NaviServer with RFC1738 defined.

   * Produce a warning, when a not properly encoded URL is passed to
     the location header field (e.g. ns_returnredirect). Some newer
     browsers might reject those redirects. Only characters, which
     have to be always coded, are flagged.

   * ns_urldecode: Make sure, that only valid percent codes are
     converted back.

   * ns_urledecode: Use URL encoding charset, when no charset is
     specified.  This fixes an old (at least 10 year old) bug which
     can allow to sneak in binary nulls into query variables, which
     provide a potential injection attack vector


  - ns_server active|all|queued: Better handling of querying data from
    concurrent threads. Handle now semi-parsed requests: these are not
    "queued", but not all information is available for querying it. We
    do now the best effort to report on fields that are trusted.

  - Complain on connection output operations when the connection is
    already closed. Previous version of NaviServer swallowed output
    operations on already-closed connections more or less silently,
    leading to hard to understand messages in the error log. This
    happens in particular often during error handling in OpenACS,
    where a "ad_script_obort" is missing. The new code raises now an
    error message, pointing to the erroneous code. When the old
    behavior is necessary for a while for legacy installations, the
    global configuration parameter "rejectalreadyclosedconn" can be
    set to "false".

  - Fix duplicate DriverClose calls in context of UDP drivers

  - Undo potential crlf-translations in the body of (e.g. POST)
    requests with Content-Type "www-form-urlencoded"

    Background: When a hidden form fields contains line-breaks, these
    are transformed pn transit into CRLF by current browsers in POST
    requests. In this step the value of a hidden form-field might
    become larger on every iteration, which might result in a
    quadratic growth on multiple edits of such a field. The new
    behavior undoes this effect.

  - Improved handling of requests in ancient (pre HTTP/1.0 1996) HTTP
    requests in combination with writer threads. Previous versions
    could run into a too restrictive sanity assert, assuming that
    reply headers must be always present (which is not the case in the
    rather informal HTTP versions before HTTP/1.0)

  - Virtual server fixes:
    * Avoid potential double-initialization of driver modules when
      multiple servers are used.
    * Postpone registration of virtual servers until all servers
      are defined. Before it was possible that NaviServer boot was
      terminated, when a default server of a driver was not yet defined.

  - Improved handling of binary data:
    NsTclObjIsByteArray() behaves now more like the (Tcl internal)
    TclIsPureByteArray() to figure out, when the bytearray data has
    to be treated as binary. Cases of pure and impure bytearrays
    are handled now by NaviServer.

  - Fixed old bug, were SSL_shutdown could hang (observed on Linux
    systems)

  - Fixed potential crashes as flagged by fb infer.

  - Improved robustness on invalid URLs: Don't Fatal on invalid URLs
    passed to NSDriverClientOpen(), but spit out a warning.

  - Improved Tcl code safety (guarding "file delete" against names
    starting with a "--")

  - Improved handling of invalid input from config file for rolling
    files

  - Make sure "ns_server serverdir" and "ns_server pagedir" return
    normalized paths to avoid potential confusions

  - Improved windows portability:
    * Cope with changes in Universal CRT in Visual Studio 2015 where e.g.
      vsnprintf() is no longer identical to _vsnprintf().

    * Improved Windows support for recent versions of visual studio
      (versions after visual studio 2010) by introducing NS_EINPROGRESS
      and NS_EINTR for abstracting from the raw constants (many thanks to
      Maurizio for the input)

    * Make nsproxy compile-able under windows

  - Improved Unix Portability:
    Support systems without RLIMIT_AS (e.g. OpenBSD 6.2)

  - Improved Tcl portability:

    * Starting with Tcl 8.7a1, Tcl has actually two different types
      for bytearrays, the old "tclByteArrayType" and a new
      "properByteArrayType", where both have the string name
      "bytearray".  NsTclObjIsByteArray() is now extended to handle
      both types.



Documentation improvements:
---------------------------

  - Updated several man pages
    * admin-maintenance.man
    * commandlist.man
    * ns_accesslog.man
    * ns_adp_exception.man
    * ns_adp_include.man
    * ns_base64decode.man
    * ns_buildsqldate.man
    * ns_cache.man
    * ns_cancel.man
    * ns_chan.man
    * ns_connchan.man
    * ns_db.man
    * ns_driver.man
    * ns_gmtime.man
    * ns_http.man
    * ns_info.man
    * ns_job.man
    * ns_locationproc.man
    * ns_parseheader.man
    * ns_perm.man
    * ns_proxy.man
    * ns_serverrootproc.man
    * ns_shutdown.man
    * ns_sockcallback.man
    * ns_urlencode.man
    * ns_write.man
    * nssock.man
    * nsssl.man
    * ns_setpriveleges.man
    * returnstatus-cmds.man
    * tcl-overview.man

  - Additional global documentation changes:
    * Added various cross references with "see also"
    * Added keywords for "global builtin" and "server builtin" to
      distinguish between per-server commands and global commands
    * Added for every module the name of the module as "keyword"
    * Make markup of options more consistent

  - Improved banner of online man pages:
    * fix markup
    * make URLs protocol agnostic
    * use new .io domain of sourceforge

  - Fixed spelling errors


Tcl API Changes:
  - ns_connchan: new option "-driver" for "ns_connchan open"
  - ns_cache_create returns now boolean to flag success
  - "ns_http queue|run" new parameter "-hostname" for handling SNI

  - Removed useless and undocumented subcommand "ns_set idelete"

   * ns_urlencode: New flag "-uppercase" added for supporting encoding
     for OAuth (RFC 5849); note that the "path" segment encoding has
     to be used to avoid coding space as "+".

   * ns_urlencode and ns_urldecode: Additional accepted value "cookies
     "for parameter "-part" to request cookie encoding/decoding.

   * ns_getcookie: New option "-all"

     Background: By using a cookie domain, it is possible that a
     cookie with a specified name is given two value, one from
     e.g. the parent domain, one from the current; this can lead to
     situation not easy to debug, when e.g. domain cookies are
     introduced from other applications in the same domain. The "-all"
     option provides infrastructure support to detect such situations
     by returning potentially multiple values for the cookie.

   * ns_connchan:
      New (experimental) subcommand
	 ns_connchan listen ?-driver d? ?-server s? ?-bind? address port script
      Start a listening connection for the ns_connchan interface;
      Still missing: https variant via specifying "-driver" +
      handling of "-server" (and obtaining defaultserver)

  * sendmail.tcl
    - For the deprecated warning, do not write entire body of email to
      the log
    - Use local time (with timezone) instead of UTC in Date header


C API Changes:
--------------

  - New API function NsConnRequire() to make handling of required
    connections and error results more regular.

  - New API functions
      . Ns_CookieEncode()
      . Ns_CookieDecode()
      . NSDriverSockNew(): create an initialized driver Sock structure
      . Ns_HttpMessageParse(): parse a (potentially incomplete) HTTP message
      . ns_uint32toa()
      . ns_uint64toa(): substantially faster conversion of integer to
	string than snprintf()

  - Ns_SockListenCallback: alter interface to (a) return the listening
    socket and (b) to be able to bind to the fresh socket. This is
    necessary for e.g. FTPD's passive server connections, where the
    server offers a freshly bound socket to the client to connect to
    (EPSV, extended passive mode).

  - driver.c:
    * Factored out LookupDriver() to improve reusability
    * New function NSDriverSockNew() to create an initialized driver Sock
      structure,


Configuration Changes:
----------------------

  - Removed need to specify port for virtual servers in */servers
    section of the config files

  - Extended sample config files:

    * nsd-config.tcl
       . Added new nslog parameter "logthreadname"
       . Added new global parameter "rejectalreadyclosedconn"

    * openacs-config.tcl
       . Added configuration for strict-transport-security (HSTS)
       . Added setting of OpenACS kernel parameter
	 "NsShutdownWithNonZeroExitCode" in sample config file
       . Eased configuration for nsssl by simply setting https port
       . Added new nslog parameter "logthreadname"
       . Added new global parameter "rejectalreadyclosedconn"

    * sample-config.tcl:
       . Included global parameter "shutdowntimeout" in sample config file
       . Included parameter "allowstaticresources" in nscgi section
       . Added new nslog parameter "logthreadname"
       . Added new global parameter "rejectalreadyclosedconn"
       . Make it work out-of-the box


Code Changes:
-------------

  - Reworked TLS cleanup (necessary for Tcl 8.7 and beyond); align it
    to the Tcl practices. Add compatibility for Tcl 8.5- w.r.t
    notifier-thread init after the fork.

  - Switched to the recommended way of creating detached threads via
    pthread attribute (see e.g.
    https://docs.oracle.com/cd/E19455-01/806-5257/6je9h032l/index.html)

 - Extended regression tests
    * ns_urlspace.test
    * cookies.test
    * ns_sls.test
    * tclresp.test
    * ns_urlencode.test
    * url2file.test
    * ns_conn_host.test
    * ns_pagepath.test
    * ns_serverpath.test
    * ns_server.test
    * ns_proxy.test
    * http.test

  - Code cleanup
    * Added const declarations
    * Added missing prototypes
    * Don't shadow local variables
    * Marked unused arguments as UNUSED
    * Removed old-style function definitions
    * Reduced implicit conversions (gcc7)
    * Prefer type use "bool" over "int" when applicable
    * Don't pass implementation-defined NULL after the last typed
      argument to a variadic function
    * Reduced number of return statements before end of function
    * Reduced variable scopes
    * Reduced size of huge switch statements

  - API modernization
    * Prefer usage of Ns_ParseObjv over manual parsing and
      error message generation
    * Prefer Tcl_SetObjResult over Tcl_SetResult
    * Stop using readdir_r() unless it is forced via compile flag
    * Aligned casts to Tcl 8.6
    * Replaced all occurrences of strcpy() with more safe variants
    * Replaced all occurrences of sprintf()  by snprintf()
      to protect better against buffer overflows
    * Prefer void* over legacy caddr_r
    * Replaced all calls to deprecated function Tcl_DStringTrunc()
      by Tcl_DStringSetLength()

 - Implemented deprecated commands as Tcl proc and complain on
   its usage
      . ns_puts
      . ns_returnadminnotice
      . env

 - Silenced static checker

 - Add CFLAGSs for convenient testing/cleanup
   (CFLAGS_TIDY, CFLAGS_DEFINITION, CFLAGS_CONVERSION)

 - Improved error messages


Modules:
--------

 - nsstats:
    . Format values in human readable form (unless &raw=1 is used in query)
    . Return proxy stats form all proxy pools
    . Include expire date of certificate in "process" page
    . Added "commit" and "rollback" to cache statistics

 - nsdbpg:
    . Report more detailed version numbers used during built process
    . Honor "logsqlerrors" settings from config file
    . Truncate SQL statement in error.log to 10.000 chars if longer
      (to avoid overlong log file entries)
    . Improved backwards compatibility with ancient versions of PostgreSQL

 - nsimap:
    . Added optional flag "-novalidatecert" to "ns_imap open"
      (name from the flag of the IMAP implementation)
    . Use NaviServer built-in config check for SSL libraries and use same libraries
    . Updated to meet engineering standards

 - revproxy:
    . Added url_rewrite_callback
    . Added timeout handling
    . Added support for handling requests with provided content (POST, PUT, ...)
    . Generalized error handling, reverse callback registration as suggested by David.
    . Close connection explicitly after every request

 - letsencrypt:
    . New module for managing server certificates via Let's Encrypt

 - websocket:
    . Varius small updates (links, spelling errors, ...)