Forum OpenACS Development: Announcement: NaviServer 4.99.17 available

Dear all,

I am pleased to announce the availability of NaviServer 4.99.17 [1,2]. This release was tested under Windows 8.1 (32bit) and 10 (64bit) many thanks to Oleg Oleinik), various Linux versions (e.g. Ubuntu 18.04, FC 27, including Raspberry PI Raspbian GNU/Linux 9.4 with ARM processor), and OmniOS and OpenBSD, with Tcl 8.5, 8.6 and 8.7 (alpha). Many thanks to all contributors!

all the best

-gustaf neumann

NaviServer 4.99.17, released 2018-11-04

 421 files changed, 17805 insertions(+), 10275 deletions(-)

New Features:

  - Added brotli support for delivering static content
    The brotli compression format (RFC 7932) achieves better results for
    compression for typical web content compared to e.g. gzip and is
    supported by most current browsers. The definition of static
    content delivery is exactly the same for "brotli" as for static
    "gzip" content.

    New configuration options for ns/fastpath:
    * "brotli_static": should static brotli delivery be checked?
    * "brotli_refresh": should outdated brotli compressed files be refreshed?
    * "brotli_cmd": OS-level command with options used for brotli compression

  - "ns_conn" additions:
    * New subcommand "ns_conn acceptedcompression" to return accepted
      compressions for the current connection (gzip or brotli)

    * New subcommands "ns_conn currentaddr" and "ns_conn currentport"
      to refer to the address and port of the currently open server side
      of the socket.

  - New feature for GDPR: add mask IP addresses optionally in access log
    This feature is similar to "anonip" IP anonymizer of the Swiss
    privacy foundation): When this feature is activated, all IP
    addresses are masked in the log file, such that the host-specific
    (= person- specific) information is masked out and the IP-address
    does not match a particular individual anymore. Still, with the
    masking turned on, reverse DNS lookup and/or geolocation is

    The option can be configured via the following parameters in the
    ".../module/nslog" section of the config file of NaviServer:
     * "masklogaddr": boolean value to turn feature on/off (default off)
     * "maskipv4": mask for IPv4 addresses (default
     * "maskipv6": mask for IPv6 addresses (default ff:ff:ff:ff::)

  - New atomic "nsv_set" commands/options

    * Obtain (old) value from nsv ARRAY and set it to a new value set foo

        set foo [nsv_set -reset ARRAY KEY NEWVALUE]

      After the operation, ARRAY(KEY) has a new value.
      The operation is similar to "nsv_set ARRAY KEY NEWVALUE",
      but the variant with "-reset" returns the old value.
      Similar to GETSET in REDIS.

   * Obtain a value from nsv ARRAY and unset it (no new value is provided).

        set foo [nsv_set -reset ARRAY KEY]

     The operations returns the old value of ARRAY(KEY) or empty, if
     it does not exist.  After the operation, the ARRAY(KEY) is unset.

   * Set a default value for a nsv ARRAY

        nsv_set -default ARRAY KEY DEFAULTVALUE

      Do nothing when ARRAY(KEY) has already a value.
      Return an error, when DEFAULTVALUE is not provided.
      Similar to SETNX in REDIS.

  - Extended functionality of "ns_http"

    * Changed return value for "ns_http run" and "ns_http wait".
      Now the commands return attribute value pairs (Tcl dict).
      Previously, the commands returned different results,
      depending on the options

      - "ns_http run" returned the name of the queued command (which
        was completely useless, this the queued command was already
      - "ns_http wait" returned sometimes the body of the reply (when
        the reply was not spooled) or empty (spooled) or "1", when the
        option "-R" was specified

      Now the result is always a dict containing "status" (HTTP status
      code), "time" (elapsed time), "headers" (ns_set of the reply
      header fields, and either the "body" (reply body) or "file" (the
      name of the spooled file). The values in the dict might mirror
      the result of other (optional) output variables, which continue
      to work.

      Since the results were irregular and not documented in the man
      pages, the backward compatibility should be high. Also, the
      regression test of NaviServer was not effected by this change.

    * Support for asynchronous “ns_http” tasks:

      When the new flag "-donecallback” is provided, then a “ns_http”
      task is started in the background and does not require an
      “ns_http wait” anymore. Instead, when it finishes it executes
      the provided donecallback which receives as additional arguments
      the Tcl result (in form of an integer) and the result dict,
      returned otherwise from e.g. “ns_http run”.

      As a consequence, requests of the form
         ns_http queue … -donecallback …
      differ from client requests without the callback
      in the following points:

      a) the command does not return a handle for “ns_http wait”
      b) The command is not automatically cleaned up (canceled)
         at the end of a connection requests
      c) Typically, donecallbacks are executed in a different
         thread than it was started.

      Some more options, which were previously only valid on
      “ns_http wait” (such as e.g. spoolsize) are now valid
      for “ns_http queue” as well.

      The new flag allows a developer to start many requests in
      parallel without having to start many threads or complex
      synchronization tasks.

    * New option "-body_file" for "ns_http wait" (similar to "ns_http
      queue" which can be used to specify a filename for the
      downloaded content)

    * New option "-outputfile" to "ns_http run" and "ns_http wait":
      by specifying the option "-outputfile" the received file will be
      always written to the disk, no matter how large it is.

    * New subcommand "ns_http stats": returns a dict (flat list of
      attributes and values). The list contains "task", "url",
      "requestlength", "sent", "replylength" and "received".

  - Support resolving a hostname against multiple IP addresses

    * In case, one domain name has multiple IP addresses registered,
      previous versions of NaviServer tried only the first returned
      address and returned an error, when this failed. This happened
      when using client functions (e.g. ns_http).

    * host names with multiple IP addresses are nowadays quite common,
      e.g. "localhost" having an IPv4 and an IPv6 address registered. In
      previous versions, when e.g. localhost has as first address the IPv6
      address, but IPv6 was not active, it returned an error.

    * resolving against IP addresses continues to work as always.

  - Extended crypto functionality

     * Added HMAC-based Extract-and-Expand Key Derivation Function (HKDF)
          ::ns_crypto::md hkdf -digest sha256 ....
       (requires OpenSSL 1.1.0 or newer)

     * Added support for elliptic curve cryptography
        . ns_crypto::eckey import ?-string string? ?-encoding encoding?
        . ns_crypto::eckey priv ?-pem pem? ?-encoding encoding?
        . ns_crypto::eckey pub ?-pem pem? ?-encoding encoding?
        . ns_crypto::eckey sharedsecret ?-pem pem? ?-encoding encoding? pubkey
        . ns_crypto::eckey generate ?-name name? ?-pem pem?

     * Added support for getting an arbitrary number of random bytes
       based on crypto support on OpenSSL:
        . ns_crypto::randombytes ?-encoding encoding? bytes

     * Added support for Authenticated Encryption with Associated Data
       (AEAD) scheme, which provides confidentiality, integrity, and authenticity.

        . ns_crypto::aead::encrypt string ?-aad aad? ?-cipher cipher? ?-encoding encoding? ?-iv iv? ?-key key? input"
        . ns_crypto::aead::decrypt string ?-aad aad? ?-cipher cipher? ?-encoding encoding? ?-iv iv? ?-key key? ?-tag tag? input"

     * Added "-encoding hex|base64url|base64" option for output encoding
        for the crypto functions. Previously, all these functions returned the
        result in "hex" format. If the option is not specified, the result
        is the same as before (hex). The encoding option was added to the
        following functions:
        - ns_crypto::hmac string
        - ns_crypto::hmac get
        - ns_crypto::md string
        - ns_crypto::md get
        - ns_crypto::md vapidsign
        - ns_crypto::enc

  - "ns_connchan" improvements

    * Callback suspension: a Tcl socket callback function might now
      return the value "2" to flag that the callback will be
      suspended, but it keeps the socket open. Previously, the Tcl
      callback could return just "0" (callback is cancelled, socket is
      closed) or "1" (continue callback handling, keep socket open).
      This can be used to suspend callback handling on bottlenecks and
      continue later, when situation improves.

    * "ns_connchan write": The function returns now the number of
      bytes sent, which might be less than the length of the data to
      be sent.

    * "ns_connchan list": the function returns now two more fields per
      entry: a) cmd name of the callback (eases debugging, when
      callbacks are changed dynamically) b) conditions used for
      registering the callback.

    * The "ns_connchan" callbacks are now listed via the nsstats
      interface as well (with proper callback info).

  - Added Possibility to add per-server "initcmds" to config file

    This new feature provides an easy means to add command which
    should be executed after server initialization into a configuration
    files. By using the "initcmds" one can start multiple instances
    of NaviServer differing just in the initialization command
    by providing different config files. The "initcmds" approach
    has the advantage over e.g. "ns_atserverstart" that it is
    executed in a state when the server is fully initialized.

  - Improved readability for config files

    The "ns_section" command has new an optional last argument for the
    parameters of its configuration section. Instead of writing e.g.

       ns_section ns/parameters
       ns_param serverlog    $logroot/error.log
       ns_param pidfile      $logroot/
       ns_param home         $homedir

    one can write now

       ns_section ns/parameters {
           ns_param serverlog   $logroot/error.log
           ns_param pidfile     $logroot/
           ns_param home        $homedir

    This block notation has two advantages: a) it shows clearly, where
    the section ends and b) when using an editor with automatic
    indentation, the log file looks nicer without manual indentation
    work. This change is fully backward compatible, old style config
    files continue to work.

  - Further new commands and features

    * "ns_hash": provide a interface for the HashStringKey() function
      (very useful for e.g. cache partitioning)

    * "ns_reflow_text".
          ns_reflow_text ?-width width? ?-offset offset? ?-prefix prefix? ?--? text
      Tcl-based solutions are surprisingly slow on largish input.

    * "ns_base64urlendcode", "ns_base64urldedcode": URL-save variants of base64,
      used in various new RFCs for passing base64 values in HTTP requests.

   * "ns_uuid": runtime efficient version of a version 4 UUID
      according to RFC 4122: A Universally Unique IDentifier (UUID)
      URN Namespace

   * Improved results of [ns_cache_stats -contents ...]. The command
     returns now a tcl list instead of a string, including reuse count.

  - Improved results of "ns_striphtml":
     * Added resolving of all HTML4 entities (including   etc.)
     * Stripped as well HTML comments.

Performance Improvements:

  - Check for epoch updates on blue-prints as well in cleanup phase
    (deallocate trace) to reduce latency for client requests.

Bug Fixes:

  - Don't try to serve redirected files in case the connection is
    already closed (will lead only to confusing messages)

  - Ns_IsBinaryMimeType(): When a chartype is specified, never assume
    the mimetype is binary.

  - ns_getformfile: Don't rely on global variables
    Before this change, Depending on configured values,
    ns_getformfile could have missed uploaded files.

  - Added missing fetch commands for obtaining handles (on obviously
    seldom used subcommands "ns_db setexception|sp_setparam")

  - ns_http:
    * Timeout and connection behavior show works now consistent
      for HTTP and HTTPS URLs.
    * Fixed a potential bug (infinite loop) when requests
      run into timeouts.

  - "ns_set print": align with documented behavior

  - "ns_parseurl": make results more robust, when a URL with a colon
    in the path was given, but no port provided (detected a port

  - "ns_parseformfile": handle also www-form-urlencoded (in
     addition to multipart/form-data as before)

  - Windows:
    * Make sure to create temporary files correctly under windows (many
      thanks to Oleg Oleinik)
    * Incorrect handle was used to perform operations on the
      process in Ns_WaitForProcess (many thanks to Oleg Oleinik)

Documentation improvements:

  - Updated several man pages

Tcl API Changes:

  - "ns_conn isconnected": returns now false, when the connection is
     already closed. Otherwise we have to add a new subcommand
     "ns_conn closed", which seems somewhat odd (isconnected true and
     closed true?)

  - "ns_cache_stats" returns now hitrate in form of a float instead of
    an integer.

C API Changes:

   - Ns_SockTimedConnect2() returns now Ns_ReturnCode to be able to
     distinguish between timeouts and errors

   - Ns_SockConnectError(): new call to provide consistent error messages
     and to set Tcl error info consistently

Configuration Changes:

  - Use same initialization for "enablelclpages" when configured over
    "ns/server/$server" or over "ns/server/$server/adp" to avoid subtle
    differences on global variables.

  - Extended sample config files:

    * nsd-config.tcl
       . Added parameter "masklogaddr" for "nslog" section
       . Added parameter for "gzip" and "brotli" static delivery
       . Added parameter for "initcmds"
       . Added setting for recommended environment variables
         HOME, LANG and RANDFILE. RANDFILE is sometimes needed 
          to avoid surprises with permissions, when OpenSSL 
          crypto functions are called from the server

    * openacs-config.tcl
       . Switched to PostgreSQL's conninfo notation in the datasource
         to hint that more such parameters are possible as well
         (e.g. connections via SSL)
       . Added comments and examples for configuring OpenACS caches
       . Added comments how to configure EmailDeliveryMode via config file
       . Added comments how to configure IncludeCallingInfo of OpenACS API browser
       . Added comments how to configure WithDeprecatedCode
       . Added example for activating more intense SQL logging
       . Added subsite-based error pages
       . Added section indicating how to configure for nsstats module
       . Added parameter settings for "reuseport"
       . Added parameter "masklogaddr" for "nslog" section
       . Added parameter for "brotli" static delivery
       . Preconfigure customized error pages based on subsites

  - Config files changed to block notation (see above)

  - Improved handling of configuration errors:
    * Provided better log message, when configured value is out of range
    * Updated config values when range check corrects the value to get the
      finally configured value via introspection.

Code Changes:

  - Extended regression test:
        * tests/adp.test
        * tests/compat.test
        * tests/http.test
        * tests/ns_base64.test
        * tests/ns_base64encode.test
        * tests/ns_cache.test
        * tests/ns_conn.test
        * tests/ns_crypto.test
        * tests/ns_nsv.test
        * tests/ns_parseurl.test
        * tests/ns_reflow_text.test
        * tests/ns_server.test
        * tests/ns_striphtml.test

 - Added version tag of NaviServer to "starting" message in log file
   to ease analysis in case of problems.

 - Implemented deprecated commands as Tcl proc and complain on
   its usage
      . ns_adp_eval
      . ns_adp_safeeval

  - Improved Platform support:
    * Extended list of platforms for manual provisioning of __PRIPTR_PREFIX
      in case it is not defined.
    * Do not rely on implementation defined print/scan format "%p", 
      but use instead standardized integer types.
    * Windows:
      . Improved type-cleanness (many thanks to Oleg Oleinik)
      . Support to compile with MSVC 2017 (many thanks to Oleg Oleinik)
      . When compiling with VS 2012, use built-in support for
        intptr_t and uintptr_t.
      . Added support to run regression test under Windows 
        (many thanks to Oleg Oleinik)

  - Changed preset encodings (when not specified in config file) for
    files with extensions .htm, .html, and .adp from iso8859-1 to

  - Improved debugging support:

    * Added compiler-flag NS_MUTEX_NAME_DEBUG for ease locating
      anonymous mutexes.
    * Provided names for all remaining mutexes to make it easier
      to pinpoint bottlenecks due to mutexes
    * Improve debugging output in various messages, when debugging
      options are activated.

  - Minor update of mime types based on actual IANA media types.

  - Provided compatibility with Tcl 8.7.2 (removed CONST and VOID macros)
  - Further improved robustness by avoiding potential NULL dereferences
    found be static code analyzers (mostly error cases)
  - Improved configure scripts
  - Make sure, we have enough space for the terminating null character
  - Added "pure" specifier for improved code generation

  - Various code smell removals
    * Prefer boolean type over int
    * Improved type cleanness
    * Reduced implicit conversions
    * Reduced number of return statements before end of function
    * Don't pass implementation-defined NULL after the last typed argument
      to a variadic function
    * Removed dead assignments
    * Added "const" declarations
    * Fixed typos, improved comments


 - nsdbbdb (many thanks to Brendan Graves):
   * Changed to work with FreeTDS 1.00.86
   * Updated outdated paths.
 - nsdbpg:
   * Added sample configuration to include hints how to use SSL
     connections to the database
   * Provided compatibility with Tcl 8.7.2
   * Minor code cleanups

 - nssmtpd, nsdns, nstftpd, nsradiusd, nsdbi, nsdbipg:
   * Brought code in sync with the current code base (including IPv6, 8.7.2 compat)
   * Improved spelling

 - nsphp (many thanks to Anthony Bennett):
   * Refactored for newer versions of NaviServer
   * Improved README and spelling

 - nsstats:
   * Added results to "ns_connchan list" to per-server info
   * Added optional query parameter "raw=0|1" to switch between 
     raw and prettified numbers
   * Improved number prettification
   * Added optional query parameter "max=..." for controlling the number of
     per-cache entries shown (default 50)
   * Improved cache statistics: added hits/req, saved/req, saved/hit; dropped saved/KB
   * Improved mutex statistics: added locks/req, potential locks/req, potential locks per second
   * Added summative statistics to mutex locks page
   * Made background locks (locks, not included in per-request statistics) 
     configurable via NaviServer config file
   * Various other small changes and beautifications (use more CSS)
   * Bumped version to 1.8

 - nsoracle:
   * Fixed potential crash when oci_error_p is called with dbh == NULL
     (many thanks to Oleg Oleinik)
   * Some more code cleanup and improved spellings.
 - websocket:
   * Fixed potential bug: make sure that upgrade header is sent as a single block.
   * More minor code cleanup.

 - revproxy:
   * Fixed potential blocking conditions when send buffer of OS overflows.
     With "-sendtimeout 0" a call to "ns_connchan write" might write less 
     bytes than desired; In this case, callbacks are suspended and continue
     when the situation improves.
   * Added new switches "-sendtimeout" and "-receivetimeout" to revproxy::upstream
   * Added "-backend_reply_callback" to ::revproxy::upstream to allow proxy the
     modification of backend header fields per request.
   * Bumped version number to 0.11

 - letsencrypt:
   * Modernized code: use built-in ns_base64urlencode (requires 4.99.17)
   * Changed order of certificates in pem file (private key first)
   * Don't hard-code OpenSSL dir on OS (many thanks to Stefan Sobernig)

 - nswebpush:
   * New module for implementing Web Push via NaviServer
     (Many thanks to Georg Prohaska)
   * For details, see:
Posted by Brian Fenton on
This is great news! Well done and many thanks.


Second Brian -- this looks like a major release very interesting improvements across the board.