Forum OpenACS Development: Announcement: NaviServer 4.99.19 available

Dear all,

I am pleased to announce the availability of NaviServer 4.99.19 [1,2]. This release contains the fixes and enhancements of the work of the last year. Many thanks to all contributors!

The upgrade to 4.99.19 is recommended. Below is a summary of changes.

all the best, and a happy new year!

-gustaf neumann

NaviServer 4.99.19, released 2020-01-06

 247 files changed, 19527 insertions(+), 8275 deletions(-)

New Features:

 - Bandwidth management: limit download rate to a maximum (specified
   in kilobytes per second)

   On servers with bad internet connections, it is possible that a few
   eager downloaders can use up all available bandwidth.  NaviServer
   allows now admins and developers to specify the maximum outgoing
   bandwidth per connection or the total rate per connection pool. The
   rate limit can be specified per connection (highest weight) or per
   connection thread pool (second highest weight) or per network
   driver. The rate limitation is only active, when writer threads are
   used. Per default, the rate limitation is turned off (value 0).

   By using this option together with the context filters, one can now
   define e.g. a pool for bots and limit the bandwidth for all bots.

   The query the per-connection limit, ns_conn has a new option:
      ns_conn ratelimit ?limit?
   Introspecton "ns_writer list" returns the current
   transfer rates per writer job.

   Configuration option for connection pools:
      ns_param connectionratelimit ...
      ns_param poolratelimit ...
   Configuration option for network drivers:
      ns_param writerratelimit ...

   Commands for changing the rate limits dynamically:
       ns_server ?-server s? ?-pool p? connectionratelimit ?value?
       ns_server ?-server s? ?-pool p? poolratelimit ?value?

 - Context filter for urlspace:

   NaviServer's urlspace trie data structure used for various purposes
   such as registering methods/URL pairs for procs or for mapping of
   requests to connection thread pools. The classical trie mapping for
   e.g. "GET /foo/bar/*.html" is based on method + path + string match
   pattern, where the last part "*.html" is used as a "filter".

   The new version of NaviServer allows in addition to the string
   based glob pattern so-called "context filter", which includes the
   IP-address or header fields in the matching process. The context
   filters are applied after the classical urlspace processing, such
   that the behavior is fully backwards compatible. The context
   filters are kept in a stable order (IP before header-based filters,
   specific before less specific) such that the behavior is not order

   The new feature can be used map e.g. bots or certain IP ranges to
   the specified connection pools. When connection pools named "bots"
   and "local" are defined, one can use the following mapping commands
   to map incoming requests to the specified pools. Examples for
   adding urlspace mappings at runtime:

	 ns_server -pool bots map "GET /* {user-agent *bot*}"
	 ns_server -pool bots map "GET /* {user-agent *crawl*}"
	 ns_server -pool bots map "GET /* {user-agent *baidu*}"
	 ns_server -pool bots map "GET /* {X-NS-ip 2a03:2880::/29}"

	 ns_server -pool local map "GET /* {X-NS-ip}"
	 ns_server -pool local map "GET /* {X-NS-ip}"

   Note that arbitrary header fields can be used for the mapping,
   containing potentially some glob style match characters. The
   special header field "X-NS-ip" denotes IP based filters, which can
   be provided fully qualified or in CIDR notation (denoting
   significant bits) for IPv4 and IPv6.

 - Allow a single driver to listen on multiple IP addresses:

   This feature greatly simplifies setups, where a single server is
   listening on multiple IP addresses (e.g. IPv4 and
   IPv6). Previously, it was necessary to define separate drivers for
   these, which need different names but which are often
   configured identically.  Furthermore, when the address is omitted,
   the server performs a lookup from the hostname to determine the IP
   address. Previously, it used the first one, now it can work with
   all returned addresses.

   To use this feature, simply specify the configure values of
   "address" as a Tcl list containing multiple IP addresses. The
   change is fully backwards compatible, old configuration files will
   continue to work.

 - Added support for sending of multiple file chunks in a
   single connection:

      ns_writer sendfiles /list of filespecs/

   Every "filespec" is a dict which must contain a "filename" element
   and can contain additionally an "-offset" and/or a "-size" element.

      ns_writer submitfiles {{filename /tmp/f1} {filename /tmp/f2 -offset 10}}

   This function is e.g. useful for video streaming applications,
   where multiple video file chunks should be transferred (sometimes)
   concatenated in a single chunk. Therefore, this function avoids the
   necessity to concatenate these files in advance (causing increased
   latency and disk usage).

 - Added per-server "extraheaders". These extra headers fields are
   merged with the following precedence (the earlier ones have more
     1) application specific headers
     2) per-server extra headers
     3) per-driver extra headers

   This change makes it possible to provide in the config files
   defaults, which can be overwritten by certain pages in the
   applications. Furthermore, one can e.g. common header fields to all
   requests of a virtual server (e.g.  Strict-Transport-Security, ...)

 - Logging improvements:

   * Support separate access logs for requests sent via
     different drivers.

     The nslog module accepts now an additional parameter named
     "driver".  This option can be used to produce different access
     logs for requests submitted via different drivers to sort out
     e.g. local server talk.  Per default, every request is
     logged. When a value is provided for "driver", only
     requests are logged in this log file when these come from a
     driver matching the provided glob pattern.

   * log-file sanitizer:
     When data entered into a log file contains special characters
     (e.g.  end line characters), these could alter the appearance of
     content within the log file. Single entries may appear as
     multiple entries.  Attackers may leverage log forging to insert
     fake entries to that obfuscate malicious acts, or these can
     confuse log-file analyzers to avoid analysis.

     The new global parameter "sanitizelogfiles" can control this
     behavior (values 0: none, 1: full, 2: human-friendly; default:

   * Additional debugging flag "Debug(access)": When activated, entries
     of the access log are mirrored in the system log
     (error.log). This option eases tracking the end of requests in
     the system log.

   * Additional debugging flag "Debug(writer)" for debugging of writer.

   * Provided access to request AND reply header fields via
     "extendedheaders" specification (in "ns_accesslog" and "nslog"
     configuration section):

     It is now optionally possible to specify the header fields in the
     provided "extendedheaders" list with a prefix "request:" or
     "response:" to denote request and reply header fields. In case,
     no tag is provided, field names are interpreted as request header
     fields (like before). The change is fully backwards compatible,

  - ns_set:

    * Added sucommand "ns_set imerge", a is the case insignificant
      version of "ns_set merge". This option is useful especially for
      working with header fields.

    * Added subcommand "ns_set iupdate": this command is the case
      insenstive counterpart of "ns_set update".

    * Fixed "nsv_set a b" when neither a key nor the array exists.

 - ns_http:
   * Revamp of ns_http and related supportive code.
   * Added chunked-encoding parser
   * Tcl channel support for body (PUT, POST) and for result.
     This makes it possible to use e.g. reflected Tcl channels
     in "ns_http".
     New Options for "ns_http run"
       - ?-body_size size?
       - ?-body_chan chan?
       - ?-outputfile fn?
       - ?-outputchan chan?
   * "ns_http run" is now the preferred interface
     (rather than "queue" + "wait")
   * Deprecated output variables of "ns_http wait" (since everything
     is included in the resulting dict)

 - New commands:

   * ns_asynclogfile:
     Facility for thread-safe writing async
     log files of various kinds using the AsyncWriterThread.
     Usage example:
	# ... at startup
	set fd [ns_asynclogfile open /tmp/test.log]
	# ... during run
	ns_asynclogfile write $fd hello\n
	# ... at shutdown
	ns_asynclogfile close $fd

   * ns_parsefieldvalue:

     This function parses the provided field value (from an HTTP
     request or reply header field) into its parts and returns these
     in the form of a list of Tcl dicts or a single Tcl dict depending
     on parameters. The syntax of the contents of these header fields
     is specified in RFC 7230 section 3.2.6.

 - New features for existing commands/subcommands

    * Binary reform: Some NaviServer commands accepted previously
      implicitly binary or non-binary input, some commands used
      the flag "-binary" to denote the differences (e.g. ns_return).
      For a more uniform and expectable behavior binary input the
      explicit option "-binary" flag to the following commands:

	ns_crypto::aead::decrypt string (for "-key", "-aad", "-iv", "input")
	ns_crypto::aead::encrypt string (for "-key", "-aad", "-iv", "input")
	ns_crypto::eckey import ("-string")
	ns_crypto::eckey sharedsecret ("pubkey")
	ns_crypto::hmac add (for "message")
	ns_crypto::hmac new (for "key")
	ns_crypto::hmac string (for "key", "message")
	ns_crypto::md add (for "message")
	ns_crypto::md hkdf (for "-salt" "-secret" "-info")
	ns_crypto::md string (for "message")
	ns_crypto::md vapidsign (for "message")
	ns_crypto::scrypt (for "-salt" "-secret")

    Note that this option is not fully backward compatible. Early
    versions of "ns_sha1" were expecting always non-binary input, some
    new version assumed binary input. The new version is now more
    compatible with AOLserver and older NaviServer versions.

    * "ns_server threads" reports now how often a thread of this pool
      was started.  This statistic can help to identify
      installations, where threads are started and stopped in a too
      eager fashion.
    * "ns_server.... stats": added "sendbodysize" and "replybodysize"
      to the dict of the per-pool statistics.
    * "ns_server ... connectionratelimit ?value?": Query or set
       the default per-connection rate limit.
    * "ns_server ... poolratelimit ?value?": Query or set
       the pool connection rate limit.
    * "ns_writer list" returns now the current transferrates per
      writer job.

    * "ns_writer submitfile -offset X -size Y" and "ns_writer size X"
      accept now memory units, the option ?-driver driver? was added
      to the "ns_writer" subcommands "size", and "streaming"
      (defaulting to the current driver).

      One can now write
	   ns_writer size 150KB
       instead of
	   ns_writer size nssock [expr 150*1024]

Bug Fixes:

  - Blueprint serializer: moved serialized objects in the blueprint
    after the namespace imports and ensemble recreators since
    constructors can call functions depending on it.

  - Include *xml* and *json* in non-binary MIME types. Note that
    there is no exact definition, what MIME types are exactly.

  - nslog:
    * Abort in error situation with an error message instead of crashing
    * Added server name in log entries to make it easier to distinguish
      messages per-server during bootup

  - Range requests:
    * Fixed potential problem with too many non-contiguous byte ranges
    * Don't silently ignore invalid syntax of range requests

  - Improved handling of already closed connections (data delivery was
    already delegated to writer threads, but driver still wants to
    return data directly; this could happen in error situations)

  - connchan:
    * Fixed potential invalid reads in (error) cases, where
      LogConnchanDebug is enabled and the callback is deleted during a
    * Fixed potential problem, where call to Tcl_Eval() might clean
      structures used in the callback handler

  - Made sure, memory allocated by Tcl is returned by Tcl (important,
    when NaviServer is compiled with -DSYSTEM_MALLOC)

  - Fixed Tcl argument parsing of "ns_roll", "ns_fmttime",
    "nsv_bucket" and "ns_critsec eval".

  - Fixed bug in "nsv_set -default" overwriting pre-existing values

  - Fixed potential crashes in (desperate) error situations, where the
    connection was already closed.

  - Fixed leaking tmp file, when writerstraming is activated and
    streaming HTML output is used. This bug could appear, when
    "writerstreaming" is turned on in the config file (default off).
    See also:

  - Made table used for enumeration values for Ns_ObjvIndex static.
    Background: This is a fix for a tricky Tcl_Obj sharing bug
    triggered by the usage of C-level IndexObjs based on volatile
    tables. Consider the following example:

       proc foo {x} {
	  return [bar -value x ...]

    The Tcl_Obj "x" is shared as name of argument and as a value of
    the non-positional parameter "-value". When bar
    (e.g. C-implemented) uses Tcl_GetIndexFromObj*() to lookup "x" in
    a table of options, which is volatile, the involved Tcl_Obj will
    be converted to an indexObj.

    On a call with wrong number of arguments( e.g."foo 1 2 3"), Tcl
    will try to give a nice error message, saying that "foo x" can be
    called only with one argument. When printing argument "x", it sees
    that "x" is an indexObj, and for these kinds of objects, "x" might
    be an abbreviated version of a full name. Since the table behind
    the indexObj is in the case above volatile, a crash might happen.

  - Added API call Ns_SockInErrorState() since SSL_shutdown() must not
    be called if a previous fatal error has occurred on a connection
    i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or

  - Provided local buffers for OpenSSL ERR_error_string handling to
    avoid potential race conditions (OpenSSL keeps otherwise error
    messages in static memory).

  - Crypto support:
    * New feature: scrypt Password-Based Key Derivation Function (RFC 7914)

      The scrypt function is a modern replacement for crypt and bcrypt
      and derives secret keys from a secret string. It is based on
      memory- hard functions, which offer added protection against
      attacks using custom hardware and GPU arrays.

      The function requires the compilation of NaviServer against
      OpenSSL 3.0 or newer (not yet released)

      Example from RFC 7914:
	% ::ns_crypto::scrypt -secret "password" -salt NaCl -n 1024 -r 8 -p 16

     * "ns_md" and "ns_hmac": added flag "-encoding"

     * Added options "-passphrase" to all "ns_crypto" commands
       accepting a PEM file. In principle, every PEM file can be
       protected by a pass phrase.

     * Signing and verifying signatures: "::ns_crypto::md string" can
       be used for signing and verifying of cryptgraphic signatures:

	% set sig [::ns_crypto::md string \
		     -digest sha1 \
		     -encoding binary \
		     -sign /usr/local/src/naviserver/myprivate.pem \

	% set vfy [::ns_crypto::md string \
		     -digest sha1 \
		     -verify /usr/local/src/naviserver/myprivate.pem \
		     -signature $sig \

Documentation improvements:

 - Improved cross references between man pages
 - Added documentation to undocumented flags
 - Added more example
 - Improved spelling

 - The following 48 man pages were updated since the last release:,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

Configuration Changes:

 - Output warning to system log, when NaviServer was compiled without
   zlib support, but the config file request compression.  Before, the
   compress enable request was silently ignored, giving no hint, why
   compression attempts of a developer were ignored.

 - Sample configurations:

    * Documented usage of connection pools in sample config files.

    * Added sample entries for new features (e.g. rate limits, log
      file sanitizer)

    * openacs-config.tcl:

       . Changed driver installation to "global" in sample configuration
	 file. This change makes it easier for site admins to add
	 further servers (e.g. for virtual hosting).

       . Simplified sample config file by using multiple IP addresses
	 per driver

   * nsd-config.tcl:

       . Changed driver installation to "global" in sample configuration
	 file. This change makes it easier for site admins to add

Code Changes:

 - Improved scalability: provided different mutex locks variables per
   urlspace IDs.

 - Improved range checking for Ns_ObjvInt, Ns_ObjvLong, Ns_ObjvWideInt
   and MemUnits:

   * built-in range checkingsupport for objv parser: This eases
     specification of the C API, makes accepted ranges clear and
     provides consistent error messages. These change fixes several
     potential crashes, when e.g.  passed-in values are used as
     array-indices (crash on negative numbers) or where the values
     were casted to unsigned types (very high unsigned values,
     potential memory alloc problems).

    * Fixed all cases, where Tcl_GetIntFromObj(), Tcl_GetLongFromObj,
      and Tcl_GetWideintFromObj was used without a range check.

 - Argument parser: skip processing of optional arguments when the
   number of provided arguments is equal to the number of required

   Background: This change allows a fully backward compatible version
   of e.g. "ns_md5 $foo", no matter what the value of "$foo" is,
   although the new version of "ns_md5" has now a "-binary" option.

 - Refactored low-level socket I/O:
   * All driver operations are now strictly non-blocking
   * Use same infrastructure for all network I/O operations
     (driver, connchan, ns_http, ...)
   * Rewritten socketfile/sendfile handling (including *BSD support)
   * Improved handling of closed and detached channels, e.g.delegated
     to writers (new macros NS_CONN_REQUIRE_CONNECTED,
   * Improved protection for SIGPIPE signals.

 - New API infrastructure: Ns_DList

   The Ns_DList is similar to Tcl_DString, but operates on pointers
   instead of characters. Like Tcl_DString, Ns_DList pre-allocates
   some data and extends it when necessary. Ns_DList structures are
   especially useful alternative to linked lists, when usually data
   is added to the end. Since the data is kept as a dense
   (potentially growing) array, the memory locality is much better
   than with linked lists, resulting in better CPU cache hit rates.

 - Distinguish between Ns_NormalizePath() and Ns_NormalizeUrl(),
   where the first is for the file system and the latter for URLs.
   Previously, both cases were handled identically.

  - Function pointer handling:
    * Created a Tcl hash type for function pointers, since ANSI/ISO C
      forbids casting/comparison of data and function pointers. The
      new type is used whenever a hash lookup is performed for a
      function pointer.
    * Fixed all other occurrences of such comparisons.

 - Regressions testing:
   * Removed false positive
   * Switch to testing via ns_http in regression tests
     (this makes it as well possible for testing https connections)
   * Extended tests:
     + Added regression tests for "ns_rand"
     + Added nsssl tests
     + Added automatic certificate generation for test-server

    * The following 35 tests were added or extended since the last release

	adp.test, encoding.test, http.test, http_byteranges.test,
	http_chunked.test, http_keep.test, https.test, misc.test,
	ns_adp_compress.test, ns_base64.test, ns_cache.test,
	ns_conn.test, ns_conn_host.test, ns_crypto.test,
	ns_driver.test, ns_hashpath.test, ns_hostbyaddr.test,
	ns_info.test, ns_limits.test, ns_md5.test, ns_nsv.test,
	ns_pagepath.test, ns_parsefieldvalue.test, ns_proxy.test,
	ns_reflow_text.test, ns_schedule.test, ns_server.test,
	ns_serverpath.test, ns_set.test, ns_sha1.test,
	ns_urlencode.test, ns_urlspace.test, ns_uuencode.test,
	ns_writer.test, tclconnio.test, tclresp.test

 - Improved portability:
   * improved compatibility with LibreSSL (2.7 and 2.9)
   * Windows changes (Many thanks to Andrew Piskorski):
     + Fixed Ns_LogRoll() to work on Windows.
     + WSASend receives as 5th argument flags, but not a pointer to flags.
     + Improved makefiles
     + Updated _MSC_VER version numbers to include Visual Studio 2019
     + Added macro NS_INLINE to achieve higher portability with (older?)
       Microsoft compilers.

 - Added warnings about potential misconfiguration of NaviServer "tcllib"
   setting (Tcl written NaviServer modules)

 - Marked compatibility wrappers Ns_SetThreadServer() and
   Ns_SetThreadServer() explicitly as deprecated. Use
   Ns_ThreadSetName() and Ns_ThreadGetName() instead.

 - Build-system
   * Added -DSYSTEM_MALLOC to default compile flags.
   * Added nsssl to the default test target.
   * Require RSA keys of size 2048: newer versions of OpenSSL refuse to work
     with RSA keys of size 1024.
   * Improved alignment with Tcl's current .m4 file to get rid of
     "-prebind" deprecated message under macOS.
   * Improved handling of posix thread library for FreeBSD and OpenBSD

 - Use native thread_local storage for log handling when available

 - Improved type cleanness for function pointers
 - Improved code locality
 - Improved error messages
 - Improved structure packing
 - Reduced variable scopes
 - Added missing "extern" declarations
 - Aligned function prototypes
 - Dropped potentially dangerous call to alloca()
 - Added typedefs for commonly used functions
 - Reduce potential dangling pointer dereferences
 - Aligned names of arguments in prototype with function definition
 - Introduced use attribute-based approach for denoting
   fall through in case statements
 - Added more declarations for PURE and CONST functions
 - Improved configurability for clang-tidy
 - Fixed macro name-clash with PostgreSQL
 - Improved spelling

 24 files changed, 1638 insertions(+), 2395 deletions(-)

- Fixed loading when multiple servers are used
- C Code cleanup (reduced warnings)

- Fixed compatibility with current versions of NaviServer
- C Code cleanup (reduced warnings, improved spelling)

- Removed deprecated calls
- fix compilation with TCL_NO_DEPRECATED
- C Code cleanup (reduced implicit conversions, improved spelling)

- C Code cleanup (reduced warnings)

- Made code compilable with TCL_NO_DEPRECATED
- C Code cleanup (reduced warnings, improved spelling)

- Made code compile cleanly

- C Code cleanup (improved spelling)

- Upgraded module to PHP 7
- Fixed memory leaks
- extended regression test
- C Code cleanup (improved spelling)
- This is a major overhaul of nsphp, which makes it possible to use
  PHP 7 (tested with PHP 7.3.5). PHP 5 reached it EOL by jan 2019. PHP
  7 has substantial changes e.g. in memory and thread management
  compared to PHP 5, several interfaces and API calls have changed,
  such that it is not possible to compile the new version of nsphp
  with still PHP 5. In case, someone wants to use nsphp with PHP 5,
  please checkout versions before this commit from the repository.

- Added link to process page from mecurial hash to version
  details on Bitbucket
- Included "ns_driver info" in the reported values
- Include number of started thread per pools in reported statistics

- C Code cleanup (don't shadow variables, clean compilation, improved spelling)

- C Code cleanup (clean compilation, improved spelling)

- C Code cleanup (clean compilation)

- Compatibility with Tcl 8.7, which has no "identity" encoding anymore
- fixed startup messages
- C Code cleanup (improved spelling)

- Support for request cancellation via url_rewrite_callback (when it
  returns an empty URL)
- Don't raise error on ECONNRESET during spooling
- Improved logging
- Improved documentation
- Made channelCleanup more robust

- Major overhaul of the letsencrypt module to support ACME v2

  ACME (the Automated Certificate Management Environment, [1]) is the
  protocol used for certificate mamagement on The API
  version v was released on 2016 but was updated in 2018 by ACME v2,
  is not backwards compatible with v1. Letsencrypt announced in march
  2018 to drop the support of ACME v1 in several steps:

   - Nov 2019: End of account registrations via ACME v1
   - Jun 2020: End of new domain registrations via ACME v1
   - Jun 2021: EOL ACME v1 certificate issuing

  The new version is based on its crypto requirements solely on
  OpenSSL, it uses the NaviServer builtins and as well the "openssl"
  binary (the usage of tcllib pki was dropped). To avoid potential
  troubles, use this with a recent version of NaviServer (currently
  the tip version form BitBucket) or with NaviServer 4.99.19 when this
  is released.
- Improved logging and documentation
- Add documentation for obtaining multi-domain attributes (SAN).

 - Added parameter "dopadding" to improve compatibility with Firefox on
   Android. It does not support padding. Many thanks to Wolfgang
   Winkler for reporting.
 - Added compatibility with current version of NaviServer
 - Always use binary format for appending (it does not seem necessary
   here, but it is used for consistency)
 - Code cleanup (improved spelling)

 - Adapted changes from Malte Sussdorf for the AOLserver version
   (from before 2008)
  - Added compile flag to force LDAP v3
  - Rebind with original credentials after bind auth try
  - Incorporated changes from Andreas Parschalk (Univ. Innsbruck)
    including better documentation
  - Provided clean compilation on recent versions of NaviServer and Tcl