Forum OpenACS Development: Announcement: NaviServer 4.99.20 available

Dear all,

I am pleased to announce the availability of NaviServer 4.99.20 [1,2]. This release contains the fixes and enhancements of the work of the last year. The short summary of changes is (see below for more details):

  • Support for time units (specify time units in minutes, seconds, milliseconds...)
  • Subsecond granularity for all NaviServer commands
  • HTTP client request logging
  • Added support for "100 CONTINUE"
  • Improved server liveliness (management of high loads and request queue overruns)
  • Support for running behind a reverse proxy server
  • Driver support for listening on multiple ports
  • Improved HTTPS support (OCSP Stapling, server-side SNI)
  • Improved support for client-side HTTP requests via ns_http (timeout handling, binary handling, log file for outgoing requests)
  • Improved logging for detecting long running background jobs
  • Dict support for nsvs
  • More complete set of sub-commands for "ns_set"
  • Improved form-processing (especially for REST processing)
  • C-level WebSocket support
  • Improved server statistics
  • Improved scalability (reduced lock contention, higher usage of rwlocks, significantly faster nsv_*appends, ...)
  • Improved portability: many improvements for windows, compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2
The following people have contributed to this release:
  • Andrew Piskorski
  • David Osborne
  • Gustaf Neumann
  • Hector Romojaro
  • Maksym Zinchenko
  • Oleg Oleinick
  • Zoran Vasiljevic
Many thanks to all contributors!

The upgrade to 4.99.20 is recommended. Below is the more detailed summary of changes.

All the best in the new year!

-gustaf neumann
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/
[2] https://bitbucket.org/naviserver/naviserver/

=======================================
NaviServer 4.99.20, released 2021-01-17
=======================================

 380 files changed, 27639 insertions(+), 12103 deletions(-)

New Features:
-------------

 - Ns_Time reform

   * Support for time units:
     . arguments of functions accepting time durations can be provided
       now with a time unit suffix (such as ms, s, m, d). These values
       can be written e.g. as 1.5ms, 3h and the like.
     . units can be used as configuration values wherever appropriate
     . time values wherever appropriate in documentation
     . commands using Ns_ObjvTime can be specified now with time units
     . This change is fully backward compatible for all commands
       and configuration options accepting seconds, which was normally the case.
     . All sample configuration files have been updated with time units

     . nsproxy: The only exception to the precious sentence is nsproxy,
       where durations we specified in earlier releases as
       milliseconds. This is a POTENTIAL INCOMPATIBILITY with previous
       versions. These parameters are
        * config parameters of the nsproxy module
          "evaltimeout", "gettimeout", "idletimeout",
          "recvtimeout", "sendtimeout", "waittimeout"
        * ns_proxy get: changed "-timeout" parameter to time unit value
        * ns_proxy wait: changed optional "timeout" parameter to time unit value

    * Change granularity from seconds to Ns_Time (internally and in Tcl API)
      . Added proper handling of negative values in Ns_Time (was
        broken/not implemented)
      . Upgraded scheduling machinery from seconds to Ns_Time values
      . Extended Ns_Time arithmetic (Ns_DiffTime()) to handle positive
        and negative Ns_Time values
      . Extended regression test for ns_time significantly
      . Ns_After(), Ns_ScheduleProc(), Ns_ScheduleProcEx(): change
        argument from int secs to Ns_Time
      . Marked Ns_ScheduleProc() as deprecated, Ns_ScheduleProcEx()
        should be used instead.
      . ns_schedule_proc, ns_after: support time specs with time units
        (instead of only full seconds)
      . Ns_TimeToMilliseconds(): new API call for converting positive
        and negative Ns_Time values to ms
      . Ns_DStringAppendTime(): format times in a more compact form (no
        trailing zeros)
      . Use consistently Ns_TimeToMilliseconds() to convert Ns_Time to ms
      . Ns_ObjvTimeRange, Ns_CheckTimeRange(): support to specify time
        ranges in argument handling
      . Trim trailing zeros after comma point when printing Ns_Time,
        represent as int when possible
      . "ns_info scheduled" returns now timestamps in sub-second
        resolution (with a second faction).  By this, the runtime of the
        scheduled procedure can be determined on the sub-second level.
        (potential incompatibility)
      . Added handling of "schedmaxelapsed" on the sub-second level.

 - HTTP client request logging

   NaviServer can now optionally write a logfile for outgoing HTTP
   requests initiated via "ns_http". This is important for webservers
   that are dependent on other web services for the healthiness and
   performance. The logfile is similar to the access.log and suitable
   for automated analysis. Previously, NaviServer was "blind" on that
   data. Now it is easier to provide data about the performance of
   external services (e.g. cloud services such as authentication, Google
   Firebase Cloud Messaging, Office 365, ...)

   To activate HTTP client request logging, a new config-file section
   "ns/server/$server/httpclient" was created where logroll, logrollfmt
   and the other usual logging parameter can be specified.

  - New command line option for NaviServer: "-T"

    When this option is used, NaviServer just checks the configuration
    file (specified as usual by "-t") for syntax errors.  This option
    should be used on production sites to reduce the downtime of a
    server in case of typos.

 - Added support for "100 CONTINUE"

   This feature is in HTTP since a very long time, but not widely
   supported although useful (best support is currently in curl). When
   the client sends "Expect: 100-continue", the server can check the
   header fields and provide an early feedback, whether it is useful to
   transmit the full body. This way, e.g. content which is too large can
   be rejected quickly.

 - Improved server liveliness under high load (improved and configurable
   handling of connection queue overrun)

   When NaviServer processes an HTTP request, it is received (head and
   body) and associated with a connection thread pool. If the thread
   pool has a thread free, the request is directly assigned to this
   thread. Otherwise, the request is added to a queue of the connection
   pool, waiting for a connection thread.  Since the queue has a limited
   size (limited by "maxconnections"), this queue might overrun. In this
   situation, the received request is added to a waiting list. Here is,
   where the behavior is now improved.

    1) previously, as soon as there were entries in the waiting list, no
       more fresh requests were accepted by the driver (just working on
       waiting requests). While this behavior is sensible if only one
       connection pool is available, it causes a blocking behavior in
       cases, when a single connection pool overruns but others could
       continue. Now, more fresh requests are still accepted in this
       situation.

    2) In addition, it is now possible to configure the overrun behavior
       via two pool parameters:
       a) pool parameter "rejectoverrun": when set to true, NaviServer sends on
          queue overruns an error "503 service unavailable". The overflowing
          request is not added to the waiting list.
       b) when (a) is set, it is possible to specify parameter "retryafter"
          with a time interval to add reply header field "Retry-After" to the
          503 response. (see https://tools.ietf.org/html/rfc7231#section-7.1.3)

   When "rejectoverrun" is false (default) then behavior is similar to
   before, accept the server does not fully block. The request is kept in
   the waiting list, which is checked, whenever a request finishes. For
   the client, there is no need to resubmit the request, which will be
   proceed eventually. Technically, this means an arbitrary large queue.

   When "rejectoverrun" is true, then the request causing the overrun is
   rejected, and will have to be resubmitted by the client (maybe
   automatically, when "Retry-After" is specified. However, the client
   support for "Retry-After" is rather bad.

   For busy sites, "rejectoverrun" is probably the better behavior,
   since no background queuing in the waiting list is performed. For
   now, we keep the old behavior as default for better backwards
   compatibility.  The default might change in the future.

 - General support for running NaviServer behind a reverse proxy server

   This change provides general support for running NaviServer behind a
   reverse proxy.  Previously, the support was rather punctual (e.g. for
   nslog). Reverse proxy mode is activated by setting the global
   parameter (in section "ns/parameters") named "reverseproxymode" to
   true (default false).

   Current limitation: In the current implementation, NaviServer supports
   only the "X-Forwarded-For" header field, which is supported by most
   proxy servers, treating the first provided IP-address as the client
   IP address (see https://en.wikipedia.org/wiki/X-Forwarded-For). The
   newer "Forwarded" header field is currently not supported.

   In the reverse proxy mode, the client address is taken as forwarded
   by the reverse proxy. This has the following consequences:

   * Reported IP address by the following commands.

          ns_conn peeraddr
          ns_server all|running
          ns_writer list
          ns_connchan list

   * Reported IP address in the access.log

   * Context filters (used e.g. for mapping requests to pools)

     In reverse proxy mode, all IP-address specific context filters
     refer to the x-forwarded-for addresses. This makes it possible to
     use context filters (of URLspace commands) based on IP-addresses
     (IPv4, IPv6, with masks) also behind reverse proxy
     servers. Previously the IP-based mapping was just possible when no
     reverse proxy was in use.

 - Allow a single driver to listen on multiple ports:

   This new feature simplifies setups, where a single server is
   listening on multiple ports using the same driver. Previously, it was
   necessary to define separate drivers for such cases, which need
   different names but which are often configured identically.  Together
   with the change introduced in 4.99.19 (support for multiple listen
   addresses) this means that when multiple addresses and multiple ports
   are specified for a single driver, it will listen on the cross
   product of all combinations. The maximum number of listening ports of
   a driver is limited by a compile time constant to
   MAX_LISTEN_ADDR_PER_DRIVER (per default 16).

   To use this feature, specify in the configuration file for "port" in
   the network driver section a space separated list of multiple
   ports. This change is fully backwards compatible, old configuration
   files will continue to work.

 - SSL/TLS improvements:

   * Support for OCSP Stapling

     . NaviServer obtains and provides information about the validity
       of the used certificates. For proving this information, the
       server performs two level of caching: in-memory caching and
       disk caching. When the server receives the first TLS request
       with OCSP stapling turned on, it checks for an already
       retrieved OCSP response. The disk cache file is saved in the
       "log" directory of the server and uses the serial number of the
       certificate to be checked as filename with ".der" as
       extension. When the disk cache file does not exist, an
       http/https request is made to the server issuing the servers
       certificate as defined by the Authority Information Access
       (AIA) Extension. The names of the file and the http/https
       request for the OCSP response can be obtained from the system
       log of the server

     . OCSP Stapling can be turned on in the configuration files (see
       sample configuration files)

     . OCSP Stapling requires OpenSSL 1.1.0 or newer

  * Support for server-side SNI (Server Name Indication (SNI)

     Server-side SNI can be used to use different certificates for
     virtual hosts. In addition to the top-level certificate, different
     certificates can be defined per virtual server (below: srv2). The
     certificate will be registered for all alternate names specified
     for the server.

     In addition to the certificate, also all other OpenSSL specific
     parameters (ciphers, protocols, ...) will be loaded form the same
     place as the certificate. The following example shows the setup for
     two virtual servers "srv1" and "srv2", where the first one is the
     default server.

        ns_section ns/modules {
          # ...
          ns_param        nsssl           ${bindir}/nsssl.so
        }
        ns_section ns/module/nsssl {
          # ...
          ns_param   defaultserver   srv1
          ns_param   certificate     /usr/local/ns/modules/nsssl/srv1.pem
        }
        ns_section "ns/module/nsssl/servers" {
           ns_param   srv1            test.org
           ns_param   srv1            www.test.org
           ns_param   srv2            someotherhost.org
        }
        ns_section "ns/server/srv2/module/nsssl" {
          # ...
          ns_param   certificate     /usr/local/ns/modules/nsssl/srv2.pem
          ns_param   ciphersuites    TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        }

     SNI support includes OCSP, DH ephemeral key exchange, reuse of SSLcontexts.


   * Support for OpenSSL "ciphersuites" (TLSv1.3 feature)

   * Support for client-side SNI: new parameter "hostname" for
     "ns_connchan open"

   * Support for removing TLSv1.0, TLSv1.1, TLSv1.2, and TLSv1.3 via
     configuration file


 - ns_http:

    * Clear separation of "-timeout" and "-expire":
       . "-timeout" is used for single communication acts (e.g. connect, read write)
       . "-expire" is used for limiting the full duration of the
         request (also useful for slow read/write attacks).

      Scripts already using the -timeout will most probably not see
      any change unless they are handling huge up/downloads. In such
      cases we will now continue with the request, instead of
      terminating it in the middle.

    * added "-binary" flag to transmit data binary and perform no
      automatic encoding transformations.

 - moved code repository on BitBucket from mercurial to git.
   * adjusted documentation
   * adjusted build system (Makefiles)



 - Logging improvements:

   * New log severity "Debug(timeout)" to force NaviServer to write an entry to the
     systemlog, whenever a timeout occurs. Therefore, the command

        ns_logctl severity Debug(timeout) on

     can be used to debug cases, where NaviServer runs into HTTP status code 503
     (Service Unavailable).

   * Logging of commands with high latency:

     Provide a consistent interface and naming convention for logging
     slow commands for nsproxy, ns_jobs, scheduled procedures and SQL
     (nsdb drivers).

     . nsproxy and nsjobs have a new configuration option "logminduration"

       This configuration option specifies a time-limit for logging
       (long) eval operations to the system log (similar to
       "logminduration" in the database drivers). Set it to a high value
       to avoid logging (e.g. 1d). The default is 1s.

     . scheduled procedures:
       Changed "schedmaxelapsed" to "schedlogminduration" for consistency.
       This is a potential incompatibility for old configurations.


 - New commands:

   * "nsv_dict":

      The new command "nsv_dict" provides thread-safe support for
      setting, getting, updating, and unsetting dict values in nsv
      array.

      Motivation: Some larger OpenACS installations have currently more
      than 1 mio (!) nsv arrays, which comes mostly from creating nsv
      arrays per packages. Most of these arrays have only a few keys
      with very short values. The number of arrays can be greatly
      reduced by supporting complex values (dictionaries), similar to
      REDIS hash values.

      . "nsv_dict" supports nested dicts like in Tcl

      . The following dict commands:

          nsv_dict append  array key dictkey dictvalue:0..n
          nsv_dict exists  array key dictkey:1..n
          nsv_dict get     array key dictkey:0..n
          nsv_dict getdef  array key dictkey:0..n default
          nsv_dict incr    array key dictkey ?increment?
          nsv_dict keys    array key ?pattern?
          nsv_dict lappend array key dictkey dictvalue:0..n
          nsv_dict set     array key dictkey:1..n dictvalue
          nsv_dict size    array key
          nsv_dict unset   array key dictkey:1..n

     . added 41 test cases in regression test for nsv_dict

   * "ns_dbquotelist": similar to "ns_dbquotevalue", but provides
     quoting for Tcl lists.

   * "ns_trim": multiline trim command:
      ns_trim ?-subst? ?-delimiter delimiter? text


 - New features for existing commands/subcommands

   * New subcommands "keys" and "values" for "ns_set":

        "ns_set keys /setId/ ?pattern?"
        "ns_set values /setId/ ?pattern?"

     The commands are similar to the Tcl command "dict keys" and "dict
     values" and can reduce the necessity for loops.

   * Form processing:

      . "ns_getformfile": return a list of filenames when "multiple"
        attribute is provided in the input field and multiple files are
        submitted

      . Decouple content-type processing from HTTP methods:

        Previously, parsing of form data in these two formats happened
        only on POST requests. However, several REST services use often
        also PUT with form data and distinguish POST and PUT methods
        mostly by "add item to a collection" (POST) and process single
        item (PUT) but not by their content-type processing. This change
        eases implementation of REST services.

      . Perform more strict parsing based on "ns_parsefieldvalue" and
        "ns_parseheader" in file-based form parser (for spooled
        content).  Previously, there was a bug with the regexp-based
        parsing, when e.g. a filename of and upload file contained
        e.g. a semicolon.

   * "ns_conn":
      . new subcommand "headerlength"
      . "ns_conn auth" returns now Information about "Bearer"
        authentication (RFC 6750) when applicable, used e.g. in OAuth2
      . "ns_conn peeraddr": New flag "-source" for specifying detailed
        information about the peer, especially useful in combination
        with reverse proxy mode; allowed values are "configured",
        "direct", "forwarded".

   * "ns_cookie": Support explicit setting of "same-site=none"

     In previous version of browsers, no explicit setting
     of the samesite flag was exactly the same as explicit setting.
     Since some browsers switched to a default of "lax", explicit
     setting became necessary.

   * "ns_dbquotevalue": provided a C implementation; was scripted
     before.

   * "ns_connchan":
      . Added option "-server" to "ns_connchan close" to be able
        to cleaning stale handles of every server.

      . Added C-level WebSocket support and buffered output handling.
        The new code is about 200x faster than the Tcl version
        (especially when bytewise XORs are needed), simpler to use (no
        Tcl-leval handling of incomplete frames, split frames, remaining
        leftover from partial writes) and more complete (handling of
        segmented WebSocket messages).

      . "ns_connchan read -websocket ....": returns a dict containing
        the WebSocket data (including payload, WebSocket status bits,
        buffering state, etc.; see documentation and NaviServer
        websocket module for details)

      . "ns_connchan write -buffered ..." handles partial writes via an
        internal buffer (Tcl developer has just to take care about
        writable callbacks when necessary)

      . "ns_connchan wsencode ...": WebSocket frame generation, returns
        binary data based on input parameters suitable for sending on a
        WebSocket channel.

      . "ns_connchan status ..." returns a dict containing the buffering
        status of a channel.

   * "ns_urelencode": added one more option "oauth1" for the "-part"
     argument. The "oauth1" encoding is defined in RFC 5849 Section 3.6
     for the construction of the signature base string and the
     "Authorization" header field.

   * "ns_ictl":
      . "ns_ictl trace idle ...": new tracetype "idle", which fires,
        when a connection thread is idle for the threadtimeout period,
        but minthreads is already reached, and the thread won't
        terminate.

      . "ns_ictl maxconcurrentupdates ?max?": Configuration option
        for specifying the maximum number of concurrent updates
        for interpreter reloads (when epoch increases, e.g. reloads
        on OpenACS).

        Background: For large (say 50-100 connection threads) and busy
        sites (sustained 800 requests per second) with large
        blueprints (e.g. OpenACS) blueprint updates can cause delays
        when all these operations are concurrently performed. The
        blueprint updates happen when the epoch is incremented
        (e.g. in OpenACS: reloads of packages).  A too high number of
        concurrent blueprint updates cases (a) delays in request
        processing (b) increased update times per thread (e.g. up to
        10 seconds). As a consequence, queueing will happen.  This new
        option limits the number of concurrent updates (e.g. to 10)
        leading to smooth operations in such situations. When the
        maximum number of concurrent blueprints is reached, the pending
        updates are delayed until the limit is sufficiently low. By
        default, the value is unlimited.

   * Improved statistics:
     . "ns_server stats" returns now as well dropped requests from queue
       overruns.

     . "ns_info locks": report number of read and write operations for
       rwlocks.

     . "ns_proxy stats": report also the number of running worker
        processes.

   * "ns_getcsv":
     . provide compatibility with RFC 4180
     . accept multi-line input
     . handle quotes in quotes and empty elements
     . optionally stripping unquoted elements (not covered by RFC)

     . optional support for a different quote character (option
       "-quotechar"). Although RFC 4180 specifies double quotes as
       quoting chars, sometimes also different quoting characters,
       which can be now successfully decoded.

   * "ns_parseurl": Support for fragment parsing.  When provided URL
     contains a tail element, this is now parsed info a "query" and/or
     "fragment" component in the result. Previously, the query and
     fragment part was included in the "tail" element.

   * Improved thread naming for log-files: previously, it was not clear,
     from which server some output of the "main" thread originated.
     The new version adds always a server name specific suffix.

   * "ns_driver info": return also "port" and "defaultport"




Bug Fixes:
----------

 - Improved support for newer versions of OpenSSL (handling of
   SSL_R_UNEXPECTED_EOF_WHILE_READING)

 - ns_cache: Base cache size calculation on the full size of a
   cache entry.

   In previous versions, the cache size calculation was based only on
   the value size, leaving the size of the key and the size of the
   overhead (size of cache entry, about 126 bytes on a 64-bit machine)
   aside. This is misleading, when the cache value
   size is 0 bytes (just using the cache key as a flag), leaving admins
   wondering about the memory consumption.

 - Report to log file, when ns_ictl traces are not executed. Previously,
   these were silently ignored (e.g. in some error cases).

 - Fixed a bug with PUT methods, when query-parameters were passed in
   addition to the put content.  Previously, these parameters were only
   available via [ns_queryget ...] when the content file was not
   spooled.

 - Fixed bug for custom error pages for HTTP methods other than GET
   (Custom error handles are defined as before via redirect sections):
    * Always use GET in the redirection not the original HTTP method.
      When e.g. "PUT /somepage" is requested, where no PUT handler
      but a custom error page /405.tcl, the previous code tried a
      PUT /405.tcl, leading the same error. Now, the error
      page us delivered like with a GET request.
    * Provided a hint as a warning, where NaviServer searches
      for non-existing pages.
    * Extended regression test: cover HEAD requests for existing
      and non-existing pages, and various forms of redirects.

 - Fixed issue with tDOM as noted by Maksym Zinchenko on the
   naviserver developer list:

   A recent change of tDOM 0.9.2 introduced a namespace local command
   "namespace" which was picked up by the blueprint generator
   incorrectly instead of the global ::namespace command.  As a fix, all
   "namespace" invocations in the blueprint generator were replaced by
   "::namespace" to avoid considering invocation context for every
   occurrence.

 - Fixed potential abortion of regression test runs under macOS:

   Under macOS, tests runs were sometimes abruptly ended with a
   broken pipe signal. The call "NsBlockSignal(NS_SIGPIPE);" does
   not avoid these signals. So, we ignore bluntly all such
   signals by using "NsBlockSignal(NS_SIGPIPE);"

 - Fixed charset handling for spooled requests.

 - Fixed several cases of invalid chunked data (potential buffer
   overflows).

 - Don't swallow errors in braced config sections in the configuration file.

 - Fixed a potential crash, when certain configuration section were missing,
   but code tried to obtain configuration values from such sections.

 - Blueprint generation:
    * Fixed bug with interp aliases defined without fully qualified
      namespace path
    * Improve robustness when "rename" is used (when source exists and
      target does not exist)

 - Support calling "ns_conn status" (with the argument setting the HTTP
   return status) in traces (without open connection).

 - Make code more robust in case Tcl and NaviServer were compiled
   with different memory allocators. The mix if memory allocators
   could crash NaviServer.

 - Race Conditions:
    * Fixed several race conditions in "ns_http"
    * Fixed race conditions in "ns_connchan list" and "ns_connchan create"

 - Use time_t for secs in Ns_Time, since long is just 32-bit on 64-bit
   windows, but time_t is 64 bits.

 - "ns_writefp": fixed to act as advertised (read to end of file when no
   length is provided)

 - "ns_sendmail": Convert leading "." in message bodies to ".." as
    required by RFC 5321 section 4.5.2

 - "ns_cache" transactions: in certain situations, where during
    concurrent entry creates, and one of the create attempts failed and
    the other succeeded, and the one of the cache transitions is rolled
    back, updates could lead to an error.

 - "ns_connchan": Fixed registry of connchans from all servers to the
    default server (not the best scalable solution, but useful as long
    we have only one "socks" thread (for more details see:
    https://sourceforge.net/p/naviserver/mailman/message/37179476/)

 - Fixed a 20-year-old time bomb in HTTP time parsing. Server was rejecting
   to parse HTTP time strings for 2021.


Performance improvements:
-------------------------

  - nsv_lappend:

    The old implementation showed a linear slowdown for nsv_lappend
    operations depending on the number of elements in the list. The
    new implementation avoids frequent string to list and vice versa
    operations by using Tcl_DStrings. It is essentially independent on
    the number of elements in the nsv.  For a list with 10K elements,
    the performance improvement is about a factor of 400.

    Test:
       nsv_set d 1 1
       time {nsv_lappend d 1 1} 10000

    OLD:
       403.7255 microseconds per iteration

    NEW:
       0.860547 microseconds per iteration

   This change makes it possible to use the OpenACS developer support
   on busy sites (before, this led to a quick standstill).

  - nsv_append:

    The performance gain is much less than with nsv_lappend (see
    previous commit), but still noticeable (about 35% faster, 10K
    ns_append operations went from 0.94 microseconds to 0.62

  - Locking:
    * Reduced mutex-locked regions in driver.

    * Use ptrhreads interface for Ns_RWLock* when available

      So far, Ns_RWLock* used an implementation based on mutexes and
      cond vars. This implementation has the advantage of being
      portable. However, since on all *nix platforms, NaviServer uses
      the pthread library, using the POSIX rwlock implementation poses
      no additional dependency.

      Performance comparison fur using mutex/old rwlocks/pthread rwlocks
      on ns_urlspace;

            macOS (8 cores) rwlock (POSIX)  0.3105    2,27 %
                            rwlock (ns)     11.5263  84,23 %
                            mutex (ns)      13.6839 100,00 %

            Linux (6 cores) rwlock (POSIX)  0.6081   43,38 %
                            rwlock (ns)     3.6259  258,66 %
                            mutex (ns)      1.4018  100,00 %

      This data shows that the POSIX rwlocks implementation
      (pthread_rwlock_rdlock etc.)  is always better than the classical
      NaviServer rwlock implementation, and that significant
      improvements can be achieved by using rwlocks. A more detailed
      performance comparison was provided on the developer mailing list.

      This change introduces:
      a) Ns_RWLock* based on pthreads when available
      b) switch to recommended configure setup for pthreads (AX_PTHREAD)
      c) New API call: Ns_RWLockSetName2() modeled after Ns_MutexSetName2()
      d) New API call: Ns_RWLockList() modeld after Ns_MutexList()
         (removed dependency of rwlocks to mutex statistics)

   * Support rwlocks for nsvs:

     Per default, rwlocks are used for all nsv* commands.  The default
     behavior can be altered by setting the configuration variable
     "nsvrwlocks" to false in the section ns/server/${server}/tcl of
     the configuration file.

     This change can lead to improved scalability especially on
     multi-core machines since in read cases, the variables can be
     accessd unblocked for parallel executions. In most applications nsv
     are typically more often read than written rwlocks offer better
     scalability for large sites.

     The following table compares the busy locks with mutex operations vs
     the number of busy locks with rwlocks for openacs.org:

       With mutex (24h)
                                    locks   busy
       nsv:3:openacs.org    27      4.71M   1.3K
       nsv:6:openacs.org    24      4.88M   1.03K
       nsv:2:openacs.org    28      3.37M   784
       nsv:7:openacs.org    23      9.11M   755
       nsv:5:openacs.org    25      2.88M   460

       With rwlocks (24h)
       nsv:7:openacs.org    1       7.22M   0
       nsv:6:openacs.org    2       3.92M   143
       nsv:3:openacs.org    5       3.31M   1
       nsv:2:openacs.org    6       2.23M   16
       nsv:5:openacs.org    3       2.16M   0

     On more busy sites, the improved scalability si much higher.

   * rwlocks are better in cases, where the number of readers is
     significantly higher than the number of writers.  So, it has to
     be carefully considered in which cases rwlocks should be
     preferred.  The new version uses rwlocks for nsv, ns_connchan,
     filters and tcl.cache (rwlocks for nsv and filters can be
     deactivated via configuration file). Earlier version used
     rwlocks already for adp tags, epoch handling and permissions.

 - Avoid multiple library initializations for OpenSSL 1.1.0 or newer.


Documentation improvements:
---------------------------

 - Update references to Tcl website (replace http://tcl.tk by
   https://core.tcl-lang.org)

 - Added hints about required OpenSSL version to man pages

 - Made manual pages more consistent
 - Consistent indentation of program examples in documentation
 - Made source code documentation more consistent

 - Improved source code documentation (provide more details, remove
   obsolete comments)

 - Removed deprecated man pages
 - Removed obsolete sections in manual pages
 - Follow the recommended spelling convention of the Linux documentation
   project
 - Improved spelling, typesetting, and cross references
 - Reduced technical jargon
 - Made code listings in documentation more consistent (e.g. indenting)
 - Added missing commands in command list (overview page)
 - Updated listing of deprecated commands and suggested alternatives
 - Replaced several "slave process" by "child process"

 - 161 manual pages were updated since the last release


Configuration Changes:
----------------------

 - New configuration options:

    * Section "ns/parameters"

      . "reverseproxymode": Activate reverse proxy mode.  The former
         configuration parameter of "nslog" named "checkforproxy" is
         obsolete now.
      . "joblogminduration": report jobs taking longer in systemlog
      . "schedlogminduration": report scheduled jobs taking longer in systemlog

    * Section "ns/server/${server}"

      . "rejectoverrun": when true send 503 when queue overruns;
         default "false"
      . "retryafter": time for Retry-After in 503 cases

    * Section "ns/server/${server}"
      . "filterrwlocks": use RWlocks for filters (default: true)

    * Section "ns/server/${server}/tcl"
      . "nsvrwlocks": use RWlocks for nsv (default: true)

    * Section "ns/server/$server/httpclient"
      . "logging": (default: off)
      . "logfile": name of http client logfile
      . "logrollfmt": format appended to log filename
      . "logmaxbackup": max number of backup log files (default: 10)
      . "logroll": perform automatic rolling (default: true)
      . "logrollonsignal": (default: false)
      . "logrollhour": specify at which hour to roll (default: 0)

 - Sample configurations:

    * Added values with time units where appropriate
    * Added new values mentioned above with short comments

    * For multi-server configuration files: use always the first (and
      not the last) server as default server.

    * openacs-config.tcl:
    * sample-config.tcl:

       . fixed config parameter: use "dnscachetimeout" instead of "keepwaittimeout"
         (name was changed in NaviServer ages ago, but change was not
         reflected in config files

   * nsd-config.tcl:

   * win32/test/nsd.tcl
     modernized test config file

   * nsproxy: Changed term "maxslaves" to "maxworkers" (the old term
     is deprecated)


Code Changes:
-------------

 - Fixed potential access to an out-of-scope stack variable.
 - Added more declarations for PURE and CONST functions
 - Replaced usage of reserved identifier [cert-dcl37-c,cert-dcl51-cpp]
 - Replaced sscanf() by strtol*() to increase code-safety
 - Improved structure packing

 - Marked "ns_adp_mime" explicitly as deprecated, and provide deprecated
   messages for it.  The command was deprecated more than 10 years ago,
   ns_adp_mimetype should be used

 - Added mime types for "heif" and "heic" which have become a preferred
   image type for iOS and macOS.

 - Regressions testing:

   * Reduced usages of old-style nstest::http-0.9 (socket and not
     ns_http-based tests)

   * Make test more robust concerning CRLF results (Windows)

   * Extended tests:
     367 tests were added since the last release
     (changes in 38 test files)

 - Improved logging:
    * Added metering for long wait times of Ns_CsEnter locks
      (e.g. Ns_MasterLock)
    * Added concurrency level to log message of interp updates
    * Added warnings, when an NsTclRequestProc or an ADP request runs
      into a timeout.

    * Reduced time sensitivity on time when rolling log files_ Rolling
      log files happens often at midnight, using often a day
      precision. When e.g. a scheduled procedure the time when this
      function is called might be slightly after the scheduled time,
      which might lead to a day jump. The problem aggravates, when
      multiple log files are rotated.  The new code identifies day wraps
      and uses in such cases the earlier day.

 - Improved portability:
   * Improved compatibility with the forthcoming OpenSSL 3.0
   * Improved compatibility with various versions of LibreSSL
   * Improved Windows (MSVC) compatibility
     . Simplified Makefile.win32 for nmake
     . Fixed thread exit result handling for 64-bit Windows
     . Fixed thread handle access
     . Fixed default temp directory for windows
     . Turned on OpenSSL by default
     . Minimal support for running regression tests under windows
     . Fixed windows size incompatibility for range requests (64-bit
       Windows uses a 32-bit sized offset type off_t)
     . Fixed ns_sockselect for 64-bit windows installations
     . Fixes for cross-compiling for Windows 64-bit (using gcc/mingw)
   * Use consequently NS_EAGAIN (can be used in *nix and Windows)
   * Fixed potential problem with GNUC_PURE for older version of gcc
   * Removed dependency of tests from an explicit "/tmp" directory
   * Fixed regression test warnings with older versions of curl for IPv6 addresses
   * Fixed m4 configuration for "crypt_r" (broken since many years)
   * Compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD
     6.8 (LibreSSL 3.1.1), FreeBSD 12.2

 - More OpenSSL changes:

   * Reactivate CRYPTO_set_mem_functions() for OpenSSL 1.1.0 or newer
     (interface om OpenSSL has changed)

   * Switched to SSL_CTX_set_dh_auto() from low-level functions: In
     newer versions of OpenSSL, all low-level DH* functions were
     deprecated.


 - Build-system
   * Compute version tag from git similarly like to the mercurial version.

 - improved support for MS windows
   * Windows math.h does not include round(), add simple implementation.

 - Added cross-platform function ns_localtime_r (localtime with
   user-provided storage): The previously defined function
   ns_localtime() uses a single per-thread local storage, so, it was not
   possible to have in one thread two time buffers with different
   values.



- Improved error code handling on driver receive operations

   Previously, higher level of API calls were not able to access
   realiably the error information detected on the lower-level (async
   I/O-based or OpenSSL based) driver operations. This made it
   impossible to provide good error messages e.g. in "ns_connchan" and
   we could not provide error processing based on Tcl error
   codes. Furthermore, the new interface works for POSIX errnos and
   OpenSSL errors and improves the situation (currently just for driver
   read operations).

   * Ns_Sock* based setting of Tcl error codes:

       NsSockSetRecvErrorCode(const Sock *sockPtr, Tcl_Interp *interp)

     Set the Tcl error code in the interpreter based on the
     information provided by Sock* (actually by recvErrno); the
     function works for OpenSSL and plain POSIX style error codes.

   * Setting Tcl error codes for POSIX errno:

        Ns_PosixSetErrorCode(Tcl_Interp *interp, int errorNum)

     Set the Tcl error code in the interpreter based on the POSIX
     errorNum and return the error message.

   * Setting Tcl error codes for OpenSSL error values

       Ns_SSLSetErrorCode(Tcl_Interp *interp, unsigned long sslERRcode)

     Set the Tcl error code in the interpreter based on the OpenSSL
     error value and return the error message.

   * Return the last error from a socket.

        Ns_SockErrorCode(Tcl_Interp *interp, NS_SOCKET sock)

     The function Ns_SockErrorCode() returns the last error from a
     socket. When the provided interp is not NULL it sets the Tcl error
     code based on the error code.


Modules:
--------
 18 files changed, 1858 insertions(+), 1195 deletions(-)

nsdb:
  * Added support for time units for nsdb LogMinDuration

nsdbpg:
  * Force bindvars to start with alnum or underscore (otherwise,
    surprises with e.g. lines ending with a colon might arise).
  * Align spelling with LDP recommendations.
  * Avoid shadowing of symbol "bind".

nssmtpd:

  * Escape all leading dots according to RFC 5321 section 4.5.2 in mail bodies
  * Fixed check, if relay host announced STARTTLS
  * Provide compatibility for versions before 4.99.19, when NS_FALL_THROUGH was not defined yet
  * Subsecond time resolution (Handled interface changes in post 4.99.19)
  * Support for 'ciphersuites' and 'protocols' parameters of 'Ns_TLS_CtxServerCreate'
  * I/O overhaul
   - Replaced select() call by poll() to ensure scalability
   - Pushed handling of EAGAIN etc. to the lowest level
   - Removed one layer of partial write handling
   - Replaced huge send buffer attempts, since these seem to be rejected sometimes
   - Improved robustness (never go into an infinite retry loop)
   - Handle potential "resource is not available" while sending SMTP data
  * Improved cleanness of compilation
  * Improved documentation
  * Align spelling with LDP
  * Code cleanup (C and Tcl code)
  * Extended regression test

nsdns:
  * Code Cleanup
    - Added NS_EXPORT to Ns_ModuleInit
    - Improved cleanness of compilation

nsudp:
   * Minor code cleanup (type cleanness, documentation)

nsphp:
   * Adjust to changed parameter types in PHP 7
   * Improved cleanness of compilation
   * Improved documentation
   * Brought test environment in sync with naviserver

nsstats:
   * Improved XSS protection for potentially user supplied strings
   * Improved statistics:
     - Added calculated maximum number of reqs/pool
     - Added "maxconnections" to the per-pool statistics,
       since it determines the queue length
     - Improved nsproxy output in "process" page (include seconds,
       don't truncate everything to 0 digits after the dot)
     - Added write-percentage for RWLock in lock statistics
     - Add statistics for dropped requests (queue overruns)
     - Added nsv usage statistics
   * Added web-interface for toggling between raw and human centric values
   * Aligned with fractional seconds interface in NaviServer 4.99.20
   * Improve help messages

nsoracle:

   * Changed the include order to avoid bad interactions between oci.h and zlib.
     with the previous include order, we run with recent oracle versions (19.5) into
        /usr/include/zlib.h:1677:4: error: unknown type name ‘off64_t’
     on 64-bit Linux systems (centos 7)

  * Added ORA_CFLAGS to allow developers to pass arbitrary include and library flags.
    Background: for some (newer?) version of the oracle client library,
    ORACLE_HOME (see README) seems to be gone.

  * Changes to build nsoracle on Windows with new Makefile.win32 file.
  * Improved cleanness of compilation

websocket:
  * Major rewrite, using C-level implementing in ns_connchan

    This version make uses of the new C-level support for WebSockets.
    The new version is several orders of magnitudes faster than the
    previous version, it is more robust (C-level handling of
    partial read and write operations) and it is more complete
    than previous versions (handling segmented messages).

  * Added deprecated function ::ws::build_msg for providing
    better backward compatibility
  * Provide name for mutex lock to ease tracing of long locks
  * Added client-side SNI (requires NaviServer 4.99.20)
  * Added simple WebSocket client interface

revproxy:
  * Improved exception handling (use try interface)
  * Preserve existing X-Forwarded-For IPs in cases where nsproxy is
    itself behind a proxy or load balancer
  * Switch to "ns_connchan write -buffered" to simplify leftover
    management for partial writes (requires NaviServer 4.99.20)
  * Fixed bug fix for potential stale handles
  * Code cleanup
    - Fine-tuning logging messages
    - Breaking overlong lines, reduce verbosity

letsencrypt:
  * Minor changes in documentation

nsldap:
  * Fixed result setting of "connected" subcommand
  * Updated modules interface to usual conventions
  * Added config parameter "schema" to specify other schemata
  * Improved documentation in README file
  * Reduced number of deprecated calls

nsshell:
  * Added HTML-escaping on results of Tcl commands before these are
    displayed in the shell.