Forum OpenACS Development: Announcement: NaviServer 4.99.18 available

Dear all,
I am pleased to announce the availability of NaviServer 4.99.18 [1,2]. This release is primarily a bug-fix release for installations still using Tcl 8.5, or installations having a half-working IPv6 installations. Upgrade from 4.99.17 is recommended. Below is a summary of changes.
Many thanks to all contributors!
all the best
-gustaf neumann
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.18/
[2] https://bitbucket.org/naviserver/naviserver/
=======================================
NaviServer 4.99.18, released 2019-02-24
=======================================

 133 files changed, 3314 insertions(+), 2106 deletions(-)


New Features:
-------------

 - Added ability to provide memory sizes via memory units (kB, MB, GB,
   KiB, MiB, GiB) in the configuration files and commands. We follow
   the approach taken by e.g. PostgreSQL and use 1024 as
   multiplier, which is e.g. useful for buffer sizes. All memory
   sizes in the config files can be specified in memory
   units. Currently, the only commands supporting memory units are
   ns_cache_create, ns_cache_configure and ns_http. The configuration
   based on integers without memory units continues to work.


 - Portable config files (windows compatibility)

   Rationale: unixoid platforms use traditionally ".so" as extension
   of binary modules, no matter what "info sharedlibextension" is. On
   macOS, the sharedlibextension returns ".dylib", while the module is
   named ".so". Under windows, it seems that the extension of the module
   has to be ".dll", so we can't write portable config
   files without extra effort.

   The new code tries now first to load the library as specified for
   backward compatibility (maybe containing a file extension). When
   this fails, it tries to append either ".so" in unixoid platforms or
   ".dll" under windows.

 
 - new API call: ns_db currenthandles

   Return information about handles currently allocated to the thread
   performing this call. The result contains a dict with the pools,
   from which handles were allocated, the names of the handles and
   their "active" state (a handle is active between a "ns_db select"
   and the last "ns_db getrow" statement). If no handles are allocated
   by this thread, empty is returned.

   For example, a possible result is "main {nsdb0 1} subquery {nsdb1
   0}" indicating that the current thread has allocated handles from
   pool "main" and "subquery", but the handle from pool "main" (i.e.
   "nsdb0") is currently active (during a select).

   This feature requires Tcl 8.5 or newer.


 - More security features: SameSite support for cookies

   The command ns_setcookie supports now the flag "-samesite" with
   permissible values "strict|lax|none". When the flag is set it
   prevents the browser from sending this cookie along with cross-site
   requests to mitigate cross site scripting attacks. Permissible
   values are [term strict], [term lax], or [term none]
   (default). While the value [term strict] prevents sending the
   cookie to the target site in all cross-site browsing context, the
   value of [term lax] allows sending the cookie when the user clicks
   on regular links. For details, see
   https://www.owasp.org/index.php/SameSite

   The "SameSite" cookie flag is not yet part of an RFC, but most major
   browsers support it. Browsers that do not support it, ignore the flag
   silently (see https://caniuse.com/#search=samesite).

   Although most cookies should probably use the flags, in order to
   provide backward compatibility, the flag can't be activated by
   default on all cookies.


 - Better entries in server log for scheduled procedures.

   With this change, one can determine exactly, which scheduled thread
   was executing an instance of which scheduled procedure. Similarly
   as for connection threads, one can now distinguish, whether two
   log-entries are from the same instance of a scheduled proc or from
   different ones.

 - nsdb stats: Return as well number of currently connected DB connections.


Performance Improvements:
-------------------------

  - nsproxy: Fixed block in reaper thread that led to excessive time
    spent in the main reaper loop, thus blocking the whole proxy
    interface. This change improves concurrency and reduces potentially long
    locks of the full nsproxy machinery.


Bug Fixes:
----------

  - Improved combability with Tcl 8.5
    * make sure package "try" is used in regression test cases where needed

  - ns_http
    * Fixed potential problem with partial writes on HTTP requests
      with large bodies (e.g. POST of larger files)
      
    * Aligned implementation with documentation (if "wait" operation has no
      timeout, use timeout of queue operation)
      
    * Improved client connect() operations with half-configured IPv6
      installations.  This change addresses especially a problem, where
      IPv6 is activated on a server, but it cannot be used to connect
      to the outside world.  In these situations, connect operations to
      hosts with domain names having both IPv4 and IPv6 addresses
      would fail, although IPv4 connections are possible. In this
      cases, one should actually either deactivate IPv6 on the machine
      or probably deactivate IPv6 name lookups in bind (option "-4",
      filter-aaaa-on-v4 yes;). With this change, this should be
      necessary. This change fixes also a memory and socket leak in
      this situation.

  - Cache transactions: make "ns_cache flush $cache" behave like
    "ns_cache_flush -glob $cache *" on "ns_cache_transaction_rollback".
    Previously, when the full cache was flushed, the rollback was not
    performed properly.
  
  - tcl_crypto:
    * align code with documentation (provide a default for option "-digest")

  - nscgi:
    * fixed potential memory leak

  - ns_sourceproc: fix deprecated function in cases in error cases,
    make clear in documentation that ns_register_procns_register_tcl
    is the preferred approach (many thanks to Russell Sorensen for
    reporting)

  - tcl/nstrace.tcl: undo change introduced with 4.99.17 to exclude
    content of ::tcl namespace since this causes problems with Tcl 8.5
    (problems with clock or "package require try" when executed during
    initial load phase.

  - Ns_VarUnset(): Deleting the entire nsv array when last value is
    deleted (many thanks to Andrew Piskorski).


Documentation improvements:
---------------------------

   - All documentation: use memory units when appropriate
   
   - doc/src/naviserver/ns_job.man: aligned documentation with
     implementaton; use itemized list for describing entries in the
     result lists.
     
   - doc/src/manual/admin-maintenance.man:
     Improved sample service file
     
   - nsssl/doc/mann/nsssl.man:
     Improved sample service file
     
   - doc/src/naviserver/ns_log.man
     Added sample configuration for configuring logging behavior

   - doc/src/naviserver/ns_register.man,
     doc/src/naviserver/ns_shortcut_filter.man: provide links (and
     hints) between "ns_shortcut_filter" and "ns_register_filter"

   - Rework man pages for scheduled procedures: group all
     related information into a single man page and improve
     cross referencing, fix broken example code.

   - doc/src/naviserver/ns_job.man:
     Fix wrong argument order in example command.
     
   - Made markup in documentation more consistent
	doc/src/naviserver/commandlist.man,
	doc/src/naviserver/ns_adp_ctl.man, doc/src/naviserver/ns_cond.man,
	doc/src/naviserver/ns_critsec.man, doc/src/naviserver/ns_env.man,
	doc/src/naviserver/ns_event.man, doc/src/naviserver/ns_info.man,
	doc/src/naviserver/ns_rwlock.man, doc/src/naviserver/ns_sema.man,
	doc/src/naviserver/ns_writer.man, nslog/doc/mann/ns_accesslog.man
	
   - Various improvements on several other man pages:
   - Removed man page for non-existing command "ns_cookietime"
     
	
Tcl API Changes:
----------------

  - Mark ns_cancel explicitly as deprecated, before its use was just
    discouraged by the man page. Use ns_unschedule_proc instead.


Configuration Changes:
----------------------

  - Turn off Nagle algorithm (parameter nodelay) per default
    since this leads in current Linux systems to bad performance
    (e.g. seeing with Apache bench ~40ms per request latency
    when keepalive is used). The default is now set to false
    in the C code as well in nsd-config.tcl and openacs-config.tcl

  - New parameter "logusecdiff": include time difference since last
    log entry with microsecond (usec) resolution. This option is useful
    for determine latencies with minimal effort.

  - All sample config files:
    * Remove file extensions for binary modules
    * Improved documentation

  - sample-config.tcl:
    * Added sample for DNS configuration
    * Added sample server log configuration
    
  - openacs-config.tcl:
    * Deactivated SSLv3 by default
    * Adjusted comments to values from source code
    * Add sorting to use always pick the library with the highest
      version number
	

Code Changes:
-------------

  - Regressions testing:
    * Removed false positive
    * Extended tests: http.test, ns_striphtml.test, ns_cache.test
    * Replaced deprecated "ns_adp_eval" in test by "ns_adp_parse"

    
  - Improved portability:
    * fixed compilation issues for versions of OpenSSL before 1.0.2
    * Windows changes (Many thanks to Andrew Piskorski):
      + Fixed compilation issues with nsssl under windows
      + Improved makefiles
      + use "I64u" or "I32u" instead of c99 conventions, since
        these seem to be broken in many (all?) MSC versions.
    
  - Reduced code security smells: add *CLOEXEC to file descriptor open
    operation. This fixes no real issue (forks are very limited in
    nsd), but silences static analyzers.

  - Replaced UTF-8 string literals by hexadecimal-escape-sequences to
    avoid potential encoding limitations from older compilers
	
  - Improved code locality
  - Removed unused macro
  - Adding more declarations for PURE and CONST functions
  - Eased live of static analysis programs
  - Added "const" declarations, made symbol table static
  - Reduced variable scopes
  - Reduced dead assignments
  - Reduce assignments in expressions


Modules:
--------

 - nsstats:
   * Added information about spooler threads to "process" page
   * Added running ns_jobs to the "process" page
   * Added running scheduled procedures to the "process" page
   * Improved HTML layout of configparams page

 - nsloopctl:
   * Fixed compilation with current versions