Forum OpenACS Development: ANNOUNCE: NaviServer 4.99.9 available

I am pleased to announce the availability of NaviServer 4.99.9 (see [1] and [2]). NaviServer 4.99.9 was tested with OpenACS 5.9.0. See below for the changes between NaviServer 4.99.8 and 4.99.9

all the best
-gustaf neumann
[1] https://sourceforge.net/projects/naviserver/
[2] https://bitbucket.org/naviserver/naviserver/

======================================
NaviServer 4.99.6, released 2016-01-11
======================================

Changes relative to 4.99.8
 146 files changed, 5595 insertions(+), 4389 deletions(-)

New Features:

  * New command "ns_urlspace": this command allows to attach and
    retrieve arbitrary data to URLs in a hierarchical manner form the
    Tcl level. The command allows e.g.  to write ns-perm like access
    control in Tcl without much hassle.

       ns_urlspace set ?-id i? ?-key k? ?-noinherit? /url/ string
       ns_urlspace get ?-exact? ?-id i? ?-key k? ?-noinherit? /url/
       ns_urlspace list
       ns_urlspace new
       ns_urlspace unset ?-id i? ?-key k? ?-noinherit? ?-recurse? /url/

  * Added ability to optionally minify css and js files on the fly
    when gzipped content is requested. This is an extension of the
    gzip_static/gzip_refresh feature (for details see "fastpath" section
    in "ns_return" man page.

  * New statistics from nsdb interface: "ns_db stats" returns
    number and time of handle and sql operations (can be used
    in connection with e.g. munin for monitoring db health)

  * Added ability to pass env to ns_proxy (for details see
    man page of ns_proxy)

  * Added ability to keep "Host:" header in http client requests
    (for details, see man page of ns_http)

  * Improved logging support:

    - Added log colorization based on ANSI color codes. This feature
      makes it easier to spot error and warning in the error log
      file. Log colorization is turned on via configuration parameter
      "logcolorize" in the "ns/parameters" section in the NaviServer
      config file. Fine-tuning is possible via "logprefixcolor",
      "logprefixintensity" and per severity via
      
          ns_logctl severity severity-level ?-color color? \
             ?-intensity intensity? ?bool?

      When viewing a colorized error.log with less, use "less -r".

    - Added ability to log selectively SQL statements depending on
      pool and duration of the SQL command (configuration parameter
      "logMinDuration" for per-pool configuration sections,
      new command "ns_db logminduration ...").
      
    - Provide context in error log entries, when Tcl eval raises
      an error (this allows better distinction from where the
      exception was raised)

    - Added TCP_FASTOPEN support for Mac OS X 10.11

Bug Fixes:
  * the urlspace code now can be compiled with and without
    __URLSPACE_OPTIMIZE__. The performance difference is
    probably for most application little.
  * fix "floating-point" format for Ns_ObjvTime():
    1.05 was interpreted as 1.5 before
  * fix binary handling and content-type in http client
    requests (e.g. for -body in ns_http command)
  * fix bad interaction between chunked encoding and gzipped
    content (when closing gzip stream)
  * Added config option for disabling directorylisting
    (fastpath config option "directorylisting none")
  * fix quoted entries in access log, when quotes are contained
    in the content
  * Don't use TCP_CORK on UDP sockets
  * Ensure list nature of errorCode is preserved in proxy calls
  * Remove potential xss attack (mostly false positives) on error
    messages that are directly generated from user strings by NaviServer
  * Improved "signal lost" handling (timeout happens, but connPtr is
    provided). On some sites the previous approach was leading to a
    state stuck in a while in the error state.
  * Fixed off-by-one error in ns_server.  NsTclServerObjCmd was
    incorrectly checking argc after parsing options. As a result,
    "ns_server -pool {}" (pool given, no subcommand) would segfault.


Documentation improvements:
  * fix obsolete documentation for ns_register_proc
  * new man page for "ns_urlspace"
  * improve documentation for main-features, ns_cache, ns_connchan,
    ns_db, ns_http, ns_job, ns_log, ns_normalizepath, ns_proxy,
    ns_register, ns_return, ns_sleep, ns_time, nsv,
  * include logo of generated html docs via css to ease Debian
    packaging (silence lintian)


Tcl API Changes:
  * improve compatibility of ns_eval with Tcl's eval.
  * new API functions
    - ns_urlspace set|get|list|new|unset ...
    - ns_logctl severity severity-level ?-color color? ?-intensity intensity? ?bool?
    - ns_db logminduration ...
    - ns_db stats
    - ns_proxy config ... -env ns-set ...
    - ns_http queue .... -keep_host_header
    
  
C API Changes:
  * Added Ns_GetTimeFromString: provide API callable from config file to
    convert strings to ns_time (supporting "sec:usec" and "sec.fraction"
    formats)
  * Mark broken and unused function Ns_UrlSpecificGetFast() as deprecated
  * Added Ns_(No)CloseOnExec implementation for Win32
  * Added Ns_SetIUpdate()
  * Ns_Cancel(), Ns_Pause(), Ns_Resume() return "bool" instead of "int"


Incompatible API Changes:
  * Potential incompatibility: "ns_eval" behaves now closer to
    Tcl "eval" (e.g. allows to define procs with comments)


Configuration Changes:
  * Improved sample configuration for OpenACS, sample-config
    and nsssl
  * Added cppcheck target to Makefile.module


Code Changes:
  * Updated list of known HTTP status codes  
  * Code cleanup:
    - use real boolean type when C99 is available
      (necessary for modules implemented in C++)
    - add const declarations
    - reduce implicit signed/unsigned conversions
    - reduce variable scopes
    - make code more uniform
    - better handling of nonnull asserts
      (don't raise warnings for these cases
       for most-recent gcc6 versions, when -pedantic is used)
    - prefer single returns from functions when possible
    - use NaviServer argv passing Ns_ParseObjv() on more
      occasions
    - replaced obsolete function 'asctime' by strftime
    - added fallback for definition of L_tmpnam, since the fallback
      from Tcl was removed on 2015-07-15
    - various changes for further silencing of static checkers
      (cppcheck and clang static checker)
 * Extended regression tests


Important changes in modules:
 * nsdbpg: fixes for "ns_pg blob*"
 * nsstats: added db-pool statistics, improved formatting
 * nsssl:
   - improved binary handling for "-body" (client procs),
   - ability to keep "Host:" header field
 * nsdbpg, nssmtpd, nsdns, nssnmp, nsudp, nsphp,nsdhcpd, nsssl, nsdbi:
   - code cleanup
 * changes in modules:  
    13 files changed, 116 insertions(+), 856 deletions(-)

Hi Steffen,

Nice to hear from you! The SQL logging feature could be done as well via pg logging, but it was always some effort to associate the entries of the pg-log with entries in the error log (if there are such entries) or with the causing query. With this change, that is not an issue anymore.

The db pool stats are particularly useful when monitoring multiple installations with varying loads (e.g. via munin). it can happen e.g. that a configuration which is running 90% perfectly might run out of db-handles, or might see a spike in SQL response times (see e.g. below). ...

One other interesting case for logging and db-stats is to get feedback, which queries run in which pool, and whether one has enough handles per pool, etc. The actual version of the nsstats modules reports the db-stats as well.

The actual driving force for ns_urlspace stuff was user tracking on certain pages (with blacklisting). One nice feature of the url-space is to associate some data to certain path ("folder") that can be inherited by all nodes under that "folder". One obvious use-case is access control: it is now possible to implement something like the nsperm module now in Tcl. Another interesting area is templating, where one can assign master templates to nodes in the tree (like "subsites") which are inherited to all the pages below this point.

Since we try to stay close the the released versions of the code, we are just starting to use these features. For obvious reasons, ns_urlspace is not used in OpenACS yet. However, in our own packages, we have several places, where urls are checked with e.g. [string match ...], these are obvious places where ns_urlspace offers a cleaner solution.

-g