Forum OpenACS Development: Announcement: NaviServer 4.99.19 available
Dear all,
I am pleased to announce the availability of NaviServer 4.99.19 [1,2]. This release contains the fixes and enhancements of the work of the last year. Many thanks to all contributors!
The upgrade to 4.99.19 is recommended. Below is a summary of changes.
all the best, and a happy new year!
-gustaf neumann
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.19/
[2] https://bitbucket.org/naviserver/naviserver/
=======================================
NaviServer 4.99.19, released 2020-01-06
=======================================
247 files changed, 19527 insertions(+), 8275 deletions(-)
New Features:
-------------
- Bandwidth management: limit download rate to a maximum (specified
in kilobytes per second)
On servers with bad internet connections, it is possible that a few
eager downloaders can use up all available bandwidth. NaviServer
allows now admins and developers to specify the maximum outgoing
bandwidth per connection or the total rate per connection pool. The
rate limit can be specified per connection (highest weight) or per
connection thread pool (second highest weight) or per network
driver. The rate limitation is only active, when writer threads are
used. Per default, the rate limitation is turned off (value 0).
By using this option together with the context filters, one can now
define e.g. a pool for bots and limit the bandwidth for all bots.
The query the per-connection limit, ns_conn has a new option:
ns_conn ratelimit ?limit?
Introspecton "ns_writer list" returns the current
transfer rates per writer job.
Configuration option for connection pools:
ns_param connectionratelimit ...
ns_param poolratelimit ...
Configuration option for network drivers:
ns_param writerratelimit ...
Commands for changing the rate limits dynamically:
ns_server ?-server s? ?-pool p? connectionratelimit ?value?
ns_server ?-server s? ?-pool p? poolratelimit ?value?
- Context filter for urlspace:
NaviServer's urlspace trie data structure used for various purposes
such as registering methods/URL pairs for procs or for mapping of
requests to connection thread pools. The classical trie mapping for
e.g. "GET /foo/bar/*.html" is based on method + path + string match
pattern, where the last part "*.html" is used as a "filter".
The new version of NaviServer allows in addition to the string
based glob pattern so-called "context filter", which includes the
IP-address or header fields in the matching process. The context
filters are applied after the classical urlspace processing, such
that the behavior is fully backwards compatible. The context
filters are kept in a stable order (IP before header-based filters,
specific before less specific) such that the behavior is not order
dependent.
The new feature can be used map e.g. bots or certain IP ranges to
the specified connection pools. When connection pools named "bots"
and "local" are defined, one can use the following mapping commands
to map incoming requests to the specified pools. Examples for
adding urlspace mappings at runtime:
ns_server -pool bots map "GET /* {user-agent *bot*}"
ns_server -pool bots map "GET /* {user-agent *crawl*}"
ns_server -pool bots map "GET /* {user-agent *baidu*}"
ns_server -pool bots map "GET /* {X-NS-ip 2a03:2880::/29}"
ns_server -pool local map "GET /* {X-NS-ip 127.0.0.1}"
ns_server -pool local map "GET /* {X-NS-ip 137.208.1.0/16}"
Note that arbitrary header fields can be used for the mapping,
containing potentially some glob style match characters. The
special header field "X-NS-ip" denotes IP based filters, which can
be provided fully qualified or in CIDR notation (denoting
significant bits) for IPv4 and IPv6.
- Allow a single driver to listen on multiple IP addresses:
This feature greatly simplifies setups, where a single server is
listening on multiple IP addresses (e.g. IPv4 and
IPv6). Previously, it was necessary to define separate drivers for
these, which need different names but which are often
configured identically. Furthermore, when the address is omitted,
the server performs a lookup from the hostname to determine the IP
address. Previously, it used the first one, now it can work with
all returned addresses.
To use this feature, simply specify the configure values of
"address" as a Tcl list containing multiple IP addresses. The
change is fully backwards compatible, old configuration files will
continue to work.
- Added support for sending of multiple file chunks in a
single connection:
ns_writer sendfiles /list of filespecs/
Every "filespec" is a dict which must contain a "filename" element
and can contain additionally an "-offset" and/or a "-size" element.
ns_writer submitfiles {{filename /tmp/f1} {filename /tmp/f2 -offset 10}}
This function is e.g. useful for video streaming applications,
where multiple video file chunks should be transferred (sometimes)
concatenated in a single chunk. Therefore, this function avoids the
necessity to concatenate these files in advance (causing increased
latency and disk usage).
- Added per-server "extraheaders". These extra headers fields are
merged with the following precedence (the earlier ones have more
weight)
1) application specific headers
2) per-server extra headers
3) per-driver extra headers
This change makes it possible to provide in the config files
defaults, which can be overwritten by certain pages in the
applications. Furthermore, one can e.g. common header fields to all
requests of a virtual server (e.g. Strict-Transport-Security, ...)
- Logging improvements:
* Support separate access logs for requests sent via
different drivers.
The nslog module accepts now an additional parameter named
"driver". This option can be used to produce different access
logs for requests submitted via different drivers to sort out
e.g. local server talk. Per default, every request is
logged. When a value is provided for "driver", only
requests are logged in this log file when these come from a
driver matching the provided glob pattern.
* log-file sanitizer:
When data entered into a log file contains special characters
(e.g. end line characters), these could alter the appearance of
content within the log file. Single entries may appear as
multiple entries. Attackers may leverage log forging to insert
fake entries to that obfuscate malicious acts, or these can
confuse log-file analyzers to avoid analysis.
The new global parameter "sanitizelogfiles" can control this
behavior (values 0: none, 1: full, 2: human-friendly; default:
2);
* Additional debugging flag "Debug(access)": When activated, entries
of the access log are mirrored in the system log
(error.log). This option eases tracking the end of requests in
the system log.
* Additional debugging flag "Debug(writer)" for debugging of writer.
* Provided access to request AND reply header fields via
"extendedheaders" specification (in "ns_accesslog" and "nslog"
configuration section):
It is now optionally possible to specify the header fields in the
provided "extendedheaders" list with a prefix "request:" or
"response:" to denote request and reply header fields. In case,
no tag is provided, field names are interpreted as request header
fields (like before). The change is fully backwards compatible,
- ns_set:
* Added sucommand "ns_set imerge", a is the case insignificant
version of "ns_set merge". This option is useful especially for
working with header fields.
* Added subcommand "ns_set iupdate": this command is the case
insenstive counterpart of "ns_set update".
* Fixed "nsv_set a b" when neither a key nor the array exists.
- ns_http:
* Revamp of ns_http and related supportive code.
* Added chunked-encoding parser
* Tcl channel support for body (PUT, POST) and for result.
This makes it possible to use e.g. reflected Tcl channels
in "ns_http".
New Options for "ns_http run"
- ?-body_size size?
- ?-body_chan chan?
- ?-outputfile fn?
- ?-outputchan chan?
* "ns_http run" is now the preferred interface
(rather than "queue" + "wait")
* Deprecated output variables of "ns_http wait" (since everything
is included in the resulting dict)
- New commands:
* ns_asynclogfile:
Facility for thread-safe writing async
log files of various kinds using the AsyncWriterThread.
Usage example:
# ... at startup
set fd [ns_asynclogfile open /tmp/test.log]
# ... during run
ns_asynclogfile write $fd hello\n
# ... at shutdown
ns_asynclogfile close $fd
* ns_parsefieldvalue:
This function parses the provided field value (from an HTTP
request or reply header field) into its parts and returns these
in the form of a list of Tcl dicts or a single Tcl dict depending
on parameters. The syntax of the contents of these header fields
is specified in RFC 7230 section 3.2.6.
- New features for existing commands/subcommands
* Binary reform: Some NaviServer commands accepted previously
implicitly binary or non-binary input, some commands used
the flag "-binary" to denote the differences (e.g. ns_return).
For a more uniform and expectable behavior binary input the
explicit option "-binary" flag to the following commands:
ns_base64decode
ns_base64encode
ns_base64urldecode
ns_base64urlencode
ns_md5
ns_sha1
ns_uudecode
ns_uuencode
ns_crypto::aead::decrypt string (for "-key", "-aad", "-iv", "input")
ns_crypto::aead::encrypt string (for "-key", "-aad", "-iv", "input")
ns_crypto::eckey import ("-string")
ns_crypto::eckey sharedsecret ("pubkey")
ns_crypto::hmac add (for "message")
ns_crypto::hmac new (for "key")
ns_crypto::hmac string (for "key", "message")
ns_crypto::md add (for "message")
ns_crypto::md hkdf (for "-salt" "-secret" "-info")
ns_crypto::md string (for "message")
ns_crypto::md vapidsign (for "message")
ns_crypto::scrypt (for "-salt" "-secret")
Note that this option is not fully backward compatible. Early
versions of "ns_sha1" were expecting always non-binary input, some
new version assumed binary input. The new version is now more
compatible with AOLserver and older NaviServer versions.
* "ns_server threads" reports now how often a thread of this pool
was started. This statistic can help to identify
installations, where threads are started and stopped in a too
eager fashion.
* "ns_server.... stats": added "sendbodysize" and "replybodysize"
to the dict of the per-pool statistics.
* "ns_server ... connectionratelimit ?value?": Query or set
the default per-connection rate limit.
* "ns_server ... poolratelimit ?value?": Query or set
the pool connection rate limit.
* "ns_writer list" returns now the current transferrates per
writer job.
* "ns_writer submitfile -offset X -size Y" and "ns_writer size X"
accept now memory units, the option ?-driver driver? was added
to the "ns_writer" subcommands "size", and "streaming"
(defaulting to the current driver).
One can now write
ns_writer size 150KB
instead of
ns_writer size nssock [expr 150*1024]
Bug Fixes:
----------
- Blueprint serializer: moved serialized objects in the blueprint
after the namespace imports and ensemble recreators since
constructors can call functions depending on it.
- Include *xml* and *json* in non-binary MIME types. Note that
there is no exact definition, what MIME types are exactly.
- nslog:
* Abort in error situation with an error message instead of crashing
* Added server name in log entries to make it easier to distinguish
messages per-server during bootup
- Range requests:
* Fixed potential problem with too many non-contiguous byte ranges
* Don't silently ignore invalid syntax of range requests
- Improved handling of already closed connections (data delivery was
already delegated to writer threads, but driver still wants to
return data directly; this could happen in error situations)
- connchan:
* Fixed potential invalid reads in (error) cases, where
LogConnchanDebug is enabled and the callback is deleted during a
Tcl_Eval().
* Fixed potential problem, where call to Tcl_Eval() might clean
structures used in the callback handler
- Made sure, memory allocated by Tcl is returned by Tcl (important,
when NaviServer is compiled with -DSYSTEM_MALLOC)
- Fixed Tcl argument parsing of "ns_roll", "ns_fmttime",
"nsv_bucket" and "ns_critsec eval".
- Fixed bug in "nsv_set -default" overwriting pre-existing values
- Fixed potential crashes in (desperate) error situations, where the
connection was already closed.
- Fixed leaking tmp file, when writerstraming is activated and
streaming HTML output is used. This bug could appear, when
"writerstreaming" is turned on in the config file (default off).
See also: https://sourceforge.net/p/naviserver/mailman/naviserver-
devel/thread/dd508519-1262-bd91-288a-fa30a7634224%40digital-
concepts.com/#msg36794472
- Made table used for enumeration values for Ns_ObjvIndex static.
Background: This is a fix for a tricky Tcl_Obj sharing bug
triggered by the usage of C-level IndexObjs based on volatile
tables. Consider the following example:
proc foo {x} {
return [bar -value x ...]
}
The Tcl_Obj "x" is shared as name of argument and as a value of
the non-positional parameter "-value". When bar
(e.g. C-implemented) uses Tcl_GetIndexFromObj*() to lookup "x" in
a table of options, which is volatile, the involved Tcl_Obj will
be converted to an indexObj.
On a call with wrong number of arguments( e.g."foo 1 2 3"), Tcl
will try to give a nice error message, saying that "foo x" can be
called only with one argument. When printing argument "x", it sees
that "x" is an indexObj, and for these kinds of objects, "x" might
be an abbreviated version of a full name. Since the table behind
the indexObj is in the case above volatile, a crash might happen.
- Added API call Ns_SockInErrorState() since SSL_shutdown() must not
be called if a previous fatal error has occurred on a connection
i.e. if SSL_get_error() has returned SSL_ERROR_SYSCALL or
SSL_ERROR_SSL.
- Provided local buffers for OpenSSL ERR_error_string handling to
avoid potential race conditions (OpenSSL keeps otherwise error
messages in static memory).
- Crypto support:
* New feature: scrypt Password-Based Key Derivation Function (RFC 7914)
The scrypt function is a modern replacement for crypt and bcrypt
and derives secret keys from a secret string. It is based on
memory- hard functions, which offer added protection against
attacks using custom hardware and GPU arrays.
The function requires the compilation of NaviServer against
OpenSSL 3.0 or newer (not yet released)
Example from RFC 7914:
% ::ns_crypto::scrypt -secret "password" -salt NaCl -n 1024 -r 8 -p 16
fdbabe1c9d3472007856e7190d01e9fe7c6ad7cbc8237830e77376634b373162
2eaf30d92e22a3886ff109279d9830dac727afb94a83ee6d8360cbdfa2cc0640
* "ns_md" and "ns_hmac": added flag "-encoding"
* Added options "-passphrase" to all "ns_crypto" commands
accepting a PEM file. In principle, every PEM file can be
protected by a pass phrase.
* Signing and verifying signatures: "::ns_crypto::md string" can
be used for signing and verifying of cryptgraphic signatures:
% set sig [::ns_crypto::md string \
-digest sha1 \
-encoding binary \
-sign /usr/local/src/naviserver/myprivate.pem \
"abcdefghijklmnopqrstuxvwxyz\n"]
% set vfy [::ns_crypto::md string \
-digest sha1 \
-verify /usr/local/src/naviserver/myprivate.pem \
-signature $sig \
"abcdefghijklmnopqrstuxvwxyz\n"]
Documentation improvements:
---------------------------
- Improved cross references between man pages
- Added documentation to undocumented flags
- Added more example
- Improved spelling
- The following 48 man pages were updated since the last release:
admin-db.man, admin-install.man, admin-maintenance.man,
admin-tuning.man, commandlist.man, ns_adp.man, ns_adp_ctl.man,
ns_adp_info.man, ns_adp_parse.man, ns_asynclogfile.man,
ns_base64.man, ns_conn.man, ns_connchan.man, ns_crypt.man,
ns_crypto.man, ns_driver.man, ns_filestat.man, ns_hmac.man,
ns_hotp.man, ns_http.man, ns_job.man, ns_locationproc.man,
ns_log.man, ns_md.man, ns_md5.man, ns_parsefieldvalue.man,
ns_parseheader.man, ns_parseurl.man, ns_rand.man,
ns_register.man, ns_return.man, ns_server.man, ns_set.man,
ns_sha1.man, ns_sls.man, ns_sockcallback.man,
ns_sockselect.man, ns_tmpnam.man, ns_totp.man,
ns_urlspace.man, ns_uudecode.man, ns_uuencode.man,
ns_writer.man, nsv.man, returnstatus-cmds.man,
tcl-lib-file.man, tcl-libraries.man, tcl-overview.man
Configuration Changes:
----------------------
- Output warning to system log, when NaviServer was compiled without
zlib support, but the config file request compression. Before, the
compress enable request was silently ignored, giving no hint, why
compression attempts of a developer were ignored.
- Sample configurations:
* Documented usage of connection pools in sample config files.
* Added sample entries for new features (e.g. rate limits, log
file sanitizer)
* openacs-config.tcl:
. Changed driver installation to "global" in sample configuration
file. This change makes it easier for site admins to add
further servers (e.g. for virtual hosting).
. Simplified sample config file by using multiple IP addresses
per driver
* nsd-config.tcl:
. Changed driver installation to "global" in sample configuration
file. This change makes it easier for site admins to add
Code Changes:
-------------
- Improved scalability: provided different mutex locks variables per
urlspace IDs.
- Improved range checking for Ns_ObjvInt, Ns_ObjvLong, Ns_ObjvWideInt
and MemUnits:
* built-in range checkingsupport for objv parser: This eases
specification of the C API, makes accepted ranges clear and
provides consistent error messages. These change fixes several
potential crashes, when e.g. passed-in values are used as
array-indices (crash on negative numbers) or where the values
were casted to unsigned types (very high unsigned values,
potential memory alloc problems).
* Fixed all cases, where Tcl_GetIntFromObj(), Tcl_GetLongFromObj,
and Tcl_GetWideintFromObj was used without a range check.
- Argument parser: skip processing of optional arguments when the
number of provided arguments is equal to the number of required
parameters.
Background: This change allows a fully backward compatible version
of e.g. "ns_md5 $foo", no matter what the value of "$foo" is,
although the new version of "ns_md5" has now a "-binary" option.
- Refactored low-level socket I/O:
* All driver operations are now strictly non-blocking
* Use same infrastructure for all network I/O operations
(driver, connchan, ns_http, ...)
* Rewritten socketfile/sendfile handling (including *BSD support)
* Improved handling of closed and detached channels, e.g.delegated
to writers (new macros NS_CONN_REQUIRE_CONNECTED,
NS_CONN_REQUIRE_OPEN, NS_CONN_REQUIRE_CONFIGURED,
NS_CONN_REQUIRE_ALL)
* Improved protection for SIGPIPE signals.
- New API infrastructure: Ns_DList
The Ns_DList is similar to Tcl_DString, but operates on pointers
instead of characters. Like Tcl_DString, Ns_DList pre-allocates
some data and extends it when necessary. Ns_DList structures are
especially useful alternative to linked lists, when usually data
is added to the end. Since the data is kept as a dense
(potentially growing) array, the memory locality is much better
than with linked lists, resulting in better CPU cache hit rates.
- Distinguish between Ns_NormalizePath() and Ns_NormalizeUrl(),
where the first is for the file system and the latter for URLs.
Previously, both cases were handled identically.
- Function pointer handling:
* Created a Tcl hash type for function pointers, since ANSI/ISO C
forbids casting/comparison of data and function pointers. The
new type is used whenever a hash lookup is performed for a
function pointer.
* Fixed all other occurrences of such comparisons.
- Regressions testing:
* Removed false positive
* Switch to testing via ns_http in regression tests
(this makes it as well possible for testing https connections)
* Extended tests:
+ Added regression tests for "ns_rand"
+ Added nsssl tests
+ Added automatic certificate generation for test-server
* The following 35 tests were added or extended since the last release
adp.test, encoding.test, http.test, http_byteranges.test,
http_chunked.test, http_keep.test, https.test, misc.test,
ns_adp_compress.test, ns_base64.test, ns_cache.test,
ns_conn.test, ns_conn_host.test, ns_crypto.test,
ns_driver.test, ns_hashpath.test, ns_hostbyaddr.test,
ns_info.test, ns_limits.test, ns_md5.test, ns_nsv.test,
ns_pagepath.test, ns_parsefieldvalue.test, ns_proxy.test,
ns_reflow_text.test, ns_schedule.test, ns_server.test,
ns_serverpath.test, ns_set.test, ns_sha1.test,
ns_urlencode.test, ns_urlspace.test, ns_uuencode.test,
ns_writer.test, tclconnio.test, tclresp.test
- Improved portability:
* improved compatibility with LibreSSL (2.7 and 2.9)
* Windows changes (Many thanks to Andrew Piskorski):
+ Fixed Ns_LogRoll() to work on Windows.
+ WSASend receives as 5th argument flags, but not a pointer to flags.
+ Improved makefiles
+ Updated _MSC_VER version numbers to include Visual Studio 2019
+ Added macro NS_INLINE to achieve higher portability with (older?)
Microsoft compilers.
- Added warnings about potential misconfiguration of NaviServer "tcllib"
setting (Tcl written NaviServer modules)
- Marked compatibility wrappers Ns_SetThreadServer() and
Ns_SetThreadServer() explicitly as deprecated. Use
Ns_ThreadSetName() and Ns_ThreadGetName() instead.
- Build-system
* Added -DSYSTEM_MALLOC to default compile flags.
* Added nsssl to the default test target.
* Require RSA keys of size 2048: newer versions of OpenSSL refuse to work
with RSA keys of size 1024.
* Improved alignment with Tcl's current .m4 file to get rid of
"-prebind" deprecated message under macOS.
* Improved handling of posix thread library for FreeBSD and OpenBSD
- Use native thread_local storage for log handling when available
(experimental).
- Improved type cleanness for function pointers
- Improved code locality
- Improved error messages
- Improved structure packing
- Reduced variable scopes
- Added missing "extern" declarations
- Aligned function prototypes
- Dropped potentially dangerous call to alloca()
- Added typedefs for commonly used functions
- Reduce potential dangling pointer dereferences
- Aligned names of arguments in prototype with function definition
- Introduced use attribute-based approach for denoting
fall through in case statements
- Added more declarations for PURE and CONST functions
- Improved configurability for clang-tidy
- Fixed macro name-clash with PostgreSQL
- Improved spelling
Modules:
--------
24 files changed, 1638 insertions(+), 2395 deletions(-)
nsdbpg:
- Fixed loading when multiple servers are used
- C Code cleanup (reduced warnings)
nsdbmysql:
- Fixed compatibility with current versions of NaviServer
- C Code cleanup (reduced warnings, improved spelling)
nssmtpd:
- Removed deprecated calls
- fix compilation with TCL_NO_DEPRECATED
- C Code cleanup (reduced implicit conversions, improved spelling)
nsdns:
- C Code cleanup (reduced warnings)
nsudp:
- Made code compilable with TCL_NO_DEPRECATED
- C Code cleanup (reduced warnings, improved spelling)
nszlib:
- Made code compile cleanly
nsimap:
- C Code cleanup (improved spelling)
nsphp:
- Upgraded module to PHP 7
- Fixed memory leaks
- extended regression test
- C Code cleanup (improved spelling)
- This is a major overhaul of nsphp, which makes it possible to use
PHP 7 (tested with PHP 7.3.5). PHP 5 reached it EOL by jan 2019. PHP
7 has substantial changes e.g. in memory and thread management
compared to PHP 5, several interfaces and API calls have changed,
such that it is not possible to compile the new version of nsphp
with still PHP 5. In case, someone wants to use nsphp with PHP 5,
please checkout versions before this commit from the repository.
nsstats:
- Added link to process page from mecurial hash to version
details on Bitbucket
- Included "ns_driver info" in the reported values
- Include number of started thread per pools in reported statistics
nsdbi:
- C Code cleanup (don't shadow variables, clean compilation, improved spelling)
nsdbipg:
- C Code cleanup (clean compilation, improved spelling)
nsoracle:
- C Code cleanup (clean compilation)
nswebsocket:
- Compatibility with Tcl 8.7, which has no "identity" encoding anymore
- fixed startup messages
- C Code cleanup (improved spelling)
revproxy:
- Support for request cancellation via url_rewrite_callback (when it
returns an empty URL)
- Don't raise error on ECONNRESET during spooling
- Improved logging
- Improved documentation
- Made channelCleanup more robust
letsencrypt:
- Major overhaul of the letsencrypt module to support ACME v2
ACME (the Automated Certificate Management Environment, [1]) is the
protocol used for certificate mamagement on letsencrypt.org. The API
version v was released on 2016 but was updated in 2018 by ACME v2,
is not backwards compatible with v1. Letsencrypt announced in march
2018 to drop the support of ACME v1 in several steps:
- Nov 2019: End of account registrations via ACME v1
- Jun 2020: End of new domain registrations via ACME v1
- Jun 2021: EOL ACME v1 certificate issuing
The new version is based on its crypto requirements solely on
OpenSSL, it uses the NaviServer builtins and as well the "openssl"
binary (the usage of tcllib pki was dropped). To avoid potential
troubles, use this with a recent version of NaviServer (currently
the tip version form BitBucket) or with NaviServer 4.99.19 when this
is released.
- Improved logging and documentation
- Add documentation for obtaining multi-domain attributes (SAN).
nswebpush:
- Added parameter "dopadding" to improve compatibility with Firefox on
Android. It does not support padding. Many thanks to Wolfgang
Winkler for reporting.
- Added compatibility with current version of NaviServer
- Always use binary format for appending (it does not seem necessary
here, but it is used for consistency)
- Code cleanup (improved spelling)
nsladp:
- Adapted changes from Malte Sussdorf for the AOLserver version
(from before 2008)
https://www.mail- archive.com/aolserver@listserv.aol.com/msg12073.html
- Added compile flag to force LDAP v3
- Rebind with original credentials after bind auth try
- Incorporated changes from Andreas Parschalk (Univ. Innsbruck)
including better documentation
- Provided clean compilation on recent versions of NaviServer and Tcl
