Forum OpenACS Development: Announcement: NaviServer 4.99.20 available
Dear all,
I am pleased to announce the availability of NaviServer 4.99.20 [1,2]. This release contains the fixes and enhancements of the work of the last year. The short summary of changes is (see below for more details):
- Support for time units (specify time units in minutes, seconds, milliseconds...)
- Subsecond granularity for all NaviServer commands
- HTTP client request logging
- Added support for "100 CONTINUE"
- Improved server liveliness (management of high loads and request queue overruns)
- Support for running behind a reverse proxy server
- Driver support for listening on multiple ports
- Improved HTTPS support (OCSP Stapling, server-side SNI)
- Improved support for client-side HTTP requests via ns_http (timeout handling, binary handling, log file for outgoing requests)
- Improved logging for detecting long running background jobs
- Dict support for nsvs
- More complete set of sub-commands for "ns_set"
- Improved form-processing (especially for REST processing)
- C-level WebSocket support
- Improved server statistics
- Improved scalability (reduced lock contention, higher usage of rwlocks, significantly faster nsv_*appends, ...)
- Improved portability: many improvements for windows, compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD 6.8 (LibreSSL 3.1.1), FreeBSD 12.2
- Andrew Piskorski
- David Osborne
- Gustaf Neumann
- Hector Romojaro
- Maksym Zinchenko
- Oleg Oleinick
- Zoran Vasiljevic
The upgrade to 4.99.20 is recommended. Below is the more detailed summary of changes.
All the best in the new year!
-gustaf neumann
[1] https://sourceforge.net/projects/naviserver/files/naviserver/4.99.20/
[2] https://bitbucket.org/naviserver/naviserver/
=======================================
NaviServer 4.99.20, released 2021-01-17
=======================================
380 files changed, 27639 insertions(+), 12103 deletions(-)
New Features:
-------------
- Ns_Time reform
* Support for time units:
. arguments of functions accepting time durations can be provided
now with a time unit suffix (such as ms, s, m, d). These values
can be written e.g. as 1.5ms, 3h and the like.
. units can be used as configuration values wherever appropriate
. time values wherever appropriate in documentation
. commands using Ns_ObjvTime can be specified now with time units
. This change is fully backward compatible for all commands
and configuration options accepting seconds, which was normally the case.
. All sample configuration files have been updated with time units
. nsproxy: The only exception to the precious sentence is nsproxy,
where durations we specified in earlier releases as
milliseconds. This is a POTENTIAL INCOMPATIBILITY with previous
versions. These parameters are
* config parameters of the nsproxy module
"evaltimeout", "gettimeout", "idletimeout",
"recvtimeout", "sendtimeout", "waittimeout"
* ns_proxy get: changed "-timeout" parameter to time unit value
* ns_proxy wait: changed optional "timeout" parameter to time unit value
* Change granularity from seconds to Ns_Time (internally and in Tcl API)
. Added proper handling of negative values in Ns_Time (was
broken/not implemented)
. Upgraded scheduling machinery from seconds to Ns_Time values
. Extended Ns_Time arithmetic (Ns_DiffTime()) to handle positive
and negative Ns_Time values
. Extended regression test for ns_time significantly
. Ns_After(), Ns_ScheduleProc(), Ns_ScheduleProcEx(): change
argument from int secs to Ns_Time
. Marked Ns_ScheduleProc() as deprecated, Ns_ScheduleProcEx()
should be used instead.
. ns_schedule_proc, ns_after: support time specs with time units
(instead of only full seconds)
. Ns_TimeToMilliseconds(): new API call for converting positive
and negative Ns_Time values to ms
. Ns_DStringAppendTime(): format times in a more compact form (no
trailing zeros)
. Use consistently Ns_TimeToMilliseconds() to convert Ns_Time to ms
. Ns_ObjvTimeRange, Ns_CheckTimeRange(): support to specify time
ranges in argument handling
. Trim trailing zeros after comma point when printing Ns_Time,
represent as int when possible
. "ns_info scheduled" returns now timestamps in sub-second
resolution (with a second faction). By this, the runtime of the
scheduled procedure can be determined on the sub-second level.
(potential incompatibility)
. Added handling of "schedmaxelapsed" on the sub-second level.
- HTTP client request logging
NaviServer can now optionally write a logfile for outgoing HTTP
requests initiated via "ns_http". This is important for webservers
that are dependent on other web services for the healthiness and
performance. The logfile is similar to the access.log and suitable
for automated analysis. Previously, NaviServer was "blind" on that
data. Now it is easier to provide data about the performance of
external services (e.g. cloud services such as authentication, Google
Firebase Cloud Messaging, Office 365, ...)
To activate HTTP client request logging, a new config-file section
"ns/server/$server/httpclient" was created where logroll, logrollfmt
and the other usual logging parameter can be specified.
- New command line option for NaviServer: "-T"
When this option is used, NaviServer just checks the configuration
file (specified as usual by "-t") for syntax errors. This option
should be used on production sites to reduce the downtime of a
server in case of typos.
- Added support for "100 CONTINUE"
This feature is in HTTP since a very long time, but not widely
supported although useful (best support is currently in curl). When
the client sends "Expect: 100-continue", the server can check the
header fields and provide an early feedback, whether it is useful to
transmit the full body. This way, e.g. content which is too large can
be rejected quickly.
- Improved server liveliness under high load (improved and configurable
handling of connection queue overrun)
When NaviServer processes an HTTP request, it is received (head and
body) and associated with a connection thread pool. If the thread
pool has a thread free, the request is directly assigned to this
thread. Otherwise, the request is added to a queue of the connection
pool, waiting for a connection thread. Since the queue has a limited
size (limited by "maxconnections"), this queue might overrun. In this
situation, the received request is added to a waiting list. Here is,
where the behavior is now improved.
1) previously, as soon as there were entries in the waiting list, no
more fresh requests were accepted by the driver (just working on
waiting requests). While this behavior is sensible if only one
connection pool is available, it causes a blocking behavior in
cases, when a single connection pool overruns but others could
continue. Now, more fresh requests are still accepted in this
situation.
2) In addition, it is now possible to configure the overrun behavior
via two pool parameters:
a) pool parameter "rejectoverrun": when set to true, NaviServer sends on
queue overruns an error "503 service unavailable". The overflowing
request is not added to the waiting list.
b) when (a) is set, it is possible to specify parameter "retryafter"
with a time interval to add reply header field "Retry-After" to the
503 response. (see https://tools.ietf.org/html/rfc7231#section-7.1.3)
When "rejectoverrun" is false (default) then behavior is similar to
before, accept the server does not fully block. The request is kept in
the waiting list, which is checked, whenever a request finishes. For
the client, there is no need to resubmit the request, which will be
proceed eventually. Technically, this means an arbitrary large queue.
When "rejectoverrun" is true, then the request causing the overrun is
rejected, and will have to be resubmitted by the client (maybe
automatically, when "Retry-After" is specified. However, the client
support for "Retry-After" is rather bad.
For busy sites, "rejectoverrun" is probably the better behavior,
since no background queuing in the waiting list is performed. For
now, we keep the old behavior as default for better backwards
compatibility. The default might change in the future.
- General support for running NaviServer behind a reverse proxy server
This change provides general support for running NaviServer behind a
reverse proxy. Previously, the support was rather punctual (e.g. for
nslog). Reverse proxy mode is activated by setting the global
parameter (in section "ns/parameters") named "reverseproxymode" to
true (default false).
Current limitation: In the current implementation, NaviServer supports
only the "X-Forwarded-For" header field, which is supported by most
proxy servers, treating the first provided IP-address as the client
IP address (see https://en.wikipedia.org/wiki/X-Forwarded-For). The
newer "Forwarded" header field is currently not supported.
In the reverse proxy mode, the client address is taken as forwarded
by the reverse proxy. This has the following consequences:
* Reported IP address by the following commands.
ns_conn peeraddr
ns_server all|running
ns_writer list
ns_connchan list
* Reported IP address in the access.log
* Context filters (used e.g. for mapping requests to pools)
In reverse proxy mode, all IP-address specific context filters
refer to the x-forwarded-for addresses. This makes it possible to
use context filters (of URLspace commands) based on IP-addresses
(IPv4, IPv6, with masks) also behind reverse proxy
servers. Previously the IP-based mapping was just possible when no
reverse proxy was in use.
- Allow a single driver to listen on multiple ports:
This new feature simplifies setups, where a single server is
listening on multiple ports using the same driver. Previously, it was
necessary to define separate drivers for such cases, which need
different names but which are often configured identically. Together
with the change introduced in 4.99.19 (support for multiple listen
addresses) this means that when multiple addresses and multiple ports
are specified for a single driver, it will listen on the cross
product of all combinations. The maximum number of listening ports of
a driver is limited by a compile time constant to
MAX_LISTEN_ADDR_PER_DRIVER (per default 16).
To use this feature, specify in the configuration file for "port" in
the network driver section a space separated list of multiple
ports. This change is fully backwards compatible, old configuration
files will continue to work.
- SSL/TLS improvements:
* Support for OCSP Stapling
. NaviServer obtains and provides information about the validity
of the used certificates. For proving this information, the
server performs two level of caching: in-memory caching and
disk caching. When the server receives the first TLS request
with OCSP stapling turned on, it checks for an already
retrieved OCSP response. The disk cache file is saved in the
"log" directory of the server and uses the serial number of the
certificate to be checked as filename with ".der" as
extension. When the disk cache file does not exist, an
http/https request is made to the server issuing the servers
certificate as defined by the Authority Information Access
(AIA) Extension. The names of the file and the http/https
request for the OCSP response can be obtained from the system
log of the server
. OCSP Stapling can be turned on in the configuration files (see
sample configuration files)
. OCSP Stapling requires OpenSSL 1.1.0 or newer
* Support for server-side SNI (Server Name Indication (SNI)
Server-side SNI can be used to use different certificates for
virtual hosts. In addition to the top-level certificate, different
certificates can be defined per virtual server (below: srv2). The
certificate will be registered for all alternate names specified
for the server.
In addition to the certificate, also all other OpenSSL specific
parameters (ciphers, protocols, ...) will be loaded form the same
place as the certificate. The following example shows the setup for
two virtual servers "srv1" and "srv2", where the first one is the
default server.
ns_section ns/modules {
# ...
ns_param nsssl ${bindir}/nsssl.so
}
ns_section ns/module/nsssl {
# ...
ns_param defaultserver srv1
ns_param certificate /usr/local/ns/modules/nsssl/srv1.pem
}
ns_section "ns/module/nsssl/servers" {
ns_param srv1 test.org
ns_param srv1 www.test.org
ns_param srv2 someotherhost.org
}
ns_section "ns/server/srv2/module/nsssl" {
# ...
ns_param certificate /usr/local/ns/modules/nsssl/srv2.pem
ns_param ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
}
SNI support includes OCSP, DH ephemeral key exchange, reuse of SSLcontexts.
* Support for OpenSSL "ciphersuites" (TLSv1.3 feature)
* Support for client-side SNI: new parameter "hostname" for
"ns_connchan open"
* Support for removing TLSv1.0, TLSv1.1, TLSv1.2, and TLSv1.3 via
configuration file
- ns_http:
* Clear separation of "-timeout" and "-expire":
. "-timeout" is used for single communication acts (e.g. connect, read write)
. "-expire" is used for limiting the full duration of the
request (also useful for slow read/write attacks).
Scripts already using the -timeout will most probably not see
any change unless they are handling huge up/downloads. In such
cases we will now continue with the request, instead of
terminating it in the middle.
* added "-binary" flag to transmit data binary and perform no
automatic encoding transformations.
- moved code repository on BitBucket from mercurial to git.
* adjusted documentation
* adjusted build system (Makefiles)
- Logging improvements:
* New log severity "Debug(timeout)" to force NaviServer to write an entry to the
systemlog, whenever a timeout occurs. Therefore, the command
ns_logctl severity Debug(timeout) on
can be used to debug cases, where NaviServer runs into HTTP status code 503
(Service Unavailable).
* Logging of commands with high latency:
Provide a consistent interface and naming convention for logging
slow commands for nsproxy, ns_jobs, scheduled procedures and SQL
(nsdb drivers).
. nsproxy and nsjobs have a new configuration option "logminduration"
This configuration option specifies a time-limit for logging
(long) eval operations to the system log (similar to
"logminduration" in the database drivers). Set it to a high value
to avoid logging (e.g. 1d). The default is 1s.
. scheduled procedures:
Changed "schedmaxelapsed" to "schedlogminduration" for consistency.
This is a potential incompatibility for old configurations.
- New commands:
* "nsv_dict":
The new command "nsv_dict" provides thread-safe support for
setting, getting, updating, and unsetting dict values in nsv
array.
Motivation: Some larger OpenACS installations have currently more
than 1 mio (!) nsv arrays, which comes mostly from creating nsv
arrays per packages. Most of these arrays have only a few keys
with very short values. The number of arrays can be greatly
reduced by supporting complex values (dictionaries), similar to
REDIS hash values.
. "nsv_dict" supports nested dicts like in Tcl
. The following dict commands:
nsv_dict append array key dictkey dictvalue:0..n
nsv_dict exists array key dictkey:1..n
nsv_dict get array key dictkey:0..n
nsv_dict getdef array key dictkey:0..n default
nsv_dict incr array key dictkey ?increment?
nsv_dict keys array key ?pattern?
nsv_dict lappend array key dictkey dictvalue:0..n
nsv_dict set array key dictkey:1..n dictvalue
nsv_dict size array key
nsv_dict unset array key dictkey:1..n
. added 41 test cases in regression test for nsv_dict
* "ns_dbquotelist": similar to "ns_dbquotevalue", but provides
quoting for Tcl lists.
* "ns_trim": multiline trim command:
ns_trim ?-subst? ?-delimiter delimiter? text
- New features for existing commands/subcommands
* New subcommands "keys" and "values" for "ns_set":
"ns_set keys /setId/ ?pattern?"
"ns_set values /setId/ ?pattern?"
The commands are similar to the Tcl command "dict keys" and "dict
values" and can reduce the necessity for loops.
* Form processing:
. "ns_getformfile": return a list of filenames when "multiple"
attribute is provided in the input field and multiple files are
submitted
. Decouple content-type processing from HTTP methods:
Previously, parsing of form data in these two formats happened
only on POST requests. However, several REST services use often
also PUT with form data and distinguish POST and PUT methods
mostly by "add item to a collection" (POST) and process single
item (PUT) but not by their content-type processing. This change
eases implementation of REST services.
. Perform more strict parsing based on "ns_parsefieldvalue" and
"ns_parseheader" in file-based form parser (for spooled
content). Previously, there was a bug with the regexp-based
parsing, when e.g. a filename of and upload file contained
e.g. a semicolon.
* "ns_conn":
. new subcommand "headerlength"
. "ns_conn auth" returns now Information about "Bearer"
authentication (RFC 6750) when applicable, used e.g. in OAuth2
. "ns_conn peeraddr": New flag "-source" for specifying detailed
information about the peer, especially useful in combination
with reverse proxy mode; allowed values are "configured",
"direct", "forwarded".
* "ns_cookie": Support explicit setting of "same-site=none"
In previous version of browsers, no explicit setting
of the samesite flag was exactly the same as explicit setting.
Since some browsers switched to a default of "lax", explicit
setting became necessary.
* "ns_dbquotevalue": provided a C implementation; was scripted
before.
* "ns_connchan":
. Added option "-server" to "ns_connchan close" to be able
to cleaning stale handles of every server.
. Added C-level WebSocket support and buffered output handling.
The new code is about 200x faster than the Tcl version
(especially when bytewise XORs are needed), simpler to use (no
Tcl-leval handling of incomplete frames, split frames, remaining
leftover from partial writes) and more complete (handling of
segmented WebSocket messages).
. "ns_connchan read -websocket ....": returns a dict containing
the WebSocket data (including payload, WebSocket status bits,
buffering state, etc.; see documentation and NaviServer
websocket module for details)
. "ns_connchan write -buffered ..." handles partial writes via an
internal buffer (Tcl developer has just to take care about
writable callbacks when necessary)
. "ns_connchan wsencode ...": WebSocket frame generation, returns
binary data based on input parameters suitable for sending on a
WebSocket channel.
. "ns_connchan status ..." returns a dict containing the buffering
status of a channel.
* "ns_urelencode": added one more option "oauth1" for the "-part"
argument. The "oauth1" encoding is defined in RFC 5849 Section 3.6
for the construction of the signature base string and the
"Authorization" header field.
* "ns_ictl":
. "ns_ictl trace idle ...": new tracetype "idle", which fires,
when a connection thread is idle for the threadtimeout period,
but minthreads is already reached, and the thread won't
terminate.
. "ns_ictl maxconcurrentupdates ?max?": Configuration option
for specifying the maximum number of concurrent updates
for interpreter reloads (when epoch increases, e.g. reloads
on OpenACS).
Background: For large (say 50-100 connection threads) and busy
sites (sustained 800 requests per second) with large
blueprints (e.g. OpenACS) blueprint updates can cause delays
when all these operations are concurrently performed. The
blueprint updates happen when the epoch is incremented
(e.g. in OpenACS: reloads of packages). A too high number of
concurrent blueprint updates cases (a) delays in request
processing (b) increased update times per thread (e.g. up to
10 seconds). As a consequence, queueing will happen. This new
option limits the number of concurrent updates (e.g. to 10)
leading to smooth operations in such situations. When the
maximum number of concurrent blueprints is reached, the pending
updates are delayed until the limit is sufficiently low. By
default, the value is unlimited.
* Improved statistics:
. "ns_server stats" returns now as well dropped requests from queue
overruns.
. "ns_info locks": report number of read and write operations for
rwlocks.
. "ns_proxy stats": report also the number of running worker
processes.
* "ns_getcsv":
. provide compatibility with RFC 4180
. accept multi-line input
. handle quotes in quotes and empty elements
. optionally stripping unquoted elements (not covered by RFC)
. optional support for a different quote character (option
"-quotechar"). Although RFC 4180 specifies double quotes as
quoting chars, sometimes also different quoting characters,
which can be now successfully decoded.
* "ns_parseurl": Support for fragment parsing. When provided URL
contains a tail element, this is now parsed info a "query" and/or
"fragment" component in the result. Previously, the query and
fragment part was included in the "tail" element.
* Improved thread naming for log-files: previously, it was not clear,
from which server some output of the "main" thread originated.
The new version adds always a server name specific suffix.
* "ns_driver info": return also "port" and "defaultport"
Bug Fixes:
----------
- Improved support for newer versions of OpenSSL (handling of
SSL_R_UNEXPECTED_EOF_WHILE_READING)
- ns_cache: Base cache size calculation on the full size of a
cache entry.
In previous versions, the cache size calculation was based only on
the value size, leaving the size of the key and the size of the
overhead (size of cache entry, about 126 bytes on a 64-bit machine)
aside. This is misleading, when the cache value
size is 0 bytes (just using the cache key as a flag), leaving admins
wondering about the memory consumption.
- Report to log file, when ns_ictl traces are not executed. Previously,
these were silently ignored (e.g. in some error cases).
- Fixed a bug with PUT methods, when query-parameters were passed in
addition to the put content. Previously, these parameters were only
available via [ns_queryget ...] when the content file was not
spooled.
- Fixed bug for custom error pages for HTTP methods other than GET
(Custom error handles are defined as before via redirect sections):
* Always use GET in the redirection not the original HTTP method.
When e.g. "PUT /somepage" is requested, where no PUT handler
but a custom error page /405.tcl, the previous code tried a
PUT /405.tcl, leading the same error. Now, the error
page us delivered like with a GET request.
* Provided a hint as a warning, where NaviServer searches
for non-existing pages.
* Extended regression test: cover HEAD requests for existing
and non-existing pages, and various forms of redirects.
- Fixed issue with tDOM as noted by Maksym Zinchenko on the
naviserver developer list:
A recent change of tDOM 0.9.2 introduced a namespace local command
"namespace" which was picked up by the blueprint generator
incorrectly instead of the global ::namespace command. As a fix, all
"namespace" invocations in the blueprint generator were replaced by
"::namespace" to avoid considering invocation context for every
occurrence.
- Fixed potential abortion of regression test runs under macOS:
Under macOS, tests runs were sometimes abruptly ended with a
broken pipe signal. The call "NsBlockSignal(NS_SIGPIPE);" does
not avoid these signals. So, we ignore bluntly all such
signals by using "NsBlockSignal(NS_SIGPIPE);"
- Fixed charset handling for spooled requests.
- Fixed several cases of invalid chunked data (potential buffer
overflows).
- Don't swallow errors in braced config sections in the configuration file.
- Fixed a potential crash, when certain configuration section were missing,
but code tried to obtain configuration values from such sections.
- Blueprint generation:
* Fixed bug with interp aliases defined without fully qualified
namespace path
* Improve robustness when "rename" is used (when source exists and
target does not exist)
- Support calling "ns_conn status" (with the argument setting the HTTP
return status) in traces (without open connection).
- Make code more robust in case Tcl and NaviServer were compiled
with different memory allocators. The mix if memory allocators
could crash NaviServer.
- Race Conditions:
* Fixed several race conditions in "ns_http"
* Fixed race conditions in "ns_connchan list" and "ns_connchan create"
- Use time_t for secs in Ns_Time, since long is just 32-bit on 64-bit
windows, but time_t is 64 bits.
- "ns_writefp": fixed to act as advertised (read to end of file when no
length is provided)
- "ns_sendmail": Convert leading "." in message bodies to ".." as
required by RFC 5321 section 4.5.2
- "ns_cache" transactions: in certain situations, where during
concurrent entry creates, and one of the create attempts failed and
the other succeeded, and the one of the cache transitions is rolled
back, updates could lead to an error.
- "ns_connchan": Fixed registry of connchans from all servers to the
default server (not the best scalable solution, but useful as long
we have only one "socks" thread (for more details see:
https://sourceforge.net/p/naviserver/mailman/message/37179476/)
- Fixed a 20-year-old time bomb in HTTP time parsing. Server was rejecting
to parse HTTP time strings for 2021.
Performance improvements:
-------------------------
- nsv_lappend:
The old implementation showed a linear slowdown for nsv_lappend
operations depending on the number of elements in the list. The
new implementation avoids frequent string to list and vice versa
operations by using Tcl_DStrings. It is essentially independent on
the number of elements in the nsv. For a list with 10K elements,
the performance improvement is about a factor of 400.
Test:
nsv_set d 1 1
time {nsv_lappend d 1 1} 10000
OLD:
403.7255 microseconds per iteration
NEW:
0.860547 microseconds per iteration
This change makes it possible to use the OpenACS developer support
on busy sites (before, this led to a quick standstill).
- nsv_append:
The performance gain is much less than with nsv_lappend (see
previous commit), but still noticeable (about 35% faster, 10K
ns_append operations went from 0.94 microseconds to 0.62
- Locking:
* Reduced mutex-locked regions in driver.
* Use ptrhreads interface for Ns_RWLock* when available
So far, Ns_RWLock* used an implementation based on mutexes and
cond vars. This implementation has the advantage of being
portable. However, since on all *nix platforms, NaviServer uses
the pthread library, using the POSIX rwlock implementation poses
no additional dependency.
Performance comparison fur using mutex/old rwlocks/pthread rwlocks
on ns_urlspace;
macOS (8 cores) rwlock (POSIX) 0.3105 2,27 %
rwlock (ns) 11.5263 84,23 %
mutex (ns) 13.6839 100,00 %
Linux (6 cores) rwlock (POSIX) 0.6081 43,38 %
rwlock (ns) 3.6259 258,66 %
mutex (ns) 1.4018 100,00 %
This data shows that the POSIX rwlocks implementation
(pthread_rwlock_rdlock etc.) is always better than the classical
NaviServer rwlock implementation, and that significant
improvements can be achieved by using rwlocks. A more detailed
performance comparison was provided on the developer mailing list.
This change introduces:
a) Ns_RWLock* based on pthreads when available
b) switch to recommended configure setup for pthreads (AX_PTHREAD)
c) New API call: Ns_RWLockSetName2() modeled after Ns_MutexSetName2()
d) New API call: Ns_RWLockList() modeld after Ns_MutexList()
(removed dependency of rwlocks to mutex statistics)
* Support rwlocks for nsvs:
Per default, rwlocks are used for all nsv* commands. The default
behavior can be altered by setting the configuration variable
"nsvrwlocks" to false in the section ns/server/${server}/tcl of
the configuration file.
This change can lead to improved scalability especially on
multi-core machines since in read cases, the variables can be
accessd unblocked for parallel executions. In most applications nsv
are typically more often read than written rwlocks offer better
scalability for large sites.
The following table compares the busy locks with mutex operations vs
the number of busy locks with rwlocks for openacs.org:
With mutex (24h)
locks busy
nsv:3:openacs.org 27 4.71M 1.3K
nsv:6:openacs.org 24 4.88M 1.03K
nsv:2:openacs.org 28 3.37M 784
nsv:7:openacs.org 23 9.11M 755
nsv:5:openacs.org 25 2.88M 460
With rwlocks (24h)
nsv:7:openacs.org 1 7.22M 0
nsv:6:openacs.org 2 3.92M 143
nsv:3:openacs.org 5 3.31M 1
nsv:2:openacs.org 6 2.23M 16
nsv:5:openacs.org 3 2.16M 0
On more busy sites, the improved scalability si much higher.
* rwlocks are better in cases, where the number of readers is
significantly higher than the number of writers. So, it has to
be carefully considered in which cases rwlocks should be
preferred. The new version uses rwlocks for nsv, ns_connchan,
filters and tcl.cache (rwlocks for nsv and filters can be
deactivated via configuration file). Earlier version used
rwlocks already for adp tags, epoch handling and permissions.
- Avoid multiple library initializations for OpenSSL 1.1.0 or newer.
Documentation improvements:
---------------------------
- Update references to Tcl website (replace http://tcl.tk by
https://core.tcl-lang.org)
- Added hints about required OpenSSL version to man pages
- Made manual pages more consistent
- Consistent indentation of program examples in documentation
- Made source code documentation more consistent
- Improved source code documentation (provide more details, remove
obsolete comments)
- Removed deprecated man pages
- Removed obsolete sections in manual pages
- Follow the recommended spelling convention of the Linux documentation
project
- Improved spelling, typesetting, and cross references
- Reduced technical jargon
- Made code listings in documentation more consistent (e.g. indenting)
- Added missing commands in command list (overview page)
- Updated listing of deprecated commands and suggested alternatives
- Replaced several "slave process" by "child process"
- 161 manual pages were updated since the last release
Configuration Changes:
----------------------
- New configuration options:
* Section "ns/parameters"
. "reverseproxymode": Activate reverse proxy mode. The former
configuration parameter of "nslog" named "checkforproxy" is
obsolete now.
. "joblogminduration": report jobs taking longer in systemlog
. "schedlogminduration": report scheduled jobs taking longer in systemlog
* Section "ns/server/${server}"
. "rejectoverrun": when true send 503 when queue overruns;
default "false"
. "retryafter": time for Retry-After in 503 cases
* Section "ns/server/${server}"
. "filterrwlocks": use RWlocks for filters (default: true)
* Section "ns/server/${server}/tcl"
. "nsvrwlocks": use RWlocks for nsv (default: true)
* Section "ns/server/$server/httpclient"
. "logging": (default: off)
. "logfile": name of http client logfile
. "logrollfmt": format appended to log filename
. "logmaxbackup": max number of backup log files (default: 10)
. "logroll": perform automatic rolling (default: true)
. "logrollonsignal": (default: false)
. "logrollhour": specify at which hour to roll (default: 0)
- Sample configurations:
* Added values with time units where appropriate
* Added new values mentioned above with short comments
* For multi-server configuration files: use always the first (and
not the last) server as default server.
* openacs-config.tcl:
* sample-config.tcl:
. fixed config parameter: use "dnscachetimeout" instead of "keepwaittimeout"
(name was changed in NaviServer ages ago, but change was not
reflected in config files
* nsd-config.tcl:
* win32/test/nsd.tcl
modernized test config file
* nsproxy: Changed term "maxslaves" to "maxworkers" (the old term
is deprecated)
Code Changes:
-------------
- Fixed potential access to an out-of-scope stack variable.
- Added more declarations for PURE and CONST functions
- Replaced usage of reserved identifier [cert-dcl37-c,cert-dcl51-cpp]
- Replaced sscanf() by strtol*() to increase code-safety
- Improved structure packing
- Marked "ns_adp_mime" explicitly as deprecated, and provide deprecated
messages for it. The command was deprecated more than 10 years ago,
ns_adp_mimetype should be used
- Added mime types for "heif" and "heic" which have become a preferred
image type for iOS and macOS.
- Regressions testing:
* Reduced usages of old-style nstest::http-0.9 (socket and not
ns_http-based tests)
* Make test more robust concerning CRLF results (Windows)
* Extended tests:
367 tests were added since the last release
(changes in 38 test files)
- Improved logging:
* Added metering for long wait times of Ns_CsEnter locks
(e.g. Ns_MasterLock)
* Added concurrency level to log message of interp updates
* Added warnings, when an NsTclRequestProc or an ADP request runs
into a timeout.
* Reduced time sensitivity on time when rolling log files_ Rolling
log files happens often at midnight, using often a day
precision. When e.g. a scheduled procedure the time when this
function is called might be slightly after the scheduled time,
which might lead to a day jump. The problem aggravates, when
multiple log files are rotated. The new code identifies day wraps
and uses in such cases the earlier day.
- Improved portability:
* Improved compatibility with the forthcoming OpenSSL 3.0
* Improved compatibility with various versions of LibreSSL
* Improved Windows (MSVC) compatibility
. Simplified Makefile.win32 for nmake
. Fixed thread exit result handling for 64-bit Windows
. Fixed thread handle access
. Fixed default temp directory for windows
. Turned on OpenSSL by default
. Minimal support for running regression tests under windows
. Fixed windows size incompatibility for range requests (64-bit
Windows uses a 32-bit sized offset type off_t)
. Fixed ns_sockselect for 64-bit windows installations
. Fixes for cross-compiling for Windows 64-bit (using gcc/mingw)
* Use consequently NS_EAGAIN (can be used in *nix and Windows)
* Fixed potential problem with GNUC_PURE for older version of gcc
* Removed dependency of tests from an explicit "/tmp" directory
* Fixed regression test warnings with older versions of curl for IPv6 addresses
* Fixed m4 configuration for "crypt_r" (broken since many years)
* Compiled with macOS 10.14.6, Ubuntu 20.04, Fedora Core 32, OpenBSD
6.8 (LibreSSL 3.1.1), FreeBSD 12.2
- More OpenSSL changes:
* Reactivate CRYPTO_set_mem_functions() for OpenSSL 1.1.0 or newer
(interface om OpenSSL has changed)
* Switched to SSL_CTX_set_dh_auto() from low-level functions: In
newer versions of OpenSSL, all low-level DH* functions were
deprecated.
- Build-system
* Compute version tag from git similarly like to the mercurial version.
- improved support for MS windows
* Windows math.h does not include round(), add simple implementation.
- Added cross-platform function ns_localtime_r (localtime with
user-provided storage): The previously defined function
ns_localtime() uses a single per-thread local storage, so, it was not
possible to have in one thread two time buffers with different
values.
- Improved error code handling on driver receive operations
Previously, higher level of API calls were not able to access
realiably the error information detected on the lower-level (async
I/O-based or OpenSSL based) driver operations. This made it
impossible to provide good error messages e.g. in "ns_connchan" and
we could not provide error processing based on Tcl error
codes. Furthermore, the new interface works for POSIX errnos and
OpenSSL errors and improves the situation (currently just for driver
read operations).
* Ns_Sock* based setting of Tcl error codes:
NsSockSetRecvErrorCode(const Sock *sockPtr, Tcl_Interp *interp)
Set the Tcl error code in the interpreter based on the
information provided by Sock* (actually by recvErrno); the
function works for OpenSSL and plain POSIX style error codes.
* Setting Tcl error codes for POSIX errno:
Ns_PosixSetErrorCode(Tcl_Interp *interp, int errorNum)
Set the Tcl error code in the interpreter based on the POSIX
errorNum and return the error message.
* Setting Tcl error codes for OpenSSL error values
Ns_SSLSetErrorCode(Tcl_Interp *interp, unsigned long sslERRcode)
Set the Tcl error code in the interpreter based on the OpenSSL
error value and return the error message.
* Return the last error from a socket.
Ns_SockErrorCode(Tcl_Interp *interp, NS_SOCKET sock)
The function Ns_SockErrorCode() returns the last error from a
socket. When the provided interp is not NULL it sets the Tcl error
code based on the error code.
Modules:
--------
18 files changed, 1858 insertions(+), 1195 deletions(-)
nsdb:
* Added support for time units for nsdb LogMinDuration
nsdbpg:
* Force bindvars to start with alnum or underscore (otherwise,
surprises with e.g. lines ending with a colon might arise).
* Align spelling with LDP recommendations.
* Avoid shadowing of symbol "bind".
nssmtpd:
* Escape all leading dots according to RFC 5321 section 4.5.2 in mail bodies
* Fixed check, if relay host announced STARTTLS
* Provide compatibility for versions before 4.99.19, when NS_FALL_THROUGH was not defined yet
* Subsecond time resolution (Handled interface changes in post 4.99.19)
* Support for 'ciphersuites' and 'protocols' parameters of 'Ns_TLS_CtxServerCreate'
* I/O overhaul
- Replaced select() call by poll() to ensure scalability
- Pushed handling of EAGAIN etc. to the lowest level
- Removed one layer of partial write handling
- Replaced huge send buffer attempts, since these seem to be rejected sometimes
- Improved robustness (never go into an infinite retry loop)
- Handle potential "resource is not available" while sending SMTP data
* Improved cleanness of compilation
* Improved documentation
* Align spelling with LDP
* Code cleanup (C and Tcl code)
* Extended regression test
nsdns:
* Code Cleanup
- Added NS_EXPORT to Ns_ModuleInit
- Improved cleanness of compilation
nsudp:
* Minor code cleanup (type cleanness, documentation)
nsphp:
* Adjust to changed parameter types in PHP 7
* Improved cleanness of compilation
* Improved documentation
* Brought test environment in sync with naviserver
nsstats:
* Improved XSS protection for potentially user supplied strings
* Improved statistics:
- Added calculated maximum number of reqs/pool
- Added "maxconnections" to the per-pool statistics,
since it determines the queue length
- Improved nsproxy output in "process" page (include seconds,
don't truncate everything to 0 digits after the dot)
- Added write-percentage for RWLock in lock statistics
- Add statistics for dropped requests (queue overruns)
- Added nsv usage statistics
* Added web-interface for toggling between raw and human centric values
* Aligned with fractional seconds interface in NaviServer 4.99.20
* Improve help messages
nsoracle:
* Changed the include order to avoid bad interactions between oci.h and zlib.
with the previous include order, we run with recent oracle versions (19.5) into
/usr/include/zlib.h:1677:4: error: unknown type name ‘off64_t’
on 64-bit Linux systems (centos 7)
* Added ORA_CFLAGS to allow developers to pass arbitrary include and library flags.
Background: for some (newer?) version of the oracle client library,
ORACLE_HOME (see README) seems to be gone.
* Changes to build nsoracle on Windows with new Makefile.win32 file.
* Improved cleanness of compilation
websocket:
* Major rewrite, using C-level implementing in ns_connchan
This version make uses of the new C-level support for WebSockets.
The new version is several orders of magnitudes faster than the
previous version, it is more robust (C-level handling of
partial read and write operations) and it is more complete
than previous versions (handling segmented messages).
* Added deprecated function ::ws::build_msg for providing
better backward compatibility
* Provide name for mutex lock to ease tracing of long locks
* Added client-side SNI (requires NaviServer 4.99.20)
* Added simple WebSocket client interface
revproxy:
* Improved exception handling (use try interface)
* Preserve existing X-Forwarded-For IPs in cases where nsproxy is
itself behind a proxy or load balancer
* Switch to "ns_connchan write -buffered" to simplify leftover
management for partial writes (requires NaviServer 4.99.20)
* Fixed bug fix for potential stale handles
* Code cleanup
- Fine-tuning logging messages
- Breaking overlong lines, reduce verbosity
letsencrypt:
* Minor changes in documentation
nsldap:
* Fixed result setting of "connected" subcommand
* Updated modules interface to usual conventions
* Added config parameter "schema" to specify other schemata
* Improved documentation in README file
* Reduced number of deprecated calls
nsshell:
* Added HTML-escaping on results of Tcl commands before these are
displayed in the shell.
