Forum OpenACS Development: Announcement: NaviServer 4.99.22 available
I am pleased to announce the availability of NaviServer 4.99.22 [1,2]. This release contains various fixes and enhancements. The short summary of changes is (see below for more details). The release was tested with with Ubuntu 20.04, Rocky Linux 8.4, OpenBSD 6.9 (clang), FreeBSD 13.0-CURRENT, macOS 11.5.2 (Intel and M1).
The following people have contributed to this release:
- Gustaf Neumann
- Ibrahim Tannir
- Oleg Oleinick
- Zoran Vasiljevic
The upgrade to 4.99.22 is recommended. The script install-ns is updated to point to the new version.
======================================= NaviServer 4.99.22, released 2021-08-25 ======================================= 93 files changed, 4216 insertions(+), 1465 deletions(-) New Features: ------------- - Added support for macOS machines with the M1 processor. - ns_http improvements: . Added proxy handling for [ns_http] . Added handling of charsets in ns_http get requests (for non-binary MIME types) - Extended default mime-types: . Added "jpeg/xl" to the default MIME types (specified in IANA provisional list) - Better handling with invalid UTF-8 (also relevant for security reasons) . new function "ns_valid_utf8" to check whether a byte-array contains a valid UTF-8 byte sequence . Check internally the validity of UTF-8 and complain in the system log when invalid characters are encountered. Note that Tcl transforms sometimes invalid UTF-8 into valid UTF-8 (e.g., URL /x%c3.) such that the exact analysis of potential attacks can become complex. . Proper encoded URLs contain just a subset of 7-bit US ASCII. The general problem of handling UTF-8 in request headers is actually quite complex and spread over several RFCs. The current HTTP specification (RFC 7230) states that "Parsing an HTTP message as a stream of Unicode characters, without regard for the specific encoding, creates security vulnerabilities due to the varying ways that string processing libraries handle invalid multibyte character sequences that contain the octet LF (%x0A)." ... older RFCs allowed explicitly ISO8859-1. However, all recent browsers code properly encode UTF-8 in the URLs. So, in theory, a server could reject uncoded "binary" data, and assume this data is coming from hacking attacks. However, this change just adds a warning for these cases. . UTF-8 validity checking is performed for URLs in percent-encoded and in non-encoded form - "ns_parseurl" reform: make URL conformant with RFC 3986 Previously, the parsing of URLs as performed by "ns_parseurl" was more driven by heuristics than by standards. The new version parses now URLs according to RFC 3986 (checking as well for valid characters in user-defined URL components when the new option "-strict" is used). Most internal usages of Ns_HttpParseHost() are non-strict to provide good backwards compatibility. The function now parses as well the userinfo in the authority. (authority = [userinfo "@"] host [":" port]). - Added command: "ns_parsehostport ?-strict? string" This command implements a subset of "ns_parseurl" by just trying to parse the provided string into "host" and "port" (when available). The command handles also the IP literal convention as specified in RFC 3986 for parsing IPv6 addresses with ports. - Added automated reloading of server certificates when SIGHUP is received. While in previous version it was necessary to restart the server, when certificates were renewed (e.g., via letsencrypt), the new version reloads certificates when it receives a SIGHUP signal. Bug Fixes: ---------- - Make sure to nul-terminate IPv4 portion in V4MAPPED addresses. - Fixed a potential race condition on peer and proxy IP address, where on e.g., pipelined requests the request structure is already reused in a new request, while the old connection is used for logging. This could result in incorrect peer addresses in the access.log - More precise execution of scheduled procs Background: previously, the scheduling of repeated scheduled procs was based on the last finish time. This has the consequence that the execution time will drift away more and more from a starting time, depending on the runtime of certain jobs. If one has e.g., a service that should run every minute, some of these minutes might be skipped by the cumulative drift on a long running server. New repeated scheduled procs on the original scheduled time rather than on last finish time. - Fixed Ns_StrTrimRight() to avoid damaging of UTF-8 characters Background: Up to now, NaviServer was using "CHARTYPE(space, c)" to determine, which characters can be trimmed at the end of a string. Unfortunately, there exists characters, which are classified as "space", but which can be trailing bytes with different semantics in multibyte UTF-8 characters (e.g., 0x85). When these bytes are stripped the result are invalid UTF-8 characters. - Added support for handling potentially negative time values when Ns_GetTime() is non-monotonous Background: The time of Ns_GetTime() is determined by gettimeofday() is therefore potentially affected by discontinuous jumps in the system time (e.g., if the system administrator manually changes the system time). - nsproxy: Disambiguate the name of the helper command with the name of the module NaviServer could get confused between the helper command "nsproxy" and the shared object of the module (called nsproxy.so). Since for module-loading, the suffix is optional (to support multi-platform config files), there as a potential confusion. The helper command is now called "nsproxy-helper". - Fixed list of charsets as returned by "ns_charsets". Previously, "ns_charsets" returned just the mapped charsets (where the name of the charset as defined in Tcl and by IANA differs). So e.g., "utf8" was not reported back. Now, the full is reported back. Additionally, more recent mappings were added. - Bugfix for "ns_http connect": Linux sometimes returned error (false positive) Sometimes "ns_http run SOMEHOST:PORT" returned under Linux an error of the form "can't connect to SOMEHOST port PORT: operation now in progress". This problem could have occurred on connections to hosts where the DNS entries have multiple IP addresses associated. This error only showed up on the first connection attempt (e.g., after a server restart, or on no connections to this host for e.g., a few hours), all later attempts with identical parameters worked without problems. It turned out that sometimes - while working through the associated IP addresses - the call "getsockopt(sock, SOL_SOCKET, SO_ERROR, ...)" retrieved errno 113 <No route to host> from the socket, maybe related with routing table lookup. - Improved log messages concerning limit of number of open files - Improved handling of running out of memory when creating threads from Tcl - Fixed potential race condition in logging during shutdown - Fix potential bug in openssl.m4 (could check for files on a wrong path) Documentation improvements: --------------------------- - Improved the following man pages doc/src/manual/admin-config.man doc/src/manual/admin-maintenance.man doc/src/manual/admin-tuning.man doc/src/manual/c-driverdb.man doc/src/manual/main-history.man doc/src/naviserver/commandlist.man doc/src/naviserver/ns_cache.man doc/src/naviserver/ns_conn.man doc/src/naviserver/ns_crypto.man doc/src/naviserver/ns_getcontent.man doc/src/naviserver/ns_http.man doc/src/naviserver/ns_locationproc.man doc/src/naviserver/ns_log.man doc/src/naviserver/ns_parseurl.man doc/src/naviserver/ns_schedule.man doc/src/naviserver/ns_sendmail.man doc/src/naviserver/ns_urlspace.man doc/src/naviserver/ns_write.man nsssl/doc/mann/nsssl.man - Added examples for "ns_getcontent" to the manual pages - Improved sample configuration files: . Added a section for the sample config files how to use the letsencrypt NaviServer module . Updated cipher configurations as recommended by Mozilla in sample configuration files. Code Changes: ------------- - Improved naming of functions - OpenSSL: aligned code with current snapshot of OpenSSL 3.0* (3.0.0-beta3-dev) - Aligned stubbed functions with Linux prototypes (use "restrict" keyword) - Extended regression test . Improved setup in tests for testing with private keys . Added testing for application/json with UTF-8 charset vis ns_http . Fixed handling of ns_hostbyaddr under macOS . Added 90 additional tests - Code Cleanup . Improved security by reducing usage of "ns_mktemp": use on the Tcl level "file tempfile ..." (introduced in Tcl 8.6) instead of "ns_mktemp" whenever possible. . Do not require to have tcllib package "try" installed when using Tcl 8.6 or newer . Use also in test cases reentrant version of localtime() . Reduced (harmless) data races . Fixed issued found by facebook infer 1.1.0 - Avoided passing NULL after the last typed argument to a variadic function - Added ability to pass "CFLAGS_OPTIMIZE=..." to "make" (eases build specific optimization) - Improved comments, fixed typos Changes in modules: nsdbsqlite: ChangeLog | 18 ------------------ nsdbsqlite.c | 12 ++++++------ nsdbpg: README | 66 ++++++++++++++++++++++++----------------------- dbpg.h | 10 ++++++++ nsdbpg.c | 90 +++++++++++++++++++++++++++++++++++++++++++--------------------- nsdbmysql: nsdbmysql.c | 4 ++-- nsocaml: nsocaml.c | 4 ++-- nssmtpd: nssmtpd.c | 16 ++++++++++------ nsdns: nsdns.c | 4 ++-- nsfortune: nsfortune.c | 4 ++-- nsicmp: nsicmp.c | 4 ++-- nsudp: nsudp.c | 4 ++-- nsaccess: nsaccess.c | 4 ++-- nschartdir: nschartdir.c | 2 +- nsexample: nsexample.c | 4 ++-- nszlib: ChangeLog | 8 -------- nszlib.c | 4 ++-- nsaspell: nsaspell.c | 4 ++-- nsimap: nsimap.c | 6 +++--- nstftpd: nstftpd.c | 4 ++-- nssyslogd: nssyslogd.c | 8 ++++---- nsphp: nsphp.c | 22 ++++++++++++++-------- nsstats: nsstats.tcl | 38 ++++++++++++++++++++------------------ nsauthpam: nsauthpam.c | 4 ++-- nsmemcache: nsmemcache.c | 4 ++-- nsvfs: ChangeLog | 4 ---- nsvfs.c | 4 ++-- nsdbi: doc/src/mann/nsdbi.man | 61 ++-- init.c | 122 ++++--- nsdbi.h | 12 +- tclcmds.c | 856 ++++++++++++++++++++++++++----------------------- nsloopctl: nsloopctl.c | 4 ++-- websocket: websocket-procs.tcl | 5 ++--- revproxy: README | 4 +- revproxy-procs.tcl | 128 ++++++++++++++++++++++++++++++++++++++++++++++------- letsencrypt: Makefile | 7 +- README | 38 +- letsencrypt-procs.tcl | 934 ++++++++++++++++++++++++++++++++++++++++++++++++++ letsencrypt.tcl | 879 +---------------------------------------------- nsldap: nsldap.c | 4 ++--