Forum OpenACS Development: ANNOUNCE: NaviServer 4.99.8 available
I am pleased to announce the availability of NaviServer 4.99.8 from  and .
See below for the changes between NaviServer 4.99.6 and 4.99.8
====================================== NaviServer 4.99.8, released 2015-04-13 ====================================== Changes relative to 4.99.6 231 files changed, 21886 insertions(+), 13620 deletions(-) New Features: * Added Interface to NaviServer to read/write directly via driver routines. The new command "ns_connchan" provides a higher level alternative to the idiom "ns_conn channel" + Tcl-commands. The new command uses the internal socketcallback infrastructure and allows to - detach the currently open connection channel from a connection thread, to - read/write/close the channel from other threads, and to - query the states to then currently detached channels (similar to "ns_writer list") By accessing the driver infrastructure, "ns_connchan" makes it possible to read/write from/to SSL/TLS channels. The new command makes it easy e.g. to implement WebSockets over SSL/TLS (ws://... and wss://...) which is not possible with the pre-existing infrastructure. Summary of Advantages: * use the same interface for http and https (and other drivers) * no (potential) problems with select() in Tcl when using many (>1024) file descriptors * less potential problems with Tcl-Sockets (buffering, translations) * lower resource consumption Implemented interface: ns_connchan detach ns_connchan close channel ns_connchan list ?-server server? ns_connchan callback ?-timeout s? ?-receivetimeout s? ?-sendtimeout s? channel command when ns_connchan read channel ns_connchan write channel string * Improved ns_log handling - New subcommand [ns_log stats] returns the statistics from calls to ns_log by severity-level. This allows to write monitoring tools to provide rough overviews at which time at which server what kind of servers/warnings/... were produced - Allow to modify logging of sql statements via ns_logctl severity Debug(sql) on at runtime - Add debugging severity for tasks "Debug(task)" * Added ability to load a driver module without listening on a port: This is useful in cases, where the driver registers client commands (such as nssock or nsssl) and only these are wanted. Loading the driver without listening can be achieved by specifying "0" for the port. * Improved support for windows compilation with native compilers for 32 and 64 bit (Many thanks to Andreas Piskorski) * ns_md5, ns_sha1: added binary support When binary data is passed to this function, use Tcl byte-array operations instead of string operations * Added ability to debug configuration of nscgi via "ns_logctl severity Debug(cgi) on" * Added config parameter "compresspreinit": Setting this parameter to true (default is false) will cause the compression stream buffers to be allocated and initialized at server startup. Without it, they will be allocated and initialized as needed. This change drops the initial memory footprint of a server with a default config significantly. Performance Improvements: - Added more compiler hints likely/unlikely - Made Ns_Encrypt threadsafe under Linux (reduce mutex) - Use getpwnam_r, getpwuid_r, getgrnam_r, and getgrgid_r instead of their thread-unsafe counterparts. This change allow these functions to run without locking. - Reduced size of compressed requests in several situations (empty io_vecs) Bug Fixes: * Fixed encoding in [ns_conn content] * Fixed incorrect result of "ns_conn protocol" (returned empty) * Fixed crash when a thread was created with "ns_thread begindetached" and an error occurred during startup * Fixed memory management for TclThread by moving TclThreadArg into thread local storage * Fixed race condition in TLS cleanup * Fixed bug in argument parsing of "ns_cond" * Fixed setting of private/shared in ns_module * Fixed bug with "ns_adp_ctl stream on" * Moved network module filtering to the front to avoid double loading of Tcl modules (many thanks to William Scott Jordan for the fix) * ns_http: - Fixed trailing slashes in some urls - Fixed possible race condition when time gap between "ns_http queue" and "ns_http wait" is larger * Minor fixes in nscgi * Windows fixes - Fixed weird bugs when Tcl-functions are called during load of .dll files (showed up as hanging/crashing functions returning time from Tcl) - Fixed Ns_NormalizePath() for //foo/bar paths on Win32 (bug #1162890) - Use distinct bit-patterns for signal detection under windows. - Fixed array unset bug * Fixed a potential race condition in writer threads * [ns_perm delperm]: Fixed ignoring of argument "-noinherit" * Fixed reply headers in HEAD requests * Fixed bug reported by Wolfgang Winkler, when " ns_urldecode --" was called (switched to regular argv parser) * Fixed bug, when "ns_conn content" was called without content potential race conditions on thread exits * Fixed potential race conditions on thread exits * ns_md5: Code generation was broken (probably since a long time, due to a mix up of somewhat tricky casts) * Fixed warning in interaction between TCP_CORK and nsssl * Fixed bug, where one thread frees a nsv-array, but an internal representation of an Tcl_Obj for this array was still active in another thread * Fixed bug where ns_imgsize returned an error where it should return success and width and height returned as 0 (according to documentation) * Avoid potential access of string past null character * Fixed a bug with in https client commands (ns_ssl) when paths and parameters are passed. * Fixed nsphp compilation (and "make php"). Many thanks to Branden Graves for feedback and testing. Documentation improvements: * Improved typesetting, wording * Removed more obsolete content (e.g. concerning CVS, old dummy parameters) * Use examples from existing code basis * Improved comment density in source code * Documented missing subcommands, such as "ns_time format" * Added documentation for ns_md5 * Improved documentation for ns_img commands * Improved documentation for nscgi Tcl API Changes: * Marked "ns_geturl" as deprecated, use ns_http instead * Marked redundant names of subcommands in "ns_thread" as deprecated - ns_thread begin -> ns_thread create - ns_thread getid -> ns_thread id - ns_thread join -> ns_thread wait - ns_thread get -> ns_thread handle * Allow time interval of "-timeout" for "ns_db gethandle" to be specified with second factions (in the usual "ns_time" format) C API Changes: * [ns_conn content]: new flag "-binary" to return always the raw bytes * [ns_thread begin|begindetached]: added flag "-name" for naming thread explicit Configuration Changes: * Updated Makefiles as recommended in documentation * Improved sample configuration for OpenACS and nsssl * Improved Makefiles (reduce redundancy for CFLAGS) * Improved rpath handling in configure.ac for Linux distros, where TCL_CC_SEARCH_FLAGS and TCL_LD_SEARCH_FLAGS are set empty, like e.g.Debian Code Changes: * Many of the changes are the result of a (sometimes over-picky) code-checkers on unix and windows systems, in particular pc-lint, cppcheck and the clang static checkers. Many thanks to Maurizio Martignano for setting up the testing framework based on SonarQube and for eagerly pull in updates. * Function prototypes and API definition - add missing function prototypes - removed occurrences of old-style prototypes (when functions are defined without arguments) - removed unneeded external references and symbols and unreferenced enumeration values - aligned argument names in function prototypes with function definitions - added non-null/returns non-null assertions for prototypes - added asserts to check non-null properties - added "const" when appropriate - added missing "static" declarations * Improved and more consistent macro definitions - use INT2PTR and PTR2INT consistently when converting between integers and pointers - use NS_TRUE/NS_FALSE instead of 1/0 - use NS_INVALID_FD instead of -1 - use NS_SOCKET and NS_INVALID_SOCKET instead of int/-1 - use NS_INVALID_PID for invalid pids - change from usage of -1 or INVALID_SOCKET (windows predefined constant) to NS_INVALID_SOCKET due to confusions - use UNUSED consistently - character properties and conversions . new macro CHARTYPE for testing character properties . new macro CHARCONV for converting characters - added a new macros PTR2NSSOCK and NSSOCK2PTR in analogy to PTR2INT and INT2PTR to mask more win incompatibilities - prefer "const" over "CONST" unless in Tcl interface - improved consistency of variable and macro naming - put macro values into parenthesis * Improved general code quality - reduced variable scopes - remove cases of undefined C behavior - prefer leaving a switch statement via "break", and not via "return" - move assignments out of if-conditions - brace body of "if" statements and "for" loops - make ignoring of function results explicit - avoid shadowing of variable names - fix potentially uninitialized local and static variables * Used more precise and consistent types - prefer type "bool" over "int" when appropriate - use unsigned types for bit operations - prefer size_t, ssize_t, socklen_t, pid_t, nfds_t over int - use consistently uintptr_t for nextids - use enumerations instead of plain defines on some occasions - prefer boolean expression - reduce implicit conversions (signed/unsigned, signed constants, loss of precision, ...) - removed many cases of casts of complex expressions - reduce prohibited operations with plain (signed) chars - recast results of < - standardize type definitions of objv in ObjCmds to the variant recommended by Tcl man pages * Improved C-standard compliance - silence "-pedantic" compilation with -stc=c99 under linux on most cases - avoid c90 identifiers starting with str[a-z], to[a-z], E[A-Z], _* - avoid c90 reserved identifier "signal", "free", "remove", "time" - added missing printf macros * Improved usage of c-library functions and OS abstraction - use arc4random() when available - use crypt_r() when available - prefer memcpy() over strncpy(), since it has in most cases the appropriate semantics and it is faster - new/fixed OS abstractions functions/macros: ns_lseek, ns_dup, ns_dup2, use ns_open, ns_close, ns_lseek, ns_dup, ns_dup2, ns_write, ns_read, ns_mkstemp * Removed potential crashes from buffer overflows - use memory bounded functions instead of potentially buffer overflowing functions - replace unsafe atof() by strtod() - replace unsafe atoi() by strtol() - replace call to deprecated function asctime() by a call to asctime_s() - remove hard-coded sizes of ip-addresses and replace it by NS_IPADDR_SIZE - don't hardcode sizes for string arrays * Use of modern-style argument parser Ns_ParseObjv() on several places instead of manual argv parsing ns_job interface, ns_truncate, ns_ftruncate, ns_addrbyhost, ns_hostbyaddr * Make mutex timings configurable via #define NS_NO_MUTEX_TIMING * Improved code formatting - add emacs editor hints to to keep formatting conventions - prefer spaces over tabs * Extended Regression Test: - Added test set for ns_md5, compared results with other implementations - Added binary regression test for ns_md5 and ns_sha1 - Added test set for ns_md5 - Improved robustness of tests for ns_parseargs - Added tests for "ns_urldecode --" - Added test set infrastructure (nstest::https, server setup) and test cases for nsssl - Added test set for ns_img* commands - Number of regression tests passed 1000 Changes in Modules worth to be noted: * websocket: New module implementing a WebSocket interface. This is a Tcl-only module defined with two sample applications (chat and a file watcher for e.g error.log and access.log). It support ws:// (WebSocket over http) and wss:// (WebSocket over https). For details and configuration, see the README file of the module. * nszlib: Added commands: ns_zlib deflate /data/ ns_zlib inflate /data/ which are compatible with the same-named php functions. [ns_zlib deflate ...] returns the raw compressed binary data. These functions are e.g. needed e.g. when implementing as sasl interface for shibboleth. * nsssl: - Ability to load the module without listening to socket by specifying 0 as port. This is useful for e.g. just using the HTTPS client command ns_ssl. - Update configuration recommendations * nsoracle: Backport of the nsoracle driver from AOLserver to NaviServer with code cleanup.
I'd be interested in a few examples of how ns_connchan works, and the first thing I wondered was whether a "ns_connchan copy $fd" would be helpful to replace a lot of code to run background delivery of files?
With recent versions of NaviServer, most of the usages of bg-delivery can be replaced by "ns_writer submitfile ..." when writer-threads are in use (look for use_writerThread in ). The writer threads are roughly the equivalence of the bgdelivery thread, but implemented in C, not relying on select(), are lightweight in memory use and fast (using the common io-machinery of NaviServer). For the introduction to the async support in NaviServer, see section "2.3. Asynchronous Architecture" in .
In fact, when one is using a recent version of xotcl-core, ad_returnfile_background uses automatically the writer threads, no application code changes are needed. we use this since a while on our production systems an on openacs.org.
Actually, the only place were we are still using - and that might be the source of your interest - is the h264 streaming code. It should be actually not hard to replace there the tcl/io (e.g. replace "puts -nonewline $channel ..." by "ns_connchan write $channel ...", use "ns_connchan callback ...", etc.). This has as well the advantage that it will work without changes with tls. When one wants to copy the fd from a channel, this would defeat this purpose. So far, i had no time to look into the details to implement h264 delivery via ns_connchan.
To see ns_connchan in action, look at the WebSocket code in .
We use the h264 delivery mechanism less and less these days ourselves, but we never actually used ad_background_returnfile for the purspose. Instead we've simply got a tcl thread handling the reading and writing. HTML5 and byte ranges have decreased the use of this drastically though.
Concerning ad_returnfile_background: it has the same interface like ns_returnfile. The changes for NaviServer in OpenACS are mostly due to using features of NaviServer not available in aolserver. If you get a recent version of OpenACS (e.g. the tar file), just grep for NaviServer, these should be pretty easy applicable for every OpenACS mutant.