Forum OpenACS Q&A: Aolserver 4.5 multithreading

Request notifications

Collapse
Posted by Jay Dubanik on
Hi to all,

We are currently load testing our new system oacs5.2/PG8/AOLserver4.5

The question we have is about forking new threads in 4.5.
In Aolserver 3.13 one could see several processes of nsd when running # top
In 4.5 it is always only one nsd process running, is this a proper behaviour or our multithreading is not working?

Collapse
Posted by Patrick Giagnocavo on
You are seeing a difference in the behavior of Linux's multi-thread implementation. It changed sometime around the RedHat 9 to Fedora Core days.

Most likely you have an older version of Linux that you run 3.13 on, and have a newer version of Linux that you run 4.5 on.

Read the man page on the "ps" command and look for info about threads.

You can run "ps -eLf" to see information about each thread, for instance. Running "ps -eLf | grep nsd" will show you that multiple threads are in fact running.

This article from IBM DeveloperWorks will have more information: http://www-128.ibm.com/developerworks/linux/library/l-threading.html?ca=dgr-lnxw07LinuxThreadsAndNPTL

Collapse
Posted by Jay Dubanik on
We have very strange behaviour of Aolserver 4.
We set MinThreads to 20 and MaxThreads to 60, after restart we should see 20 threads running? This worked in 3.13 but it seems it does not in 4.5. Whatever we put in MinThreads/MaxThreads under load it only goes to 15 threads, and then the server chokes.
And the nstelemetry.adp page tells us that we have MinThreads=10 MaxThreads=10 running=10.
Did anybody experienced this with 4.5?
Collapse
Posted by Claudio Pasolini on
There has been a lot of talk, mainly in the aolserver list, about the new, poorly documented way, of configuring Aolserver 4.5.
This is one significant post by tom Jackson: http://www.mail-archive.com/aolserver@listserv.aol.com/msg10976.html
Collapse
Posted by Brian Fenton on
Collapse
Posted by Malte Sussdorff on
In acs-tcl from HEAD there exists a pools-init.tcl file which will initialize the parameters as you would expect them to be initialized. Kudos for Tom to write it.
Collapse
Posted by Tom Jackson on
So did someone add this file based upon my post here? That is good for now, but it comes with a big IF, maybe several:

The current AOLserver4.5 has a single global namespace (not a Tcl namespace) for threadpool names. The built in threadpools are named default and error.

threadpools are shared among all virtual servers, usually this is not a problem with OpenACS because the usual installation only has one virtual server per nsd process.

The ns_pools Tcl command can be accessed from any virtual server, so if you have more than one, it could be a security risk. Chances are it will not be, but as a general rule, it eventually needs to be fixed.

I have written a fix which provides each virtual server with an independent namespace for threadpools and only allows access to change the current virtual server's threadpool settings. You can still see all threadpools, including the default and error threadpools.

This fix hasn't been committed to the AOLserver cvs, and this will remain the case until ns_limits is also addressed. Both of these have the same problem, and in fact, I have discovered that there are lots of parameters which we usually put in the config file that are no longer used. Bottom line is that ns_pools is the tip of the iceberg, the entire performance tuning system needs to be explained and documented.

The good news is that AOLserver performs so well with the default settings that most of us can ignore this, and the temporary fix should work okay. But I haven't reviewed it yet.

Collapse
Posted by Jay Dubanik on
Could somebody please point me to a place from where I could get this pools-init.tc file, I dont think I have acces to where it is now.

Thanks
Jay

Collapse
Posted by Tom Jackson on
Look at this message, and maybe the whole thread:

http://openacs.org/forums/message-view?message_id=1146363

You might want to post exactly what your config file has in it (minus passwords) and what you are trying to do, plus any problems you think you are having. You can also post at the Development Forum, as this is more or less a devel question.

Collapse
Posted by Jay Dubanik on
Below is our config.tcl.

Problem:
Server chokes under havy load and clients getting time out.

We have added few days ago entries in order to increase default pool min and maxthreads as you will notice in the config file.

What we want to achieve:

We would like to be able to create new threadpools and map them to certain sections of the site. ie. have a dedicated pool for /admin to be able to access server even when is nonresponsive to others.

Also have dedicated fast pool for images, if this makes sense and separate pools for POST from GET.

Here is the config.tcl file we would like to configure properly.

=======================================

ns_log notice "nsd.tcl: starting to read config file..."

######################################################################
#
# Instance-specific settings
# These default settings will only work in limited circumstances
# Two servers with default settings cannot run on the same host
#
######################################################################

#---------------------------------------------------------------------
# change to 80 and 443 for production use
set httpport 80
set httpsport 443
# If setting port below 1024 with AOLServer 4, read comments in file:
# /var/lib/aolserver/service0/packages/etc/daemontools/run

# The hostname and address should be set to actual values.
# setting the address to 0.0.0.0 means aolserver listens on all interfaces
set hostname mywebsite.com
#set address [ns_info address]
set address xxxxxxxxxxxxxxxxxxxx

# Note: If port is privileged (usually < 1024), OpenACS must be
# started by root, and, in AOLserver 4, the run script have a
# '-b address' flag which matches the address according to settings (above)

set server "myserver"
set servername "myserver Community"

set serverroot "/var/lib/aolserver/${server}"

#---------------------------------------------------------------------
# which database do you want? postgres or oracle
set database postgres

set db_name $server
set db_user $server
set db_password "xxxxxxxxxxxxxx"
set db_host localhost
set db_port 5432

#---------------------------------------------------------------------
# if debug is false, all debugging will be turned off
set debug false
#set debug true

set homedir /usr/local/aolserver45
set bindir [file dirname [ns_info nsd]]

set max_file_upload_mb 20
set max_file_upload_min 5

######################################################################
#
# End of instance-specific settings
#
# Nothing below this point need be changed in a default install.
#
######################################################################

#---------------------------------------------------------------------
#
# AOLserver's directories. Autoconfigurable.
#
#---------------------------------------------------------------------
# Where are your pages going to live ?
set pageroot ${serverroot}/www
set directoryfile index.adp,index.html,index.htm
#set charset iso-8859-2

ns_section ns/parameters
ns_param serverlog ${serverroot}/log/error.log
ns_param home $homedir
# maxkeepalive is ignored in aolserver4.x
ns_param maxkeepalive 30
ns_param logroll on
ns_param maxbackup 5
ns_param debug $debug
ns_param mailhost localhost

# # setting to Unicode by default
# # see http://dqd.com/~mayoff/encoding-doc.html
ns_param HttpOpenCharset utf-8
ns_param URLCharset utf-8
ns_param HackContentType 1
ns_param OutputCharset iso-8859-2

#---------------------------------------------------------------------
# Thread library (nsthread) parameters
#---------------------------------------------------------------------
ns_section ns/threads
ns_param mutexmeter true ;# measure lock contention
# The per-thread stack size must be a multiple of 8k for AOLServer to run under MacOS X

ns_param stacksize [expr 128 * 8192]

#
# MIME types.
#
ns_section ns/mimetypes
# Note: AOLserver already has an exhaustive list of MIME types:
# see: /usr/local/src/aolserver-4.{version}/aolserver/nsd/mimetypes.c
# but in case something is missing you can add it here.
ns_param Default text/plain
ns_param NoExtension text/plain
ns_param .pcd image/x-photo-cd
ns_param .prc application/x-pilot
ns_param .xls application/vnd.ms-excel
ns_param .doc application/vnd.ms-word

#---------------------------------------------------------------------
#
# Server-level configuration
#
# There is only one server in AOLserver, but this is helpful when multiple
# servers share the same configuration file. This file assumes that only
# one server is in use so it is set at the top in the "server" Tcl variable
# Other host-specific values are set up above as Tcl variables, too.
#
#---------------------------------------------------------------------

ns_section ns/servers
ns_param $server $servername

#
# Server parameters
#
ns_section ns/server/${server}
ns_param directoryfile $directoryfile
ns_param pageroot $pageroot
ns_param maxconnections 250 ;# max connections to put on queue
ns_param maxdropped 0
ns_param maxthreads 60
ns_param minthreads 20
ns_param threadtimeout 120 ;# Idle threads die at this rate
ns_param globalstats false ;# Enable built-in statistics
ns_param urlstats true ;# Enable URL statistics
ns_param maxurlstats 1000 ;# Max number of URL's to do stats on
# ns_param directoryadp $pageroot/dirlist.adp ;# Choose one or the other
# ns_param directoryproc _ns_dirlist ;# ...but not both!
# ns_param directorylisting fancy ;# Can be simple or fancy
#jd adding from v.3.13
ns_param ConnsPerThread 125 ;# after this number thread will exit good for getting around mem leaks

#jd new 4.5 entries
ns_pools set default -minthreads 20
ns_pools set default -maxthreads 60
ns_pools set default -maxconns 125
ns_pools set default -timeout 120
ns_log Notice "Default Pool: [ns_pools get default]"
#/jd

#---------------------------------------------------------------------
#
# ADP (AOLserver Dynamic Page) configuration
#
#---------------------------------------------------------------------
ns_section ns/server/${server}/adp
ns_param map /*.adp ;# Extensions to parse as ADP's
# ns_param map "/*.html" ;# Any extension can be mapped
ns_param enableexpire false ;# Set "Expires: now" on all ADP's
ns_param enabledebug $debug ;# Allow Tclpro debugging with "?debug"
ns_param defaultparser fancy
#adding for tuning purposes
ns_param cache true ;# in memory cache of ADP's
ns_param cachesize [expr 1000*1024]

ns_section ns/server/${server}/adp/parsers
ns_param fancy ".adp"

ns_section ns/server/${server}/redirects
ns_param 404 "global/404"
ns_param 403 "global/forbidden.html"
ns_param 500 "global/500"

#
# Tcl Configuration
#
ns_section ns/server/${server}/tcl
ns_param library ${serverroot}/tcl
ns_param autoclose on
ns_param debug $debug
#jd adding nsvbucktes for server tuning
ns_param nsvbuckets 33 ;#No. of buckets to hold nsv's

#---------------------------------------------------------------------
#
# Rollout email support
#
# These procs help manage differing email behavior on
# dev/staging/production.
#
#---------------------------------------------------------------------
ns_section ns/server/${server}/acs/acs-rollout-support

# EmailDeliveryMode can be:
# default: Email messages are sent in the usual manner.
# log: Email messages are written to the server's error log.
# redirect: Email messages are redirected to the addresses specified
# by the EmailRedirectTo parameter. If this list is absent
# or empty, email messages are written to the server's error log.
# filter: Email messages are sent to in the usual manner if the
# recipient appears in the EmailAllow parameter, otherwise they
# are logged.

# ns_param EmailDeliveryMode redirect
# ns_param EmailRedirectTo mailto:somenerd@yourdomain.test, mailto:othernerd@yourdomain.test
# ns_param EmailAllow mailto:somenerd@yourdomain.test,mailto:othernerd@yourdomain.test

#---------------------------------------------------------------------
#
# WebDAV Support (optional, requires oacs-dav package to be installed
#
#---------------------------------------------------------------------
ns_section ns/server/${server}/tdav
ns_param propdir ${serverroot}/data/dav/properties
ns_param lockdir ${serverroot}/data/dav/locks
ns_param defaultlocktimeout "300"

ns_section ns/server/${server}/tdav/shares
ns_param share1 "OpenACS"
# ns_param share2 "Share 2 description"

ns_section ns/server/${server}/tdav/share/share1
ns_param uri "/dav/*"
# all WebDAV options
ns_param options "OPTIONS COPY GET PUT MOVE DELETE HEAD MKCOL POST PROPFIND PROPPATCH LOCK UNLOCK"

#ns_section ns/server/${server}/tdav/share/share2
# ns_param uri "/share2/path/*"
# read-only WebDAV options
# ns_param options "OPTIONS COPY GET HEAD MKCOL POST PROPFIND PROPPATCH"

#---------------------------------------------------------------------
#
# Socket driver module (HTTP) -- nssock
#
#---------------------------------------------------------------------
ns_section ns/server/${server}/module/nssock
ns_param timeout 120 ;#120
ns_param address $address
ns_param hostname $hostname
ns_param port $httpport
# setting maxinput higher than practical may leave the server vulnerable to resource DoS attacks
# see http://www.panoptic.com/wiki/aolserver/166
ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes
ns_param recvwait [expr {$max_file_upload_min * 60}] ;# Maximum request time in minutes

#---------------------------------------------------------------------
#
# Access log -- nslog
#
#---------------------------------------------------------------------
ns_section ns/server/${server}/module/nslog
ns_param debug false
ns_param dev false
ns_param enablehostnamelookup false
ns_param file ${serverroot}/log/${server}.log
ns_param logcombined true
ns_param extendedheaders COOKIE
# ns_param logrefer false
# ns_param loguseragent false
ns_param logreqtime true
ns_param maxbackup 1000
ns_param rollday *
ns_param rollfmt %Y-%m-%d-%H:%M
ns_param rollhour 0
ns_param rollonsignal true
ns_param rolllog true

#---------------------------------------------------------------------
#
# nsjava - aolserver module that embeds a java virtual machine. Needed to
# support webmail. See http://nsjava.sourceforge.net for further
# details. This may need to be updated for OpenACS4 webmail
#
#---------------------------------------------------------------------
ns_section ns/server/${server}/module/nsjava
ns_param enablejava off ;# Set to on to enable nsjava.
ns_param verbosejvm off ;# Same as command line -debug.
ns_param loglevel Notice
ns_param destroyjvm off ;# Destroy jvm on shutdown.
ns_param disablejitcompiler off
ns_param classpath /usr/local/jdk/jdk118_v1/lib/classes.zip:${bindir}/nsjava.jar:${pageroot}/webmail/java/activation.jar:${pageroot}/webmail/java/mail.jar:${pageroot}/webmail/java

#---------------------------------------------------------------------
#
# CGI interface -- nscgi, if you have legacy stuff. Tcl or ADP files inside
# AOLserver are vastly superior to CGIs. I haven't tested these params but they
# should be right.
#
#---------------------------------------------------------------------
#ns_section "ns/server/${server}/module/nscgi"
# ns_param map "GET /cgi-bin/ ${serverroot}/cgi-bin"
# ns_param map "POST /cgi-bin/ ${serverroot}/cgi-bin"
# ns_param Interps CGIinterps

#ns_section "ns/interps/CGIinterps"
# ns_param .pl "/usr/bin/perl"

#---------------------------------------------------------------------
#
# PAM authentication
#
#---------------------------------------------------------------------
ns_section ns/server/${server}/module/nspam
ns_param PamDomain "pam_domain"

#---------------------------------------------------------------------
#
# OpenSSL for Aolserver 3.3 and 4
#
#---------------------------------------------------------------------
if { [ns_info version] < 4} {

#---------------------------------------------------------------------
# OpenSSL for Aolserver 3.3
#---------------------------------------------------------------------

ns_section "ns/server/${server}/module/nsopenssl"

ns_param ModuleDir ${serverroot}/etc/certs

# NSD-driven connections:
ns_param ServerPort $httpsport
ns_param ServerHostname $hostname
ns_param ServerAddress $address
ns_param ServerCertFile certfile.pem
ns_param ServerKeyFile keyfile.pem
ns_param ServerProtocols "SSLv2, SSLv3, TLSv1"
ns_param ServerCipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
ns_param ServerSessionCache true
ns_param ServerSessionCacheID 1
ns_param ServerSessionCacheSize 512
ns_param ServerSessionCacheTimeout 300
ns_param ServerPeerVerify false
ns_param ServerPeerVerifyDepth 3
ns_param ServerCADir ca
ns_param ServerCAFile ca.pem
ns_param ServerTrace false

# For listening and accepting SSL connections via Tcl/C API:
ns_param SockServerCertFile certfile.pem
ns_param SockServerKeyFile keyfile.pem
ns_param SockServerProtocols "SSLv2, SSLv3, TLSv1"
ns_param SockServerCipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
ns_param SockServerSessionCache true
ns_param SockServerSessionCacheID 2
ns_param SockServerSessionCacheSize 512
ns_param SockServerSessionCacheTimeout 300
ns_param SockServerPeerVerify false
ns_param SockServerPeerVerifyDepth 3
ns_param SockServerCADir internal_ca
ns_param SockServerCAFile internal_ca.pem
ns_param SockServerTrace false

# Outgoing SSL connections
ns_param SockClientCertFile certfile.pem
ns_param SockClientKeyFile keyfile.pem
ns_param SockClientProtocols "SSLv2, SSLv3, TLSv1"
ns_param SockClientCipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
ns_param SockClientSessionCache false
ns_param SockClientSessionCacheID 3
ns_param SockClientSessionCacheSize 512
ns_param SockClientSessionCacheTimeout 300
ns_param SockClientPeerVerify false
ns_param SockServerPeerVerifyDepth 3
ns_param SockClientCADir ca
ns_param SockClientCAFile ca.pem
ns_param SockClientTrace false

# OpenSSL library support:
# ns_param RandomFile /some/file
ns_param SeedBytes 1024

} else {

#---------------------------------------------------------------------
# OpenSSL for Aolserver 4
#---------------------------------------------------------------------
ns_section "ns/server/${server}/module/nsopenssl"

# Note this portion of the configuration is not perfect, and you
# will get errors in the your error.log. However, it does
# work. Fixes welcome.

# this is used by acs-tcl/tcl/security-procs.tcl to get the
# https port.
ns_param ServerPort $httpsport
# setting maxinput higher than practical may leave the server vulnerable to resource DoS attacks
# see http://www.panoptic.com/wiki/aolserver/166
# must set maxinput for nsopenssl as well as nssock
ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# Maximum File Size for uploads in bytes

# We explicitly tell the server which SSL contexts to use as defaults when an
# SSL context is not specified for a particular client or server SSL
# connection. Driver connections do not use defaults; they must be explicitly
# specificied in the driver section. The Tcl API will use the defaults as there
# is currently no provision to specify which SSL context to use for a
# particular connection via an ns_openssl Tcl command.
ns_section "ns/server/${server}/module/nsopenssl/sslcontexts"
ns_param users "SSL context used for regular user access"
# ns_param admins "SSL context used for administrator access"
ns_param client "SSL context used for outgoing script socket connections"

ns_section "ns/server/${server}/module/nsopenssl/defaults"
ns_param server users
ns_param client client

ns_section "ns/server/${server}/module/nsopenssl/sslcontext/users"
ns_param Role server
ns_param ModuleDir ${serverroot}/etc/certs
ns_param CertFile certfile.pem
ns_param KeyFile keyfile.pem
# ns_param CADir ca-client/dir
# ns_param CAFile ca-client/ca-client.crt
# for Protocols "ALL" = "SSLv2, SSLv3, TLSv1"
ns_param Protocols "SSLv3, TLSv1"
ns_param CipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
ns_param PeerVerify false
ns_param PeerVerifyDepth 3
ns_param Trace false

# following from bartt's nsd4.tcl, might help stablize openssl connections?
# http://www.mail-archive.com/aolserver@listserv.aol.com/msg07092.html
ns_param SessionCache true
ns_param SessionCacheID 1
ns_param SessionCacheSize 512
ns_param SessionCacheTimeout 300

# ns_section "ns/server/${server}/module/nsopenssl/sslcontext/admins"
# ns_param Role server
# ns_param ModuleDir /path/to/dir
# ns_param CertFile server/server.crt
# ns_param KeyFile server/server.key
# ns_param CADir ca-client/dir
# ns_param CAFile ca-client/ca-client.crt
# for Protocols "ALL" = "SSLv2, SSLv3, TLSv1"
# ns_param Protocols "All"
# ns_param CipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
# ns_param PeerVerify false
# ns_param PeerVerifyDepth 3
# ns_param Trace false

ns_section "ns/server/${server}/module/nsopenssl/sslcontext/client"
ns_param Role client
ns_param ModuleDir ${serverroot}/etc/certs
ns_param CertFile certfile.pem
ns_param KeyFile keyfile.pem
# ns_param CADir ${serverroot}/etc/certs
# ns_param CAFile certfile.pem
# for Protocols "ALL" = "SSLv2, SSLv3, TLSv1"
ns_param Protocols "SSLv2, SSLv3, TLSv1"
ns_param CipherSuite "ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP"
ns_param PeerVerify false
ns_param PeerVerifyDepth 3
ns_param Trace false

# following from bartt's nsd4.tcl, might help stablize openssl connections?
# http://www.mail-archive.com/aolserver@listserv.aol.com/msg07092.html
ns_param SessionCache true
ns_param SessionCacheID 1
ns_param SessionCacheSize 512
ns_param SessionCacheTimeout 300

# SSL drivers. Each driver defines a port to listen on and an explitictly named
# SSL context to associate with it. Note that you can now have multiple driver
# connections within a single virtual server, which can be tied to different
# SSL contexts.
ns_section "ns/server/${server}/module/nsopenssl/ssldrivers"
ns_param users "Driver for regular user access"
# ns_param admins "Driver for administrator access"

ns_section "ns/server/${server}/module/nsopenssl/ssldriver/users"
ns_param sslcontext users
# ns_param port $httpsport_users
ns_param port $httpsport
ns_param hostname $hostname
ns_param address $address
# following added per
# http://www.mail-archive.com/aolserver@listserv.aol.com/msg07365.html
# Maximum File Size for uploads:
ns_param maxinput [expr {$max_file_upload_mb * 1024 * 1024}] ;# in bytes
# Maximum request time
ns_param recvwait [expr {$max_file_upload_min * 60}] ;# in minutes

# ns_section "ns/server/${server}/module/nsopenssl/ssldriver/admins"
# ns_param sslcontext admins
# ns_param port $httpsport_admins
# ns_param port $httpsport
# ns_param hostname $hostname
# ns_param address $address
}

#---------------------------------------------------------------------
#
# Database drivers
# The database driver is specified here.
# Make sure you have the driver compiled and put it in {aolserverdir}/bin
#
#---------------------------------------------------------------------
ns_section "ns/db/drivers"
if { $database == "oracle" } {
ns_param ora8 ${bindir}/ora8.so
} else {
ns_param postgres ${bindir}/nspostgres.so ;# Load PostgreSQL driver
}

if { $database == "oracle" } {
ns_section "ns/db/driver/ora8"
ns_param maxStringLogLength -1
ns_param LobBufferSize 32768
}

# Database Pools: This is how AOLserver ``talks'' to the RDBMS. You need
# three for OpenACS: main, log, subquery. Make sure to replace ``yourdb''
# and ``yourpassword'' with the actual values for your db name and the
# password for it, if needed.
#
# AOLserver can have different pools connecting to different databases
# and even different different database servers. See
# http://openacs.org/doc/openacs-5-1/tutorial-second-database.html

ns_section ns/db/pools
ns_param pool1 "Pool 1"
ns_param pool2 "Pool 2"
ns_param pool3 "Pool 3"

ns_section ns/db/pool/pool1
ns_param maxidle 600
ns_param maxopen 2400
ns_param connections 20
ns_param verbose $debug
ns_param extendedtableinfo true
ns_param logsqlerrors $debug
if { $database == "oracle" } {
ns_param driver ora8
ns_param datasource {}
ns_param user $db_name
ns_param password $db_password
} else {
ns_param driver postgres
ns_param datasource ${db_host}:${db_port}:${db_name}
ns_param user $db_user
ns_param password $db_password
}

ns_section ns/db/pool/pool2
ns_param maxidle 600
ns_param maxopen 2400
ns_param connections 20
ns_param verbose $debug
ns_param extendedtableinfo true
ns_param logsqlerrors $debug
if { $database == "oracle" } {
ns_param driver ora8
ns_param datasource {}
ns_param user $db_name
ns_param password $db_password
} else {
ns_param driver postgres
ns_param datasource ${db_host}:${db_port}:${db_name}
ns_param user $db_user
ns_param password $db_password
}

ns_section ns/db/pool/pool3
ns_param maxidle 600
ns_param maxopen 2400
ns_param connections 20
ns_param verbose $debug
ns_param extendedtableinfo true
ns_param logsqlerrors $debug
if { $database == "oracle" } {
ns_param driver ora8
ns_param datasource {}
ns_param user $db_name
ns_param password $db_password
} else {
ns_param driver postgres
ns_param datasource ${db_host}:${db_port}:${db_name}
ns_param user $db_user
ns_param password $db_password
}

ns_section ns/server/${server}/db
ns_param pools "*"
ns_param defaultpool pool2

### LDAP settings

ns_section "ns/ldap/pool/ldap"
ns_param user "cn=admin,ou=SysAdmin,dc=myserver,dc=net"
ns_param password "xxxxxxxxxxxxxx"
ns_param host "box-2.mywebsite.com"
ns_param connections 2
ns_param verbose On

ns_section "ns/ldap/pools"
ns_param ldap "LDAP Pool"

ns_section "ns/server/${server}/ldap"
ns_param pools "*"
ns_param defaultpool ldap

#---------------------------------------------------------------------
# which modules should be loaded? Missing modules break the server, so
# don't uncomment modules unless they have been installed.
ns_section ns/server/${server}/modules
ns_param nssock ${bindir}/nssock.so
ns_param nslog ${bindir}/nslog.so
ns_param nssha1 ${bindir}/nssha1.so
ns_param nscache ${bindir}/nscache.so
ns_param libthread /usr/local/aolserver/lib/thread2.6.4/libthread2.6.4.so

# openacs versions earlier than 5.x requires nsxml
# ns_param nsxml ${bindir}/nsxml.so

#---------------------------------------------------------------------
# nsopenssl will fail unless the cert files are present as specified
# later in this file, so it's disabled by default
# ns_param nsopenssl ${bindir}/nsopenssl.so

# authorize-gateway package requires dqd_utils
# ns_param dqd_utils dqd_utils[expr {int($tcl_version)}].so

# Full Text Search
# ns_param nsfts ${bindir}/nsfts.so

# LDAP authentication
ns_param nsldap ${bindir}/nsldap.so

# PAM authentication
# ns_param nspam ${bindir}/nspam.so


# These modules aren't used in standard OpenACS installs
# ns_param nsperm ${bindir}/nsperm.so
# ns_param nscgi ${bindir}/nscgi.so
# ns_param nsjava ${bindir}/libnsjava.so
# ns_param nsrewrite ${bindir}/nsrewrite.so

if { [ns_info version] >= 4 } {
# Required for AOLserver 4.x
ns_param nsdb ${bindir}/nsdb.so
} else {
# Required for AOLserver 3.x
ns_param libtdom ${bindir}/libtdom.so
}

# nsthread library which should become standard in 5.3
# if {[file exists ${homedir}/lib/thread2.6.4/libthread2.6.4[info sharedlibextension]]} {
# ns_param libthread ${homedir}/lib/thread2.6.4/libthread2.6.4[info sharedlibextension]
# }
#ns_param libthread /usr/local/aolserver/lib/thread2.6.4/libthread2.6.4.so

ns_log notice "nsd.tcl: using threadsafe tcl: [info exists tcl_platform(threaded)]"
ns_log notice "nsd.tcl: finished reading config file."

Collapse
Posted by Tom Jackson on

Jay,

I'm going to post here a little script which you can run to produce the actual contents of your config after startup.

You can run this in a tcl page, or during startup (which helps if the server doesn't actually startup).

I'm asking you to do this so that it is easy to figure out what settings are being used, and to avoid any extra junk.

namespace eval ::config {
 
    set Debug 1
    
    if {[info exists Debug] && $Debug} {
	
	set sections [ns_configsections]
	
	set output [list]
	
	foreach section $sections {
	    set name [ns_set name $section]
	    set section_txt ""
	    append section_txt "ns_section $name\n"
	    
	    set size [ns_set size $section]
	    for {set i 0} {$i < $size} {incr i} {
		append section_txt "\t[list ns_param [ns_set key $section $i] [ns_set value $section $i]]\n"
	    }
	    lappend output $section_txt
	    
	}
	set output "[join [lsort $output] "\n"]\n"
	puts "\n$output\n"
    }
 
}

ns_log does not work in a config script, the calls may show up if the server starts up, but will not if there is an error, you can use puts as above.

Collapse
Posted by Jay Dubanik on
Hi Tom,
Thanks for the script, here is the result of running it.

FILE Start===============

ns_section ns/db/drivers
ns_param postgres /usr/local/aolserver/bin/nspostgres.so

ns_section ns/db/pool/pool1
ns_param maxidle 600
ns_param maxopen 2400
ns_param connections 20
ns_param verbose false
ns_param extendedtableinfo true
ns_param logsqlerrors false
ns_param driver postgres
ns_param datasource localhost:5432:myserver
ns_param user myserver
ns_param password xxxxxxxxxxxxxxxxxxxx

ns_section ns/db/pool/pool2
ns_param maxidle 600
ns_param maxopen 2400
ns_param connections 20
ns_param verbose false
ns_param extendedtableinfo true
ns_param logsqlerrors false
ns_param driver postgres
ns_param datasource localhost:5432:myserver
ns_param user myserver
ns_param password xxxxxxxxxxxxxxxxxxxx

ns_section ns/db/pool/pool3
ns_param maxidle 600
ns_param maxopen 2400
ns_param connections 20
ns_param verbose false
ns_param extendedtableinfo true
ns_param logsqlerrors false
ns_param driver postgres
ns_param datasource localhost:5432:myserver
ns_param user myserver
ns_param password xxxxxxxxxxxxxxxxxxxx

ns_section ns/db/pools
ns_param pool1 {Pool 1}
ns_param pool2 {Pool 2}
ns_param pool3 {Pool 3}

ns_section ns/ldap/pool/ldap
ns_param user cn=admin,ou=SysAdmin,dc=myserver,dc=net
ns_param password xxxxxxxxxxxxxxxxxxxx
ns_param host box-2.myserver.net
ns_param connections 2
ns_param verbose On

ns_section ns/ldap/pools
ns_param ldap {LDAP Pool}

ns_section ns/mimetypes
ns_param Default text/plain
ns_param NoExtension text/plain
ns_param .pcd image/x-photo-cd
ns_param .prc application/x-pilot
ns_param .xls application/vnd.ms-excel
ns_param .doc application/vnd.ms-word

ns_section ns/parameters
ns_param serverlog /var/lib/aolserver/myserver/log/error.log
ns_param home /usr/local/aolserver45
ns_param maxkeepalive 30
ns_param logroll on
ns_param maxbackup 5
ns_param debug false
ns_param mailhost localhost
ns_param HttpOpenCharset utf-8
ns_param URLCharset utf-8
ns_param HackContentType 1
ns_param OutputCharset iso-8859-2

ns_section ns/server/myserver
ns_param directoryfile index.adp,index.html,index.htm
ns_param pageroot /var/lib/aolserver/myserver/www
ns_param maxconnections 250
ns_param maxdropped 0
ns_param maxthreads 60
ns_param minthreads 20
ns_param threadtimeout 120
ns_param globalstats false
ns_param urlstats true
ns_param maxurlstats 1000
ns_param ConnsPerThread 125

ns_section ns/server/myserver/acs/acs-rollout-support

ns_section ns/server/myserver/adp
ns_param map /*.adp
ns_param enableexpire false
ns_param enabledebug false
ns_param defaultparser fancy
ns_param cache true
ns_param cachesize 1024000

ns_section ns/server/myserver/adp/parsers
ns_param fancy .adp

ns_section ns/server/myserver/db
ns_param pools *
ns_param defaultpool pool2

ns_section ns/server/myserver/ldap
ns_param pools *
ns_param defaultpool ldap

ns_section ns/server/myserver/module/nsjava
ns_param enablejava off
ns_param verbosejvm off
ns_param loglevel Notice
ns_param destroyjvm off
ns_param disablejitcompiler off
ns_param classpath /usr/local/jdk/jdk118_v1/lib/classes.zip:/usr/local/aolserver/bin/nsjava.jar:/var/lib/aolserver/myserver/www/webmail/java/activation.jar:/var/lib/aolserver/myserver/www/webmail/java/mail.jar:/var/lib/aolserver/myserver/www/webmail/java

ns_section ns/server/myserver/module/nslog
ns_param debug false
ns_param dev false
ns_param enablehostnamelookup false
ns_param file /var/lib/aolserver/myserver/log/myserver.log
ns_param logcombined true
ns_param extendedheaders COOKIE
ns_param logreqtime true
ns_param maxbackup 1000
ns_param rollday *
ns_param rollfmt %Y-%m-%d-%H:%M
ns_param rollhour 0
ns_param rollonsignal true
ns_param rolllog true

ns_section ns/server/myserver/module/nsopenssl
ns_param ServerPort 443
ns_param maxinput 20971520

ns_section ns/server/myserver/module/nsopenssl/defaults
ns_param server users
ns_param client client

ns_section ns/server/myserver/module/nsopenssl/sslcontext/client
ns_param Role client
ns_param ModuleDir /var/lib/aolserver/myserver/etc/certs
ns_param CertFile certfile.pem
ns_param KeyFile keyfile.pem
ns_param Protocols {SSLv2, SSLv3, TLSv1}
ns_param CipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
ns_param PeerVerify false
ns_param PeerVerifyDepth 3
ns_param Trace false
ns_param SessionCache true
ns_param SessionCacheID 1
ns_param SessionCacheSize 512
ns_param SessionCacheTimeout 300

ns_section ns/server/myserver/module/nsopenssl/sslcontext/users
ns_param Role server
ns_param ModuleDir /var/lib/aolserver/myserver/etc/certs
ns_param CertFile certfile.pem
ns_param KeyFile keyfile.pem
ns_param Protocols {SSLv3, TLSv1}
ns_param CipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP
ns_param PeerVerify false
ns_param PeerVerifyDepth 3
ns_param Trace false
ns_param SessionCache true
ns_param SessionCacheID 1
ns_param SessionCacheSize 512
ns_param SessionCacheTimeout 300

ns_section ns/server/myserver/module/nsopenssl/sslcontexts
ns_param users {SSL context used for regular user access}
ns_param client {SSL context used for outgoing script socket connections}

ns_section ns/server/myserver/module/nsopenssl/ssldriver/users
ns_param sslcontext users
ns_param port 443
ns_param hostname myserver.net
ns_param address xxx.xxx.xxx.xxx
ns_param maxinput 20971520
ns_param recvwait 300

ns_section ns/server/myserver/module/nsopenssl/ssldrivers
ns_param users {Driver for regular user access}

ns_section ns/server/myserver/module/nspam
ns_param PamDomain pam_domain

ns_section ns/server/myserver/module/nssock
ns_param timeout 120
ns_param address xxx.xxx.xxx.xxx
ns_param hostname myserver.net
ns_param port 80
ns_param maxinput 20971520
ns_param recvwait 300

ns_section ns/server/myserver/modules
ns_param nssock /usr/local/aolserver/bin/nssock.so
ns_param nslog /usr/local/aolserver/bin/nslog.so
ns_param nssha1 /usr/local/aolserver/bin/nssha1.so
ns_param nscache /usr/local/aolserver/bin/nscache.so
ns_param libthread /usr/local/aolserver/lib/thread2.6.4/libthread2.6.4.so
ns_param nsldap /usr/local/aolserver/bin/nsldap.so
ns_param nsdb /usr/local/aolserver/bin/nsdb.so

ns_section ns/server/myserver/redirects
ns_param 404 global/404
ns_param 403 global/forbidden.html
ns_param 500 global/500

ns_section ns/server/myserver/tcl
ns_param library /var/lib/aolserver/myserver/tcl
ns_param autoclose on
ns_param debug false
ns_param nsvbuckets 33

ns_section ns/server/myserver/tdav
ns_param propdir /var/lib/aolserver/myserver/data/dav/properties
ns_param lockdir /var/lib/aolserver/myserver/data/dav/locks
ns_param defaultlocktimeout 300

ns_section ns/server/myserver/tdav/share/share1
ns_param uri /dav/*
ns_param options {OPTIONS COPY GET PUT MOVE DELETE HEAD MKCOL POST PROPFIND PROPPATCH LOCK UNLOCK}

ns_section ns/server/myserver/tdav/shares
ns_param share1 OpenACS

ns_section ns/servers
ns_param myserver {myserver Community}

ns_section ns/threads
ns_param mutexmeter true
ns_param stacksize 1048576

=========File End

Collapse
Posted by Tom Jackson on

Okay, so the basic default and error threadpools will use the info in

ns_section ns/server/myserver
ns_param directoryfile index.adp,index.html,index.htm
ns_param pageroot /var/lib/aolserver/myserver/www
ns_param maxconnections 250
ns_param maxdropped 0
ns_param maxthreads 60
ns_param minthreads 20
ns_param threadtimeout 120
ns_param globalstats false
ns_param urlstats true
ns_param maxurlstats 1000
ns_param ConnsPerThread 125

If you want to setup additional threadpools, you should look at the following example:

http://rmadilo.com/m2/servers/jnm/config/

The virtual server setup is in server.tcl, this ends up calling the other files in that directory for individual modules. But note, that you need to select different names than default or error for your specialized threadpools, otherwise the name will overwrite the values above.

You should also load the nscp module (ns control port), so that you can login to the running server and use [ns_pools list] and [ns_pools get] to get a list of the current threadpools and their settings:

 tom@localhost ~ $ telnet localhost 9999
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
login: nsadmin
Password: x

Welcome to jnm running at /web/nsd45/bin/nsd (pid 21784)
AOLserver/4.5.0 (aolserver4_5) for linux built on Aug 12 2007 at 09:18:31
CVS Tag: $Name:  $
jnm:nscp 1> ns_pools list
jnm-fast tutos-default default tutos-fast error
jnm:nscp 2> ns_pools get jnm-fast
minthreads 2 maxthreads 10 idle 2 current 2 maxconns 100 queued 0 timeout 120
jnm:nscp 3>         

The nscp.tcl file has the setup needed, the password is x. Note that this is to localhost, only a local user can login, you have to use ns_encrypt to create a new password.

Collapse
Posted by Jay Dubanik on
Tom,
Thank you for this info.
I have sucessfuly created new threadpools.

as for mapping.
Is there a way to find out which threadpool is actualy taking your request?
and lastly should myDefault be mapped with { GET /} or {GET /*} ?

Thanks

Collapse
Posted by Tom Jackson on
I've written a patch to 4.5 which prints the threadpool used for each log statement printed. It hasn't been added to CVS yet. I also added a new subcommand to ns_pools:

ns_pools query server method url

It returns the name of the pool which would be used given the server (the $server string in your config file), method (GET/POST/HEAD/whatever) and the request url.

I'm not sure if {GET /} works or not, if you want it to be a default, use {GET /*} just to be clear.

Here is what the patched server shows:
[13/Sep/...][-nssock2:driver-] Notice: GetPool: matched method GET and url /ws/a.jpg on server tutos using pool tutos-default
[13/Sep/...][-tutos-default:2-] Notice: Running file-not-found.tcl for GET /ws/a.jpg HTTP/1.1
...
[13/Sep/...][-nssock2:driver-] Notice: GetPool: matched method GET and url /ws/a-thumb.jpg on server tutos using pool tutos-fast
[13/Sep/...][-tutos-fast:0-] Notice: Running file-not-found.tcl for GET /ws/a-thumb.jpg HTTP/1.1

where tutos-fast and tutos-default are the two threadpool names. In unpatched versions, you would get -conn:0- instead.

Collapse
Posted by Jay Dubanik on
Tom, I hope your patch to 4.5 to print the threadpool name will get in quickly, I'd like to do more tests as I am still not happy with the performance.

The problem is that even that threadpools seem to work, or at least are loaded (see below), when I run "ps -eLf" I get only 15 threads open.

1. In my understanding there in fact should be at least as many as the sum of all threadpools minthreads in my case around 90.

2. When looking at network stats with "netstat -s" (see below) the number of active connections is strangely low compared to passive conns and high number of failed connections.

Tcp:
20778 active connections openings
1798957 passive connection openings
574669 failed connection attempts
98681 connection resets received
112 connections established
45683111 segments received
57018897 segments send out
866619 segments retransmited
667 bad segments received.
220484 resets sent

========config-info===============
ns_section ns/server/myserver/pool/admin
ns_param maxconnections 100
ns_param minthreads 2
ns_param maxthreads 2
ns_param threadtimeout 120
ns_param map {GET */admin/*}
ns_param map {POST */admin/*}
ns_param map {GET /acs-admin/*}
ns_param map {POST /acs-admin/*}

ns_section ns/server/myserver/pool/fast
ns_param maxconnections 200
ns_param minthreads 10
ns_param maxthreads 20
ns_param threadtimeout 120
ns_param map {GET /*.jpg}
ns_param map {GET /*.swf}
ns_param map {GET /image/*}

ns_section ns/server/myserver/pool/mainpool
ns_param maxconnections 200
ns_param minthreads 80
ns_param maxthreads 80
ns_param threadtimeout 120
ns_param map {GET /*}
ns_param map {POST /}

ns_section ns/server/myserver/pools
ns_param fast {fast pool}
ns_param admin {admin pool}
ns_param mainpool {mainpool pool}

Collapse
Posted by Tom Jackson on

I admit that I'm no patch expert. The change needed to log the threadpool used is pretty simple:

cvs diff queue.c
Index: queue.c
===================================================================
RCS file: /cvsroot/aolserver/aolserver/nsd/queue.c,v
retrieving revision 1.38
diff -r1.38 queue.c
361c361
<     sprintf(name, "-conn:%d-", poolPtr->threads.nextid++);
---
>     sprintf(name, "-%s:%d-", poolPtr->name, poolPtr->threads.nextid++);

So line 361 gets a little change.

I also notice that your mainpool threadpool only maps "/" for the method POST. I haven't experimented, but this might not do anything but url = "/". If you make the change to the queue.c, maybe report if mainpool is used for POST on urls other than "/".

Collapse
Posted by Jay Dubanik on
Tom.Thanks for that.

Could you confirm on your system that setting min (and max)threads does work. In my case the number of open threads stays same regardless. As I mentioned in my post above I should see around 90 open threads and there are only 15 and the number never goes up, users are getting 'time-outs' and no new threads are created.

This worked fine in 3.13 and if this is not working in 4.5 then 4.5 in my view is not commercialy usable.

Collapse
Posted by Tom Jackson on
Without seeing your actual server.log file it is hard to say very much.

I did a test where jnm-fast was started with minthreads=2. I used nscp to change this to 12. Maybe the expectation is that 10 new threads will appear. However, new threads may never be created if the current number is enough. At any rate, you have to look at the log file to see that the threadpool is actually being used. You may have one threadpool sucking up all the requests.

Next test was to change my config to minthreads=12 for jnm-fast and restart. I got 10 extra threads at startup. So in this limited test, minthreads is working. I've done other tests which indicate that min/max are working. And the timeout is also working.

If you are really interested in using 4.5, maybe I should provide you the other changes I have made. But, I'll probably make further changes before they are accepted into AOLserver CVS. My C code changes allow the threadpools configuration in 4.0.10 to work, but you also need to replace the pools.tcl file.

One benefit is that you get a new command:

ns_pools query server method url

which returns the name of the threadpool which would be used for the given server, method and url.

Collapse
Posted by Jay Dubanik on
Tom, I have some more news since yestarday.

As I said setting threadpools did nothing in terms of number of threads actualy started, until I have entered the following lines into config.tcl file.

ns_pools set default -minthreads 80
ns_pools set default -maxthreads 200
ns_pools set default -maxconns 200
ns_pools set default -timeout 120
ns_log Notice "Default Pool: [ns_pools get default]"

Suddenly I have got 80 threads opened when checked with "ps -eLF"
This however lasted only some time and threads started to die below the -minthreads setting, as low as 13.
This tells me that -minthreads works only at startup and then when threads die it is not checked anymore, which looks like a bug to me.

Collapse
Posted by Tom Jackson on

Last week there was a similar question on the AOLserver list, here is what I guessed was going on:

You're not doing anything wrong. You have a thread timeout. So after 60 
seconds, the idle thread exits. Maybe there is a bug in pools.c, but what is 
happening is that at startup, minthreads are created. If you go in and 
increase minthreads after starup of the pool, the minthreads will not 
increase, ever. But if new requests come in, new threads will be created. 
They are probably being serviced so fast that you never get too many at once 
showing up. If this isn't the case, then it could be a problem. 

So maybe play around with the timeout value, maybe if it is zero they don't 
timeout. If you want threads to exit at some point, use maxconnections to 
make them exit. 

So the problem is very likely that as maxconns is reached, threads are 
deleted, but there is never any check that numthreads have fallent below a 
minimum value. Hopefully you will reach a number of threads necessary to 
service the requests without going over maxthreads, but minthreads appears to 
be only a hint for startup, after that the threadpools regulates itself to 
minimum resources. However...does it ever reach zero, because if it does, I 
have noticed that the threadpool no longer responds to requests. This would 
be a definite bug. 

tom jackson

On Monday 01 October 2007 07:45, ** ** wrote:

> You said that current should never drop below min. But in my case after
> threads have serviced configured number of requests they start to die. idle
> and current keeps dropping  until 0 and never comes up again.

Collapse
Posted by Tom Jackson on
I should clarify this comment. If maxthreads is set to zero after startup, as new requests are serviced, the thread will exit until all threads are gone. Since maxthreads is zero, no new threads will be created...but the driver thread still hands off requests which are never serviced. It should probably be illegal to set maxthreads to less than 1.

Maintaining at least minthreads could become difficult. I think you could consider it a bug, but maybe not. The result is this: your server will regulate threads down to the number required, a 'high water mark' less than or equal to maxthreads.

The best way to test this is to use something like apache bench. Set the concurrency above the minthreads and see if you can get it to go back over that amount. (Then let it rest and see threads exit, during testing you can set a low thread timeout.) Then set concurrency to something above maxthreads and see how it performs. You can test pages which are slow and see what happens. You will probably also need to keep pushing up the number of request to run (-n 500, or whatever), so that each thread gets multiple requests, and the queue gets a little full. You can also set the maxconnections to different values (very low and very high) and test that. Threads are started very quickly, but their startup cost is spread over maxconnections requests.

I'm guessing that what you will discover is that minthreads is a good hint at startup, but may be too many. If it is always too many, then you are wasting memory. Only by watching a production system can you determine a reasonable value for minthreads. If you have massive fluctuations in the number of requests over a time larger than thread timeout, then minthreads is pretty useless, you can expect it to drop below that soon after startup.

Collapse
Posted by Nima Mazloumi on
Hi,

my server suddenly freezes under load. The server is still runing by not responding and there log freezes as well.

I use nscp to get some details:

nscp 22> ns_pools get default
minthreads 0 maxthreads 150 idle 0 current 100 maxconns 750 queued 21891 timeout 120

Any idea?

I've been playing with maxthreads and maxconnections. With smaller values the server freezes even earlier. This is a 2 CPU system with 4 cores, 64bit linux suse and 4 GB ram. I am using aolserver 4.5.

Collapse
Posted by Gustaf Neumann on
Nima, there are a few fixes in this regard in aolserver 4.5.1. I would recommend to upgrade.

-gustaf neumann