Forum OpenACS Development: Re: OpenACS Performance Tests
The first tests with 100 simultaneous connections in the start page using JMeter had a top of 7900ms response time and an average 2500ms. With 1000 simultaneous connections we had 9000ms top and 4000ms average.
Does that mean average response was 2.5 seconds with 100 connections?
If so the next step is to figure out where the bottleneck is. You should be under 1 second, and 7.9 seconds is totally unacceptable.
When the test is running what is the load like? How much memory is nsd and postgres using, how much cpu etc?
Offhand I am not sure if 50 connections for the main pool is enough for 100 simultaneous connections or not. Depends on how long the queries run, but a standard request does quite a few queries so, the threads might be waiting for open postgres connections.
It looks like you have MORE than enough shared buffers at 1.5G. The work_mem is 8mb. That is probably enough for most queries that would be found on a community home page.
Thank you for the answers. I can say that this average response time is unacceptable. There's something very interesting about response time: when I look at the XoTcl request monitor page, the average response time is about 200ms with 100 simultaneous connections, but JMeter shows me (instantly) tops of 7900ms and average 2500ms (yes, 2.5s). I don't know where this difference comes from. Maybe somebody who knows the package better can tell how it measures the average.
However, even with this response time, I could navigate with no problems. It was a little bit slow, but nothing impossible to use. In the other hand, with 1000 simultaneous connections it was impossible to navigate.
About the db pools, I think we have a problem here. No matter how much I raise this umber, I can only see in htop something between 20 and 23 connections at the DB. Maybe this has something to do with AOLServer 4.5.0 thread pools change. Can somebody confirm that?
When I'm running JMeter, I can see all the processors with 100% (four processors) and I'm using only 4GB RAM on the machine. It seems like I can't make the machine use more memory, and that's why I've increased shared_buffers to 1.5GB.
Maybe I have to find a more clever way to track queries elapsed time. I'm going to install ptop, but I don't know if that's good enough. Does anybody know a better tool?
when I look at the XoTcl request monitor page, the average response time is about 200ms with 100 simultaneous connections, but JMeter shows me (instantly) tops of 7900ms and average 2500ms (yes, 2.5s).
the xotcl-request-monitor measures the time between request-start and request-end but it does not know the time the request was sitting in the socket or driver queue. There is a bug in the released aolserver 4.5, that i have fixed in the head version of aolserver 4.5 in Oct 2007. If aolserver is swamped with more requests than it can process, these requests can pile up in a queue. This happens easily when running out of resources (e.g. only a few connection threads are defined, or in benchmark situations).
You should try the head version of aolserver 4.5.
From the config-file i see that you have defined maxthreads=100 and in your first pool the connections 50. This value is too small (make it as well 90 or 100), otherwise the connection threads will have to wail for the db-connection.
see below for some other small changes that i would recommend, which will not impact your performance much.
another question: did you compile tcl+aolserver and postgres with 64bit?
what are the queries you are benchmarking?
ns_param maxconnections 250 ;# Max connections to put on queue
ns_param threadtimeout 120 ;# Idle threads die at this rate
ns_param maxconnections 5000 ;# Max connections to put on queue
ns_param threadtimeout 120 ;# Idle threads die at this rate
ns_param maxidle 1000000000
ns_param maxopen 1000000000
# ns_param maxidle 0
# ns_param maxopen 0
Thank you very much for your answers. They are being very helpfull, as allways.
We really have a problem in measurement here. I guess JMeter results will be more accurate in response time, because it can track all the time that the request is stucked. Do you think this diference between JMeter and XoTcl Request Monitor could be an I/O problem? I'm sorry if this looks like a silly question, but I'm still learning this thing.
Concerning AOLServer, I've been following the discussions in the mail list and I've realized that it has some performance problems that people are fixing. So I've checked it out from CVS using Cognovis script, like this:
cvs -z3 -d:pserver:mailto:firstname.lastname@example.org:/cvsroot/aolserver co aolserver
I can say I have the latest version. Do you know how I can check the source code to make sure your patches are there?
About the threads, I can see some kind of problem, maybe even a bug in thread pools definition for AOLServer 4.5.0: no matter how much I increase the number of connections in pool1, I can only see something between 20 and 23 DB connections. This makes me thing the parameter change is not working. Do you know how can I confirm that?
Your last consideration is maybe the most important: I don't know if AOLServer is compiled for 64 bits. I used Cognovis script, so I can assume it's not compiled for 64 bits. Is that right? Do you know where I can find some documentation about how to do it?
PostgreSQL is compiled for 64 bits, I'm sure, because it was installed from backports.org postgresql-8.2 package. I forgot to mention that my OS is Debian Etch.
Last question: what are these parameters maxidle e maxopen for?
Thank you again for your help.
just by guessing i would not be surprised if your are not get more than 10 to 15 connections threads, since most likely, maxthreads is not used. If this is true, you don't need more db-connections.
Add the following script (from Tom Jacksons pool.tcl) to the end of your config file, which takes the old style configuration and uses the 4.5 ns_pools command.
set cfgsection "ns/server/[ns_info server]" set minthreads [ns_config $cfgsection minthreads 0] set maxthreads [ns_config $cfgsection maxthreads 10] set maxconns [ns_config $cfgsection maxconnections 0] set timeout [ns_config $cfgsection threadtimeout 0] ns_pools set default -minthreads $minthreads -maxthreads $maxthreads -maxconns $maxconns -timeout $timeout
If you use e.g. 100 connection threads, and you load dotlrn, then you won't be able to run this in 32bit mode (2 GB). In this case, you have to go to 64 bit (necessary for everything, except the postgres server). For compiling e.g. Tcl, you have to configure with "--enable-64bit". I have no idea, what's available in debian etch in 64 bit.
In order to figure out, if a binary (e.g. nsd) or a library (.so, .o) was compiled with 64bit, try the "file" command.
maxidle and maxopen are timeout values, necessary as bugfix for ancient versions of aolserver, obsolete since a few years, but passed from config file to config file.