Forum OpenACS Development: Re: maxidle and maxopen

Collapse
2: Re: maxidle and maxopen (response to 1)
Posted by Gustaf Neumann on 03/23/07 12:37 PM
set maxidle and maxopen to 0
Collapse
3: Re: maxidle and maxopen (response to 2)
Posted by Maurizio Martignano on 03/23/07 02:11 PM
Dear Gustaf,
thanks a lot for your reply. Some few additional questions:

1. what is the actual effect of specifying

ns_param maxidle 0

?

That there's no timeot on the idle connections (this is the same as sspecifying a timeout of roughly 32 years) or that idle connections are immediately killed?

2. Wouldn't be better to put somethign like
ns_param maxidle 600
ns_param maxopen 3600

? I.e. some actual finite values?

Thanks a lot in advance,
Maurizio

Collapse
4: Re: maxidle and maxopen (response to 3)
Posted by Gustaf Neumann on 03/23/07 06:15 PM
these ns_parameters were introduced with the 1000000000 setting to cope with a bug in aolserver 3.* which was fixed years ago. by setting these values to 0, the default values are used, which are - surprise, surprise - exactly the values you wrote.
Collapse
5: Re: maxidle and maxopen (response to 4)
Posted by Maurizio Martignano on 03/23/07 06:29 PM
Dear Gustaf,
once again thanks a lot for your reply.
This is what I read on the Aolserver documentation:

MaxIdle - The maximum length of time in seconds that a database connection within this pool can remain open and idle. The default setting causes connections that are idle for 10 minutes to be closed. Note that MaxIdle will not have an effect if it is equal to MaxOpen. Setting MaxIdle to 0 makes it impossible for database handles to become stale, unless the MaxOpen time expires.

MaxOpen - The maximum length of time in seconds that a database connection within this pool can remain open. It is recommended that MaxOpen be a multiple of MaxIdle; otherwise, MaxOpen may be off by as much as (MaxOpen mod MaxIdle). The default setting causes all connections to be closed after one hour, regardless of activity level. Database pool connections will then be opened again as needed up to the value of the Connections parameter. Setting MaxOpen to 0 makes it impossible for database handles to become stale when they are not idle.

Setting MaxIdle or MaxOpen to 0 does not (DOES NOT) correspond to using their default values.

Cheers and thanks,
Maurizio

Collapse
8: Re: maxidle and maxopen (response to 5)
Posted by Gustaf Neumann on 03/24/07 12:19 PM
You are right, 0 does not mean "default" values. sorry, i misread a "<0" as a "<=0". The actual code says that handles are closed, when
  • handles are not used for some time (unless maxidle is 0), or
  • handles are already open for a long time (unless maxopen is 0), or
  • handles are stale
So, setting these values to 0 disables time-specific closing of handles, which corresponds to the intention of the old default value of "1000000000". The old value can lead to some strange hanging problems due to to integer overflows on 32bit unixes, where the added value overshoots "Unix End of Times". So, 0 should be used to reach the no-close effect.

For details, see: http://dossy.org/archives/000319.html

Collapse
9: Re: maxidle and maxopen (response to 8)
Posted by Maurizio Martignano on 03/24/07 05:35 PM
Dear Gustaf,
thanks a lot for your reply.

Some few weeks ago I published a post about having some memory allocation problems with aolserver/openacs. Initially I thought these problems were due to the aolserver/TCL allocator (i.e. the standard mechanism, or Zippy - the default one, or VTMalloc). But then I observed that the majority of the menory was consumed by PostgreSQL connections. By setting the parameters MaxIdle and MaxOpen to their defualt values (600 and 3600) I managed to fix this problem.

I would recommend to update accordingly these parameters in the standard startup file "congig.tcl" which gets shipped together with the OpenACS distribution(s). This way newbies like me won't fall again in this memory problem and invest a lot of time and resources trying to solve it.

Thanks a lot in advance,
Maurizio

Collapse
10: Re: maxidle and maxopen (response to 9)
Posted by Gustaf Neumann on 03/24/07 09:20 PM
Maurizio,

this is very interesting. I'll try this on my configurations and check, how the memory behavior changes...

Collapse
11: Re: maxidle and maxopen (response to 10)
Posted by Maurizio Martignano on 03/25/07 11:15 AM
Gustaf,
I'll be very interested in knowing your results.
BTW: While you're working on this issue, I noticed that changing the "connections" parameters from 5 to 1 might also help in reducing the memory occupation.

Cheers and thanks,
Maurizio

Collapse
12: Re: maxidle and maxopen (response to 11)
Posted by Gustaf Neumann on 03/25/07 12:36 PM
"working" is an exaggeration. On four of my installations, i have commented out the maxopen/maxidle statements, falling back to the default values of aolserver. So far, i have not noticed significant differences, but its (a) sunday and (b) to early to tell.

Btw, reducing connections from 5 to 1 might reduce memory, but the same time your are reducing greatly the ability to run db-queries in parallel. If you have multiple http requests running, and all of these require handles from the pools, and you have ony 1 connection configured, the queries are sequentialized, since each query has to wait until a db connection is available. This means that he waiting time for handles will go up. If most of your http requests use the database the value for maxthreads must be aligned with connections. On our main site, we have 90 connections configured.

Collapse
13: Re: maxidle and maxopen (response to 12)
Posted by Maurizio Martignano on 03/25/07 03:23 PM
Dear Gustaf, thanks a lot. I'll be waiting for some additional tests from yuor side. Please let me know what is the amount of the differences you noticed, if any.

I do agree that the change in the "connection" value reduces the ability to run db-quesires in parallel.

Cheers and thanks,
MAurizio

Collapse
14: Re: maxidle and maxopen (response to 13)
Posted by Gustaf Neumann on 03/26/07 10:24 AM
Maurizio,

there are no significant changes on my sites. I altered the configuration between Sat. and Sun. On Sat we had 421370 hits, on Sun 488751. The chart shows that the growth of the virtual size is happening with and without the MaxIdle/MaxOpen = 0 settings: http://media.wu-wien.ac.at/download/nsdsize-week-2007-03-26.png

Btw, the M should be a G in the chart. Also my other installation did not show improvements.

How did you measure savings? Is this maybe Windows specific?

Collapse
15: Re: maxidle and maxopen (response to 14)
Posted by Maurizio Martignano on 03/26/07 03:17 PM
Hello Gustaf, this is what I got on a SUSE Enterprise Linux Server 10 with Linux 2.6.16.27-0.6-smp on a test/benchmnark installation with aolserver 4.0.10 and OpenACS 4 
ns_section ns/db/pool/pool1
ns_param   maxidle            60   instead of 1000000000
ns_param   maxopen            360  instead of 1000000000
ns_param   connections        1    instead of 5


  PID MINFLT MAJFLT      VSTEXT  VSIZE  RSIZE  VGROW  RGROW  MEM CMD     1/11
 4143  17341      0          1K 51036K 33860K 51036K 33860K   3% nsd
 4146  10226      0       2140K 135.0M 27376K 135.0M 27376K   2% postmaster
 4147   1204      0       2140K 132.8M  4964K 132.8M  4964K   0% postmaster
 3874     58      0       2140K 131.8M  3084K     0K     4K   0% postmaster
 4148    638      0       2140K 132.3M  2716K 132.3M  2716K   0% postmaster


ATOP - icaro              2007/03/22  17:44:27              600 seconds elapsed
  PID MINFLT MAJFLT      VSTEXT  VSIZE  RSIZE  VGROW  RGROW  MEM CMD     1/7
 4143  10433      0          1K 66576K 47568K 15540K 13708K   4% nsd
 4192  69619      0       2140K 134.9M 110.0M 134.9M 110.0M   9% postmaster
 4203    998      0       2140K 132.6M  4144K 132.6M  4144K   0% postmaster
 3874     59      0       2140K 131.8M  3084K     0K     0K   0% postmaster
 3878      7      0       2140K  6472K   892K     0K     0K   0% postmaster


ATOP - icaro              2007/03/22  17:54:27        
 4143   2374      0          1K 66576K 47560K     0K    -8K   4% nsd
 4239   1039      0       2140K 132.6M  4312K 132.6M  4312K   0% postmaster
 3874     47      0       2140K 131.8M  3084K     0K     0K   0% postmaster
 3878      5      0       2140K  6472K   892K     0K     0K   0% postmaster
 3877      0      0       2140K  7304K   712K     0K     0K   0% postmaster
After the first set of data I stayed put (i.e. I did not touch/used the system) and as you can see in the following sets of data that PostgreSQL released some memory (have a look at the VSIZE value). When I was using timeouts with values of 1000000000 or of 0 the memory did not get released by PostgreSQL.
Yes it is true, I put some very short timeouts. But I did that to observe some changes in the allocated memory as soon as possible.

Hope it helps,
Maurizio