Forum OpenACS Q&A: Problem

Collapse
Posted by Richard Hamilton on
Just working through the docs and I think there is a problem with the bit shown below.

If you follow this instruction to set shared buffers to 15200, the memory request to the kernel exceeds the figure 134217728. It actually requests 134987776 instead which causes a start failure.

The postgres docs say that shared buffers should be set to 2*the max_connections. Our docs do not mention increasing max_connections which seems odd.

Are we sure that we are not just allocating memory that then sits unused?

__________________________________________________________

Tune postgres. (OPTIONAL). The default values for PostgreSQL are very conservative; we can safely change some of them and improve performance.

Change the kernel parameter for maximum shared memory segment size to 128Mb:

[root root]# echo 134217728 >/proc/sys/kernel/shmmax
[root root]#
Make that change permanent by editing /etc/sysctl.conf to add these lines at the end:

# increase shared memory limit for postgres
kernel.shmmax = 134217728
Edit the PostgreSQL config file, /usr/local/pgsql/data/postgresql.conf, to use more memory. These values should improve performance in most cases. (more information)

#      Shared Memory Size
#
shared_buffers = 15200      # 2*max_connections, min 16

#      Non-shared Memory Sizes
#
sort_mem = 32168            # min 32

#      Write-ahead log (WAL)
#
checkpoint_segments = 3    # in logfile segments (16MB each), min 1
___________________________________________________________

Collapse
2: Re: Problem (response to 1)
Posted by Richard Hamilton on
Just as follow up. Reading the reference given in the docs, I think that the settings that we are recommending are a bad idea. The reference given describes how more is not necessarily better - performance can drop off again if settings are too high.

I reckon that on a server with 1GB of RAM we should need no more than 8192KB for shared_buffers.

The setting recommended for sort_mem is I think way too high. The info below states clearly that this is per sort, so you could very quickly clog up the machine with sorting for a small number of users if set like this.

I think that 2048KB for sort_mem (per sort, per thread) should be adequate unless you are running a server with a very small number of users who run really huge sort intensive queries.

Also, it seems to me that the parameter that we really need to give some guidance on is the effective_cache setting.

Regards
Richard
________________________________________________________

Start at 4MB (512) for a workstation
Medium size data set and 256-512MB available RAM: 16-32MB (2048-4096)
Large dataset and lots of available RAM (1-4GB): 64-256MB (8192-32768)
PLEASE NOTE. PostgreSQL counts a lot on the OS to cache data files and hence does not bother with duplicating its file caching effort. The shared buffers parameter assumes that OS is going to cache a lot of files and hence it is generally very low compared with system RAM. Even for a dataset in excess of 20GB, a setting of 128MB may be too much, if you have only 1GB RAM and an aggressive-at-caching OS like Linux.

There is one way to decide what is best for you. Set a high value of this parameter and run the database for typical usage. Watch usage of shared memory using ipcs or similar tools. A recommended figure would be between 1.2 to 2 times peak shared memory usage.

2.2 Sort memory
This parameter sets maximum limit on memory that a database connection can use to perform sorts. If your queries have order-by or group-by clauses that require sorting large data set, increasing this parameter would help. But beware: this parameter is per sort, per connection. Think twice before setting this parameter too high on any database with many users. A recommended approach is to set this parameter per connection as and when required; that is, low for most simple queries and higher for large, complex queries and data dumps.

2.3 Effective Cache Size
This parameter allows PostgreSQL to make best possible use of RAM available on your server. It tells PostgreSQL the size of OS data cache. So that PostgreSQL can draw different execution plan based on that data.
Say there is 1.5GB RAM in your machine, shared buffers are set to 32MB and effective cache size is set to 800MB. So if a query needs 700MB of data set, PostgreSQL would estimate that all the data required should be available in memory and would opt for more aggressive plan in terms of optimization, involving heavier index usage and merge joins. But if effective cache is set to only 200MB, the query planner is liable to opt for the more I/O efficient sequential scan.

While setting this parameter size, leave room for other applications running on the server machine. The objective is to set this value at the highest amount of RAM which will be available to PostgreSQL all the time.

Collapse
3: Re: Problem (response to 1)
Posted by Torben Brosten on
Good points, Richard.

The tuning section is optional. I had to use much different numbers and different memory tuning format when installing on a freebsd with 256mb.

I plan to update some info here, including links to the postgres and various related os tuning pages which are helpful for tuning, since there are numerous environmental factors that make this somewhat unique. I have to get a local cvs setup first.

Collapse
4: Re: Problem (response to 3)
Posted by Jeff Davis on
I have seen something like 8-15% of physical memory for shared buffers on the pgsql-performance list. As for sortmem, I think 2mb is probably too small unless you are running a really small site or are sharing the server with a lot of other aolserver instances. It's probably really hard to make a firm recommendation for sort_mem, but I would err on the side of making it a little to large and setting it lower if you see the server thrashing.

Keep in mind that what happens of you have queries that go over the sort_mem size they use an on disk sort (or use queru plans which do table scans rather than has joins for example) which can have a tremendous negative impact on performance as well.

Collapse
5: Re: Problem (response to 1)
Posted by Jade Rubick on
Richard, thanks for looking into this.

Can you post a diff of the .xml file for the tuning section? Or post the changes yourself?

Collapse
6: Re: Problem (response to 5)
Posted by Richard Hamilton on
I'm certainly happy to do that but not sure how. If there are some instructions on how to do that I will follow them or alternatively shall I just hand edit the relevant file in /docs and then email it?
Collapse
7: Re: Problem (response to 6)
Posted by Jade Rubick on
Richard,

I added a section to the FAQ to answer your question (and anyone else who wants to contribute to the documentation)

https://openacs.org/faq/one-faq?faq_id=161579#197337

--------------------------------

Can someone give me permission to edit this page?

https://openacs.org/projects/openacs/doc-project/

Collapse
8: Re: Problem (response to 7)
Posted by Richard Hamilton on
Thanks Jade. I will do it shortly.

Those general patch creating instructions will be extremely useful! Thank you.