Forum OpenACS Q&A: Memory usage - what is normal?

Collapse
Posted by Ken Chakiris on
I am running linux system using postgres beta3 and ASC/pg beta2.

When I run top and sort by memory I find that the nsd processes are
using 8M and postmaster has one process with 12M and the other
postmaster processes are 8M and 2M. This is a system with no users
accessing the ASC/pg except me for testing purposes. This is just
after restarting the AOL server and accessing ASC/pg a few times.

Is this memory usage normal for the usage level I am at?  How does the
memory usage scale up as the number of hits increases?

Collapse
Posted by Don Baccus on
How are you calculating that?  Are you including shared memory?
<p>
In the case of Postgres, there are three main consumers of memory:
<ul>
<li>The code - this is shared between the backend processes
<li>The shared buffer cache - again, shared between backends
<li>Temp storage - used for building hash tables for hash joins, to sort tables, to hold query returns, etc.
</ul>
The total memory used is the size of the code, the size of the shared buffer cache, and the sum of temp storage currently being used by all the backends.
<p>
In essence, for scaling up you want to decide how much of your machine  to allocate to the Postgres shared buffer cache (max 16mb without rebuilding your kernel) and how much space to allow Postgres to use for sorts, etc before it spills to disk (I'd make this value fairly large).  The maximum memory used will then be the size of the cache, the size of the image, the sum of the private space you allow for sorts, etc plus stack storage for each process (not much).
<p>
If you just blindly read the numbers from top without figuring out which are shared between processes, you'll think a lot more memory's being used than really is.
<p>
On the dual P450 I'm setting up as my production machine, I have 256 MB RAM.  I intend to devote 64MB to the shared buffer cache and to let  each backend use up to 8 or 16 MB for sorting, etc (I forget now just  how much I configured it for).  In reality, the chance that more than  one or two backends will be doing large sort/merge or hash joins at any one point in time is pretty low so I could set that figure to be larger, I imagine.  In a straight ACS site, the size of joins will be smaller on average than with my bird species distribution database.
<p>
AOLserver's less configurable AFAIK, but the code footprint is only a few megabytes.  It does caching, etc, too of course.
Collapse
Posted by Ken Chakiris on
Don,

I am more concerned with the nsd processes at the moment.  From reading Phil's book I always assumed that the database would eat up a lot of memory.

Each nsd process uses 8M of memory and has 1.3M of shared memory. This seems high unless I am doing something wrong. When I start up a non-ASC AOL server each nsd process uses something like 2M with 1.2 of shared memory.  Why are the ASC ones so much bigger?

Collapse
Posted by Don Baccus on
The ACS one is probably bigger because it spawns stuff on its on, i.e.  goes off to stuff the robot detection table, starts checking for things to spam users with, etc.  In other words, it's not quiescent even if you aren't doing anything with it yourself.

Since AOLserver's threaded, it doesn't have to use inter-process shared memory to cache stuff that's visible to each thread.  I'm not aware of how top or other linux utilities compute the various memory parameters for a threaded program.  Threads in Linux are kernel threads, implemented as a kind of light-weight process (this is why they're visible in the process table to top etc), and I've never dug into the details.

So I can't be of much help other than pointing out that the ACS rolls up its sleeves and gets to work right away when you fire it up.