Forum OpenACS Development: cache statistics

Collapse
Posted by Malte Sussdorff on
Can we have a statistic for all variations of cache OpenACS is using (like site-node cache or cache for I18n), just as we have for util_memoize.

At the moment I am astonished that our AOLserver grabs around 3GB of memory, not sure why, but we restart it every night. Havent gone to 4.5 though yet (still running 4.0.10 and no, no TCL leak (aka we use 8.4.13)

Collapse
2: Re: cache statistics (response to 1)
Posted by Gustaf Neumann on
The "caches" for the site-map and localization messages use nsv. The only nice cache statistics are implemented on the c-level for ns_cache. In addition to nsv and ns_cache, we have multiple stores with somewhat similar properties but different interfaces and c level support (also: ns_section/ns_param, the per-thread stores ns_set, Tcl variables, associative arrays (* namespaces)).

So, while it is possible to measure the amount of memory that is put into these stores, it is not possible (without significant C programming) in the general case to figure out, what this content consumes per store (e.g. storing one byte in a tcl variable in a namespace needs significantly more than one byte, since it needs the variable structure, variable name, tcl_obj, maybe a new namespace with its hash tables, ...). Furthermore, a major problem with the current aolserver memory allocator is that it is optimized for high concurrency, but it tends to fragment easily after some time, which increases the memory footprint and reduces the hardware cache efficiency.

How busy is your site with the 3GB memory footprint? Is it an opteron machine? what's your maxthreads and your threadtimeout? Memory consumption is directly related to the number of threads (connection threads and scheduled threads), since every thread keeps a private copy of the tcl procs (+ vars). In a typical dotlrn instance we have are about 8000 procs. So, if one has e.g. 50 threads (e.g. 40 connection threads + 10 scheduled), this makes 400.000 procs....

Collapse
3: Re: cache statistics (response to 2)
Posted by Malte Sussdorff on
I feared there would not be an easy answer to the cache view.

The site has minthread of 30 and maxthreads of 40, as the company has only 35 employees, though most of them work all the time.

Prozessor is a Xeon 2.8GhZ. Debian Sarge System, total memory 6GB, though I did not recompile the kernel yet to make Linux aware of the 2 additional Gb (it currently only sees 4GB).

I assume we have a couple more procs, as we have dotlrn plus contacts/ams/project-manager/invoices .... installed, so this could explain some things...

Did you find a way around this problem?

The fragmentation is probably also a reason as we do not run into serious out of memory problems (aka: writes to disk), especially as we restart the server every night, but the footprint is still high and I'm not sure if AOLserver is still efficient with this high a footprint.

Collapse
4: Re: cache statistics (response to 1)
Posted by Dave Bauer on
Malte,

You can get a general idea of the size of the NSVs just by calculating the total string lengths of it, but obviously it won't be perfect. In general I think the NSVs are not that big, Jeff did some calculations before and it was not that much.

If minthreads is < maxthreads you are killing and restarting threads while the server is running, and that will increase your RAM consumption steadily. Make them equal.

I seriously doubt you need 40 threads for 35 concurrent users anyway, but only testing could prove that.

Collapse
5: Re: cache statistics (response to 4)
Posted by Rocael Hernández Rizzardini on
Equal min and max threads leads us to continuos reboots (aolserver dying), probably due threads that never die that had errors on it.
Collapse
9: Re: cache statistics (response to 5)
Posted by Dave Bauer on
Rocael, continuous as in immediate reboots? How often? Do you need to reboot it manually, or does it crash and restart?

As far as I know, you should ALWAYS use maxthreads == minthreads and that has been the "official" advice for a very long time.

Collapse
10: Re: cache statistics (response to 9)
Posted by Rocael Hernández Rizzardini on
Dave, I don't think that applies any longer. Any intensively used site will see crashes if you set them equal, how often is variable, but no more than one hour usually. We have threadtimeout set to 1200.
Collapse
12: Re: cache statistics (response to 10)
Posted by Dave Bauer on
Rocael,

What version of AOLserver are you using.

Collapse
13: Re: cache statistics (response to 12)
Posted by Rocael Hernández Rizzardini on
4.0.10 branch.
Collapse
6: Re: cache statistics (response to 4)
Posted by Malte Sussdorff on
Dave said:

I seriously doubt you need 40 threads for 35 concurrent users anyway, but only testing could prove that.

My assumption was I should calculate one thread per concurrent user and have a little bit of space for externals that access the site occasionally. Wrong?

How could I test this?

Collapse
7: Re: cache statistics (response to 6)
Posted by Patrick Giagnocavo on
Well, one way to test it would be to calculate how long a page takes to be generated. For instance, 10 threads which serve pages that are generated in 0.5 seconds can thus serve 20 pages per second.

However, on top on that, you have the fact that most users take a while to read each page. So, you need to calculate page loads per minute.

So our 10 threads above can serve 20 pages per second * 60 = 1200 pages per minute. If you have longer page generaton times, you need to adjust accordingly.

The most threads I have seen on any server is 35 threads. This application served some 20,000 to 35,000 users per week doing a multistep data entry/questionnaire application.

Collapse
8: Re: cache statistics (response to 1)
Posted by Gustaf Neumann on
our AOLserver grabs around 3GB
...
Prozessor is a Xeon 2.8GhZ

Hmm, this does not fit to my picture. All nsd threads are sharing 2GB on 32bit linuxes, therefore i have certain doubts, that the 3GB are true. How did you measure this?

The site has minthread of 30 and maxthreads of 40, as the
company has only 35 employees..

This looks to much. we have on our production site (up to about 1300 concurrent users) min threads not set and maxthread to 40. under easy load (300 concurrent users), we have between 20 and 30 threads maximum active.

We have the threadtimeout set to 300.

If minthreads is < maxthreads you are killing and
restarting threads while the server is running, and that
will increase your RAM consumption steadily. Make them equal.

This is not our experience. This might have been true with older versions of the aolserver, but seems fixed in newer generations. We see as well a growing memory consumption over time, but my interpretation is that this is due to fragmentation.

Per CPU you would not be able to handle more than about 4 active jobs (cpu intensive). Unless you have many deliveries of
large files over slow lines (which i doubt from your description), i would say that 20 connection threads are plenty already. For handling the file-deliveries, i would recommend the background file delivery based on libthread, which does not block connection threads.

You don't need per user a thread (how could we be able to handle 1300 users?). A thread can and will be used by many users, as every request occupies a thread for only a short time. This amount of time certainly depends on what the thread is doing (e.g. handling a .css file or doing a series of database queries).

In the xotcl-request monitor, you can see the under request-monitor/running the concurrently running jobs. I would assume with your 35 users, you will seldomly see more than 2 or three of these.

If you assume, that a page view takes 0.5 seconds on overage, an the user takes 1 second on viewing the page, an active user can generate about 40 views per minute. So, 30 users can generate 1200 views per minute, getting the value of Patrick mentioned above.

As a comparison, we have currently (semester not started) 325 users, an average page view response time of 0.11 seconds and we had in the last hours between 500 and 800 page views per minute (not counting static files like .css and images). Our peaks are around 3000 views per minute.

So, 1200 page views per minute looks like a challenge for 30 users. I would recommend to check out your usage by installing the xotcl request monitor and to monitor the actual views per minute on that site.

Collapse
11: Re: cache statistics (response to 8)
Posted by Rocael Hernández Rizzardini on
Gustaf,

> For handling the file-deliveries, i would recommend the
>background file delivery based on libthread, which does not
>block connection threads.

do you have something working for this?

Collapse
14: Re: cache statistics (response to 11)
Posted by Gustaf Neumann on
Sure, we use this in production since more than one year. For example today (no semester yet) we had so far 150.000 file deliveries.

The asynchrounous background delivery requires a small patch (2 changes, one is a backport from naviserver) and the tcl thread library (by zoran). The application code is in XOTcl is only a few lines of code and is included in xotcl-core (bgdelivery-procs.tcl).

One needs the following patch to aolserver 4.0.10
http://media.wu-wien.ac.at/download/aolserver-nsd-conn.patch

with this patch and xotcl-core, you can replace

ns_returnfile 200 $mime_type $filename

by

ad_returnfile_background 200 $mime_type $filename

e.g. in cr_write_content in acs-content-repository/tcl/revision-procs.tcl
to activate it and to deliver files from the content-repository (file-store) in the background.

The connection thread is only used for permission management, localization of the file and writing the the reply header, the actual delivery of the file is performed via asychronous io without using up many resources. This can handle probably a couple of thousand concurrent file deliveries without running out of resources.

Btw, the streaming variety of the xowiki chat uses the same infrastructure to implement synchronous chats without polling.

That's cool stuff. Neophytos uses this on phigita.net since at least half a year and is as well very happy with it.

Collapse
Posted by Andrew Piskorski on
Gustaf, that sounds like an exceedingly useful patch. Did you ask for AOLserver cvs commit on SourceForge so you can add it there?
Collapse
Posted by Gustaf Neumann on
The patch is now in cvs head of aolserver at sourceforge.
Collapse
Posted by Jose Agustin Lopez Bueno on
Hi!

What about this patch on Aolserver 4.5?

Best regards,
Agustín

Collapse
Posted by Jose Agustin Lopez Bueno on
Hello!

Thanks for the patch, Gustaf.
I have already applied it to my installation.

Regards,
AGustin