Forum OpenACS Q&A: Kernel Parameter: PerformanceModeP and PermissionCache

PerformanceModeP should be enabled for production sites, it will speed request processing. The effect isn't huge, but it is worth doing. Do not do this for sites in which you're actively mounting and dismounting packages, unless you're willing to do a server restart after doing so.

PermissionCache is a bad idea used to speed .LRN. I think Sloan may be using it (they wrote it), but I recommend against it unless you really really really need it. It was added back when the permissions system was really slow, now that it's been speeded up the impact of permission caching is not so great. If PermissionCache is set, changes to an individual's permissions or their privilege in regard to an object may not be seen until the cache flushes (defaults to five minutes, I think). Parts of .LRN will flush the cache when permissions change, but underlying OpenACS code for the most part is unaware of the caching ... which was never meant to be a permanent solution.

Nima,

You need more data, specifically, there could be a few queries that are slow tying up the database, or your system just does not have enough RAM to handle the number of simultaneous database handles that are in use.

In some respects, you are asking the wrong question. The first question to ask is, what is the resource that is being used most heavily? Is it disk I/O, CPU usage, network bandwidth, or RAM/virtual memory? My guess is disk I/O or RAM.

Once you know what the bottleneck is, then you can figure out how to solve it.

Use iostat, sar, and vmstat to figure out what is going on. I have found sar to be useful, but you have to turn it on first and it is more of a "what happened" rather than "what is currently happening" tool.

I don't know what your logfile look like, but you could try to look for thread IDs and try to discover some more information.

Try cross-referencing the access.log files with the info in your error.log or server.log files - since they are timestamped, you may be able to see cases where a request took a long time to serve; and you could then look for clues in that part of the logfile.

That may or may not be worth your time, however.

Hi Nima,

you should log the queries in your postgresql instance. Then you can analyze the longest running/slowest queries. You can use the "Postgres Query Analyzer" to do that. get it here: http://pqa.projects.postgresql.org/

In the postgresql.conf file you have to activate:
# - What to Log -
log_duration = true
log_pid = true
log_statement = true
log_timestamp = true

I would also propose:
# - Query/Index Statistics Collector -
stats_start_collector = true
stats_command_string = true

Then you can interactively see the current running queries with "select * from pg_stat_activity ;"

In addition you should monitor aolserver thread usage with the telemetry page from the "nstelemetry" package (from aolserver.com). Maybe you have to few aolserver threads available (ns_section "ns/server/${server}" MaxThreads). With the telemetry page you can also find out how may page-request are currently in the wait queue of the aolserver.

In the proc "db_with_handle" add some code to find out how long you have to wait for available db-handles:

<snip>
set pool [db_nth_pool_name -dbn $dbn $db_state(n_handles_used)]
set start_time [clock clicks -milliseconds]
set errno [catch {
    set timer [clock clicks -milliseconds]
    set db [ns_db gethandle $pool]
    set duration [expr [clock clicks -milliseconds] - $timer]
    if { $duration > 100 } {
      ns_log Notice "Got db-handle from $pool after $duration MS"
    }
} error]
</snip>

Maybe you have to few db-handles available and therefor threads are hanging around waiting without doing anything. Keep in mind that you have to increase the available db-connections when you rise the number of aolserver threads.

Install some system statistic collector. I propose HotSanic, it generates nice html-stats you can link into your web-site. (get it here: http://hotsanic.sourceforge.net/). You can use it to find bottlenecks and monitor resource usage.

Last but not least i could provide you with a simple benchmarking application that starts aolserver threads and issues db-queries. requests are not coming from the outside of the server (via http) but from within, but it works quite nice to stress the server. It is written in xotcl and you would have to modify it for your needs, but the framework of starting threads, monitoring transactions, running custom transactions and the like should provide you with a start.

hope that helps, peter