Forum OpenACS Development: Tcl Profiling

Though it may come as a surprise, I'm still actively doing lots of things on OpenACS (or dotCommunity as we like to call our branch of it!).

I've recently been doing some load testing of Surrey's Olympics website which is a subsite on one of our OpenACS installations - www.gosurrey.info.

They want to be able to handle a *big* increase in traffic around Olympics time, so I've been doing some load testing using Tsung (which is excellent, by the way!).

By cranking up the CPU power on the virtual server that runs it I can get it to handle about 300 visitors per minute (where each visitor loads a few pages). That's still some way off the ideal target.

It's entirely CPU bound - the db and files are all in RAM. What I found reasonably surprising is that most of the CPU time is nsd, not in postgresql.

So, I was wondering if anyone had any success in profiling the Tcl side of things recently? I have found various old discussions on using tclprof and nsprofile but nobody seemed to have much success.

I imagine it'd be possible to hook directly into ad_proc, but the overhead of adding timing to *every* procedure call might make everything unbearably slow!

I promise to post results here if anyone can give me any clues 😊

Thanks, and happy new year!

Joe

You may find the cheapest / simplest way to improve performance is to put images and .js files on either a CDN or a separate machine with e.g. nginx ; and, put nginx in front of your AOLserver to keep the nsd from tying up threads serving slower clients.

Dear Joe,

Handling the load you are describing (300 * 5 -> 1,500 views per minute) is not too much (we are experiencing on our production site sometimes more than 10,000 views per minute, practically all of these are dynamic), but hits/views are very hard to compare, since the computational effort per request can be dramatically different.

First of all, CPU-bounded-ness is typically no big problem for aolserver/naviserver, when you have multiple cores. I would recommend to configure 2-4 connection threads per core for good scaling. If you cannot use multiple cores due to your virtual server, consider moving to the real machines - quad core (or even 8 core) machines are a commodity today.

Additionally, using a reverse proxy (such as nginx) for static content and baground delivery [1] will help you additionally to scale up.

Going towards high performance without serious monitoring is hardly possible, since every server is a fast as is weakest component. I am sure, you are aware of the options and tools, such as

• aolserver/naviserver specific tools xotcl-request monitor (statistics from actual to daily values) and nsstats.tcl (cache and lock statistics).
• db-tools monitoring (e.g. postgres statistics, both in-server statistics and log analyzing tools) and
• system monitoring tools (like e.g. hotsanic, symon, ...)

From that monitoring one might find out that e.g. you are just able to use 2 of the 8 cores, etc. This information is important for configuring aolserver (e.g. connection threads, db-handles, ns-cache sizes, ...), that a few SQL queries slow everything down, etc. Tuning the aolserver configuration and the server can improve scaling significantly.

Coming now to your question - Tcl profiling:

There are at least 2 approaches:

1. dtrace: newer Tcl versions have support for dtrace provider. This allows you to turn on/off various "probes" from an external program .... provided you have a platform supporting dtrace (or access to such a platform to run your tests)
   See [2]

2. the next scripting framework (child of xotcl) has among many features support for running tcl-procs with non-positional argument support (like ad-procs). We have a small extension to openacs that allows to map all ad-procs with non-pos args to that nsf-procs. We use since in production since may. By this move,
   • (1) the performance overhead per ad-proc invocation is greatly reduced (argument processing is done in C),
   • (2) the memory footprint shrinks (non of the automatically generated argument parsers are needed)
   • (3) one can use the built-in profiling support (when using xowiki, one can obtain even class diagrams with profiling information)
   For this, you need an fairly new OpenACS, Tcl 8.5, the next scripting framework installed, an a small patch to lat ad-procs use nsf-procs.
   See [3]

For giving more specific hints, one would need more information about you application, the used components, platform, etc. Anyway, hope this is still helpful
-gustaf neumann

[1] http://www.openacs.org/xowiki/openacs-performance-tuning
[2] http://wiki.tcl.tk/19923
[3] http://www.next-scripting.org/

Moving static files into the filesystem without permission checks is the best bet.

OR If the URLs match a consistent pattern, or you can write a URL handler for those type of content repository resources, you could proxy them with something like NGINX.

This might be the simplest solution. NOTE that the content repository does include a feature to publish content repository items to the filesystem specifically to remove the datbase overhead. If you published them to a /resources directory you'd get the additional benefit of no permission checks.

On top of the filesystem publish, proxying to remove all Tcl overhead, all /resources URL should give the maximum benefit.

If you put in a dedicated handler for those, you'd also do well to set expiry/cache-control headers on the response; how aggressive depends on your needs of course but most recommendations are pretty high (months or more).