Forum OpenACS Q&A: OpenACS clusters

Posted by Claudio Pasolini on
I'm just starting to test a cluster setup and I wonder if there is a recommended set of patches to apply or if the standard OpenACS code is enough.

I know that there are a few installations out there (the Valencia University posted one patch) that hopefully could give some advice.

My current setup consists of 5 virtual machines: one front-end (NGINX), two application servers connected to the PostgreSQL database server and one file server connected via NFS to the application servers.

The file server owns all the OpenACS files except the log and etc folders, residing on the application servers. Is there a better way to share the code?

Is it safe to share the content items into one content-repository-content-files folder?

The first benchmarks I made are not very exciting. Actually the development server (a real 4CPU server with 10 GB RAM) is faster than the cluster.

Every virtual machine is a blade of an IBM blade center with external SUN storage and the RAM is: FE 2GB, AS1 and AS2 4GB, DB 4GB and FS 2GB.

I will post here further experiences.

2: Re: OpenACS clusters (response to 1)
Posted by Andrew Piskorski on
This is somewhat off the topic of your questions, but...

The AOLserver clustering support in OpenACS seems like a useful addition for flexibility, but I'm not sure how relevent it really is for a current site with any sort of reasonable hardware budget.

Supermicro now sells 4-socket Opteron servers at no price premium over 1 and 2 socket boxes; they're priced essentially linearly with the number of sockets. That means you can easily buy an either 1u or 2u box with 4 sockets, 48 cores, and 256 GB of RAM for $15k or so. (You can go up to 512 GB RAM on that same 4-socket motherboard, it just gets pricier.) How many OpenACS sites need more than 48 cores and 256 to 512 GB of RAM for their web servers? My guess is that's plenty even for big sites like WU-Wien. Spend a little more on disks and possibly a good hardware RAID controller, and you could also use that same 2u box for your RDBMS.

For more disk-intensive RDBMS loads, particularly analytics rather than the more OLTP-like loads typical of most OpenACS sites, I'd like to try some of Scalable Informatics' JackRabbit and/or DeltaV boxes. They seem to give massive streaming IO bandwith, and if one 48 disk box isn't enough, they also offer them configured as a storage clusters, optionally with a high-end Infiniband interconnect.

For the high-end storage cluster route, I'm not sure if any of the current cluster file systems are suitable for running an RDBMS, but if not, it'd probably work reasonably well to have your single RDBMS box use each of those 48-disk boxes as a giant "local" RAID-10 array. You'd want to make sure the PCI-E bus(es) on your RDBMS box can handle the bandwith, though. At some point that single RDBMS box will top out, and to get even more CPU and/or disk IO, you'd need to either look at more specialized (and pricey) giant SMP/ccNUMA boxes, or real shared-nothing clustered databases - Teradata or their newer cheaper competitors (some of which are PostgreSQL based).

3: Re: OpenACS clusters (response to 2)
Posted by Claudio Pasolini on
I agree, Andrew, and if I could choose I would go the hardware way, but in this case I have to use the client infrastructure and try to take advantage of the OpenACS clustering.
4: Re: OpenACS clusters (response to 1)
Posted by Eduardo Santos on
Hi Claudio,

I'm working with Clusters for some time, and the adjustments are not as simple as it can look like. I've had different problems, and you have to change some OpenACS configurations so it works properly. First, take a look at this thread, where I've runned a lot of tests:

I'll try to resume some of my learnings here, and feel free to ask any further questions.

1 - If you can avoid it DO NOT use Virtual Machines. Most of the performance problems in Clusters are related to thread creation/destruction proccess, and this is mostly an I/O operation. When you get to the point where I/O is a bottleneck, VM's are bad because the disks are shared between these VM's. If you can't avoid this, try to setup one single disk for each VM. If there's no way you can do this, well, you are going to have performance issues.

2 - Cofigure Kernel parameters to Cluster so util_memoize cache works. This config can be a little anoying, but it's totally necessary.

3 - Avoid NFS if you can. Just remember OpenACS won't startup if you lose some of the basic dirs, such as content-repository. The best solution I could find was to setup an exclusive communication interface between servers (like a cross cable) and have a separated content-repository partition to every server. From times to times I use an rsync to sync the files on both servers. If you have a storage or anything, it would be better.

These are the first things I can think about now. Let me know if you have any other questions.

5: Re: OpenACS clusters (response to 4)
Posted by Claudio Pasolini on
Thank You for answering, Eduardo, and for pointing me to a very useful resource.

Unfortunately I can't avoid using Virtual Machines: this is a customer decision.

Actually my absolute priority is to make things work and only then worry about performance.

The first concern is about the cache management. I already configured the Cluster kernel parameters and set to '1' PermissionCacheP, but I wonder if this is enough or if I should apply some patch to my standard OpenACS 5.5.

The second concern is about the file-storage which is configured to store files into the file system. Actually the content-repository-content-files folder is shared by the application servers via NFS and I wonder if some conflict could arise.

I don't understand how I could setup 'an exclusive communication interface between servers': can you kindly elaborate more?

6: Re: OpenACS clusters (response to 5)
Posted by Eduardo Santos on
Hi Claudio,

I know using VM's is usually a customer decision, but you will see some problems in the future. Just to let you know. :)

Take a look at this thread about caching:

If you configure the parameters correctly, you should see no problems.

About the file-storage, there's no problem about storing it on files; most people do it. there should be no problem to share it with the servers, because the folders are controlled by PostgreSQL. As long as the system access the same database, you will be ok.

About the exclusive communication, I have a cross cable plugged on servers ethernet interface just to replicate content-repository folders. My experience is that I/O operations should be isolated from user access operations, so should be PostgreSQL access. There should be no large TCP/IP queue on this configuration.

Hope it helps

7: Re: OpenACS clusters (response to 6)
Posted by Gustaf Neumann on
On of the major strengths of OpenACS + aolserver/naviserver is to exploit the power of multi-core and multi-processor architectures. AOLserver was pioneering in this regard and is still at least among the best in this category. Since multi-core architectures are already a commodity, and substantial multi-core systems (32+ cores) are becoming cheap (see Andrews posting above), these are actually good times for this architecture.

But certainly, there is as well the trend towards virtualization, and in most cases, a VM boils down to a machine with a single core (from the hosted OS point of view). Such a setup will not be be able to use the scalability strength of the server. The situation will not improve significantly by the cluster setup, which requires substantial inter-cluster communication, which is currently built on top of HTTP. Problem areas are the caches and nsvs. I am pretty sure, the cluster support can be significantly improved, but i am not sure, this is worth the effort. Virtualization on its own is not the problem, but virtual machines with single cores are bad, as well as missing transparency of the bottlenecks of the vm configuration (e.g. I/O) and missing knowledge about the influence of other processes running on the same hardware.

We are using in production as well a virtualized environment (p570, POWER6) but our virtual machine (called lpar in the IBM world) has for our production server 64 logical CPUs. This machine is not in the the price range of the Opteron machines that Andrew mentioned, but scales perfectly and it allows even to move running VMs from one physical hardware to another one if necessary (requires an appropriate storage architecture, etc.). After the fire last year we had to squeeze 10 of our servers to one smaller physical machine; this worked very well, even with the 1600 concurrently active users, we had at that time.

In short, for heavy load applications, i would not recommend to use a cluster of single-cpu VMs. Remember, there was a time when was very slow and unreliable. This was at the time when it was running on a vm with a single CPU. Many of us (and the oct) were at that time not able to pinpoint and eliminate the problem. Interestingly enough, the problem was not there on a much older and slower hardware before, and disappeared, when the server moved to Vienna in June 2008.

8: Re: OpenACS clusters (response to 1)
Posted by Claudio Pasolini on
I did some simple tests and the cache flushing mechanism between the application servers works well, as long as the application calls the util_memoize_flush proc.

I'm surprised that changing a package parameter doesn't call the flushing proc: perhaps this is a feature and not a bug, but eventually this could be easily fixed.

I also played loading, retrieving and deleting attachments to my application objects, so hitting the content-repository-content-files folder, apparently without any error.

I want to thank everyone for giving their advice and sharing their experience. Should the clustering fail for any reason, I'm prepared to downgrade to a proven setup, with one DB server and one AS.

Meanwhile, thanks to some tuning on the VMWare side and 1 GB more on every AS, the cluster performance is now comparable to that of the physical development server, but at the same time gives me the perspective to be more scalable.

9: Re: OpenACS clusters (response to 8)
Posted by Eduardo Santos on
Hi Claudio,

I'm glad everything is working ok. The problems you can see are going to happen when the first threads die and the system tries to create them again, usually a situation you can see only in production environments.

Keep an eye on possible memory leaks and make sure you use Gustaff's patch for background delivery. This is the feature in OpenACS that makes the best performance improvement so far.

Best regards

10: Re: OpenACS clusters (response to 9)
Posted by Maurizio Martignano on
Dear Eduardo,
I like the part where you mention "memory leaks"...
All the mechanism described by Claudio is exactly a way of coping with the memory leaks present in web/application servers and offer anyhow to the end customers highly available systems. This is better described here:
12: Re: OpenACS clusters (response to 1)
Posted by Armando Garcia on

I also have been working with the openacs cluster.
I have the next configuration.
One instance "A" running in the port 8000 another one "B" in the port 8002. Both use the same database in postgresql.

Also i have dotlrn running in this configuration.

The problem is when i create a new community "Test" in the instance "A", when i want to access to "Test" community in "B" instance i get the following error "The requested URL was not found on this server."

I configured the parameters for the cluster in package "acs-kernel". After some problems i got it working and in the log i can see something like:

[05/Sep/2011:18:28:01][28048.3070536592][-conn:openacs3::3] Notice: Distributed flush of portal::element_params_not_cached 6839
[05/Sep/2011:18:28:01][28048.3069483920][-conn:openacs3::4] Notice: Distributed flush of dotlrn_community::get_site_template_id_not_cached -community_id 6553
[05/Sep/2011:18:28:01][28048.3068431248][-conn:openacs3::0] Notice: Distributed flush of permission::permission_p_not_cached -party_id 1081 -object_id 6553 -privilege admin
[05/Sep/2011:18:28:01][28048.3072641936][-conn:openacs3::1] Notice: Distributed flush of dotlrn_community::get_dotlrn_master_not_cached -community_id 6553
[05/Sep/2011:18:28:01][28048.3071589264][-conn:openacs3::2] Notice: Distributed flush of sec_lookup_property 1 dotlrn which_tab_selected
[05/Sep/2011:18:28:01][28048.3070536592][-conn:openacs3::3] Notice: Distributed flush of sec_lookup_property 1 dotlrn home_tab_selected_p

I understand that the instance "B" has gotten the request from "A" to flush his cache. But when i try to get in to the community "Test" I have the same error.

Any advice?

My best regards

13: Re: OpenACS clusters (response to 12)
Posted by Eduardo Santos on
Hi Armando,

Just one question before we address your cluster environment: are you running two AOLServer instances on the same box in different ports? Because this is the worst case scenario for AOLServer.

Posted by Armando Garcia on
Hi Eduardo.

Yes, for testing purpose i am running two instance of openACS on the same installation of AOLServer, i only change the port for each instance.
The final scenario will be to run this on different servers.

15: Re: OpenACS clusters (response to 14)
Posted by Eduardo Santos on
Hi Armando,

When you are testing a cluster scenario you need to use a setup like the real one, because some problems you can face are related to the configuration, not directly to OpenACS.

Concerning OpenACS, I can tell you in first place that the scenario you've created (more than one instance on the same box) won't work. The cluster setup is done for more than one server, and everything is designed to work on this situation.

My advice to you is to abandon this setup and try to get as close as you can to the reality.

16: Re: OpenACS clusters (response to 15)
Posted by Armando Garcia on
Hi Eduardo.
We have set up the cluster using two separate servers and we are still getting the same error.

I have even tried doing a manual flush via the hyperlink "Flush" located in Developer's Administration : Cache Control. (http://x.x.x.x:8000/acs-admin/cache/flush-cache?suffix=util_memoize) and we are still getting the message "The requested URL was not found on this server." when trying to access the "Test" community from server B

17: Re: OpenACS clusters (response to 16)
Posted by Eduardo Santos on
Hi Armando,

The cache flush on Cluster is managed by acs-kernel parameters. The basic work is simple: if cluster is being used and you call for something in an util_memoize cache, the server is going to try it locally. If the requested item is not found on local server, the system send an HTTP request to the other server and tries to retrieve that cached item.

You can see this behaviour in your log files if you set the parameter EnableLoggingP to 1. The file that gets the cached item is at the path server_root/www/SYSTEM/flush-memoized-statement.tcl

I'm assuming that you have configured all the acs-kernel parameters the way they should. Anyways, it doesn't hurt to double check the configuration here:

- CanonicalServer: In the list of IPs above, which is the canonical (primary) server? If a port is not listed, we assume port 80.

- ClusterAuthorizedIP: a space separated list of which machines can issues requests (e.g., flushing) to the cluster. Can use glob matching notation (10.0.0.*)

- ClusterEnabledP: is clustering enabled?

- ClusterPeerIP: A space separated list of servers in the cluster. This server's IP may be included too.

You also have to make sure all the servers access the same database (it's obvious, but it doesn't hurt to say either) and make sure your network don't block communication between the servers on needed ports.

If everything above is done, then your cluster has another problem that do not rely on standard configurations. Let's try to see if we can get to your error. What you are doing is:

1 - Go to /dotlrn/admin/club-new and add a new community on server A;
2 - Set it as an open community;
3 - Go to community page (/dotlrn/clubs/test) on server A;
4 - Try to access the same page on server B.

Is it the activity sequence you are following?

18: Re: OpenACS clusters (response to 7)
Posted by Mark Aufflick on
I agree with everyone about using hardware with AOLServer - it flies.

I am, though, in the progress of moving some small openacs sites from a (very old, and still fast) physical server to linode which is going pretty well so far.

I thought people might be interested to know that linode VMs are multicore:

mark@li165-84:~$ cat /proc/cpuinfo |egrep 'processor|model name'
processor   : 0
model name  : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
processor   : 1
model name  : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
processor   : 2
model name  : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
processor   : 3
model name  : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
Of course this is all shared, but with the right plan and assuming the right level of contention, you are getting benefits from the aolserver threading.