Forum OpenACS Q&A: OpenACS clusters

The AOLserver clustering support in OpenACS seems like a useful addition for flexibility, but I'm not sure how relevent it really is for a current site with any sort of reasonable hardware budget.

Supermicro now sells 4-socket Opteron servers at no price premium over 1 and 2 socket boxes; they're priced essentially linearly with the number of sockets. That means you can easily buy an either 1u or 2u box with 4 sockets, 48 cores, and 256 GB of RAM for $15k or so. (You can go up to 512 GB RAM on that same 4-socket motherboard, it just gets pricier.) How many OpenACS sites need more than 48 cores and 256 to 512 GB of RAM for their web servers? My guess is that's plenty even for big sites like WU-Wien. Spend a little more on disks and possibly a good hardware RAID controller, and you could also use that same 2u box for your RDBMS.

For more disk-intensive RDBMS loads, particularly analytics rather than the more OLTP-like loads typical of most OpenACS sites, I'd like to try some of Scalable Informatics' JackRabbit and/or DeltaV boxes. They seem to give massive streaming IO bandwith, and if one 48 disk box isn't enough, they also offer them configured as a storage clusters, optionally with a high-end Infiniband interconnect.

For the high-end storage cluster route, I'm not sure if any of the current cluster file systems are suitable for running an RDBMS, but if not, it'd probably work reasonably well to have your single RDBMS box use each of those 48-disk boxes as a giant "local" RAID-10 array. You'd want to make sure the PCI-E bus(es) on your RDBMS box can handle the bandwith, though. At some point that single RDBMS box will top out, and to get even more CPU and/or disk IO, you'd need to either look at more specialized (and pricey) giant SMP/ccNUMA boxes, or real shared-nothing clustered databases - Teradata or their newer cheaper competitors (some of which are PostgreSQL based).

I'm working with Clusters for some time, and the adjustments are not as simple as it can look like. I've had different problems, and you have to change some OpenACS configurations so it works properly. First, take a look at this thread, where I've runned a lot of tests: http://www.openacs.org/forums/message-view?message_id=1539800

I'll try to resume some of my learnings here, and feel free to ask any further questions.

1 - If you can avoid it DO NOT use Virtual Machines. Most of the performance problems in Clusters are related to thread creation/destruction proccess, and this is mostly an I/O operation. When you get to the point where I/O is a bottleneck, VM's are bad because the disks are shared between these VM's. If you can't avoid this, try to setup one single disk for each VM. If there's no way you can do this, well, you are going to have performance issues.

2 - Cofigure Kernel parameters to Cluster so util_memoize cache works. This config can be a little anoying, but it's totally necessary.

3 - Avoid NFS if you can. Just remember OpenACS won't startup if you lose some of the basic dirs, such as content-repository. The best solution I could find was to setup an exclusive communication interface between servers (like a cross cable) and have a separated content-repository partition to every server. From times to times I use an rsync to sync the files on both servers. If you have a storage or anything, it would be better.

These are the first things I can think about now. Let me know if you have any other questions.

I know using VM's is usually a customer decision, but you will see some problems in the future. Just to let you know. :)

Take a look at this thread about caching: https://openacs.org/forums/message-view?message_id=393986

If you configure the parameters correctly, you should see no problems.

About the file-storage, there's no problem about storing it on files; most people do it. there should be no problem to share it with the servers, because the folders are controlled by PostgreSQL. As long as the system access the same database, you will be ok.

About the exclusive communication, I have a cross cable plugged on servers ethernet interface just to replicate content-repository folders. My experience is that I/O operations should be isolated from user access operations, so should be PostgreSQL access. There should be no large TCP/IP queue on this configuration.

Hope it helps

I'm surprised that changing a package parameter doesn't call the flushing proc: perhaps this is a feature and not a bug, but eventually this could be easily fixed.

I also played loading, retrieving and deleting attachments to my application objects, so hitting the content-repository-content-files folder, apparently without any error.

I want to thank everyone for giving their advice and sharing their experience. Should the clustering fail for any reason, I'm prepared to downgrade to a proven setup, with one DB server and one AS.

Meanwhile, thanks to some tuning on the VMWare side and 1 GB more on every AS, the cluster performance is now comparable to that of the physical development server, but at the same time gives me the perspective to be more scalable.

I also have been working with the openacs cluster.
I have the next configuration.
One instance "A" running in the port 8000 another one "B" in the port 8002. Both use the same database in postgresql.

Also i have dotlrn running in this configuration.

The problem is when i create a new community "Test" in the instance "A", when i want to access to "Test" community in "B" instance i get the following error "The requested URL was not found on this server."

I configured the parameters for the cluster in package "acs-kernel". After some problems i got it working and in the log i can see something like:

[05/Sep/2011:18:28:01][28048.3070536592][-conn:openacs3::3] Notice: Distributed flush of portal::element_params_not_cached 6839
[05/Sep/2011:18:28:01][28048.3069483920][-conn:openacs3::4] Notice: Distributed flush of dotlrn_community::get_site_template_id_not_cached -community_id 6553
[05/Sep/2011:18:28:01][28048.3068431248][-conn:openacs3::0] Notice: Distributed flush of permission::permission_p_not_cached -party_id 1081 -object_id 6553 -privilege admin
[05/Sep/2011:18:28:01][28048.3072641936][-conn:openacs3::1] Notice: Distributed flush of dotlrn_community::get_dotlrn_master_not_cached -community_id 6553
[05/Sep/2011:18:28:01][28048.3071589264][-conn:openacs3::2] Notice: Distributed flush of sec_lookup_property 1 dotlrn which_tab_selected
[05/Sep/2011:18:28:01][28048.3070536592][-conn:openacs3::3] Notice: Distributed flush of sec_lookup_property 1 dotlrn home_tab_selected_p

I understand that the instance "B" has gotten the request from "A" to flush his cache. But when i try to get in to the community "Test" I have the same error.

Any advice?

My best regards

Yes, for testing purpose i am running two instance of openACS on the same installation of AOLServer, i only change the port for each instance.
The final scenario will be to run this on different servers.

I have even tried doing a manual flush via the hyperlink "Flush" located in Developer's Administration : Cache Control. (http://x.x.x.x:8000/acs-admin/cache/flush-cache?suffix=util_memoize) and we are still getting the message "The requested URL was not found on this server." when trying to access the "Test" community from server B

I am, though, in the progress of moving some small openacs sites from a (very old, and still fast) physical server to linode which is going pretty well so far.

I thought people might be interested to know that linode VMs are multicore:

mark@li165-84:~$ cat /proc/cpuinfo |egrep 'processor|model name'
processor   : 0
model name  : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
processor   : 1
model name  : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
processor   : 2
model name  : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
processor   : 3
model name  : Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz