Forum OpenACS Q&A: OpenACS and Load Balancers

Collapse
Posted by Steve Manning on
Were looking to implement a layer 4 load balancer (probably a Baracuda LB Model 440 supplied by the ISP) in front of our OACS 5.3.2 servers for both HTTP and HTTPS requests to spread the load and gain some resilience.

Just wondering if there are there any pitfalls of which we need to be aware?

TIA

Steve

Collapse
Posted by Mark Aufflick on
Not a very timely reply, but this might help.

Assuming you already realise that you need to use a single backend database ;) the question is going to be about nsv and similar shared data. OpenACS makes use of AOLServer's excellent thread safe shared data structures. Obviously these are local to a server process.

You can mitigate this issue by using what most load balancers call "session affinity" which means that if the same person (identified by a cookie etc.) hits your site twice they will always get the same server (unless it goes down).

Or you could suck it and see, but being aware there would *definately* be code changes you would have to make.

Collapse
Posted by Patrick Giagnocavo on
Steve, there are 2 answers to this:

1. If you are going to be spending money anyways, buy the highest-end systems you can afford that have the most memory bandwidth.

Then run the front-end AOLserver on Solaris 10 or OpenSolaris (best multi-threading, better tuned VM system, best chance to take advantage of memory via e.g. MPO).

If you really want to spend, use OpenSolaris in combination with the flash drive (SSD) based read and write cache capability of newer versions of ZFS.

Put Pound or another SSL capable proxy (nginx, lightppd) on a separate machine to handle SSL negotiation, setup, teardown, etc. Put gigabit ethernet links with jumbo frames turned on, between Pound and the AOLserver machine. If you need even more bandwidth you can combine gigabit links (link aggregation) to get 2, 3, or 4 Gbps.

Architecture thus looks like this:

for both SSL and non-SSL:
Internet -- Pound Web/SSL proxy -- AOLserver front end -- database server

Cost: $2K for proxy, $5K for system with SSDs, $8K for database server with more SSDs (like an 8-SAS or SATA drive system); so $15K and 3-4U of rack space to get started (if you are handling a lot of data you will need to add more rack space for your storage system).

Note that in such a situation you still have single points of failure; but with the SSDs you will have the performance you need. In my experience, the main failures are the power supplies , so specify systems that have dual power supplies.

2. The other alternative is to use the nsv_* procs to ensure that the things that matter are shared amongst multiple front-end servers and that the cache is invalidated when it should be. I think that Malte S. spent some time on this, but I don't know the status of his work on it.

#2 will give you more resilience but will require more programming and testing.

Collapse
Posted by Steve Manning on
Thanks to both for that reply. We've actually moved off the load balancer approach for now, the new server is giving us 4x the throughput in our capacity tests so were going with that for now.

We have implemented a warmstart database backup use WAL shipping which is due to go live next week, that gives us better resilience on the DB and the old server as a hot backup to the app server in case of problems there.

The idea is not to spend the money this year (who is?) but to see what the growth in traffic is like. Once we get through the busy period - Oct last year peaked at 28mil reqs/day, we'll then be planning for Oct 2010 and probably the load balanced approach.

FYI The new server is running Centos 5.3 64bit on 2 x Quad Core Zeons (with HT this gives 16 cores seen by the O/S), 36Gb RAM and 8 x 145Gb HDDs split into a RAID 1+0 volume for the DB and a RAID 5 vol for the O/S with one spare for hot swap.

- Steve

Collapse
5: Re: OpenACS infrastructure (response to 3)
Posted by Mark Aufflick on
Patrick - do you have any measurements (or anecdotal evidence) of OpenACS/aolserver performance improvements on OpenSolaris?

Also, what sort of percentage load reduction do you see by offloading ssl onto another box? If you really want SSL to fly you could also consider a hardware ssl acceleration card on the proxy box with a supported proxy/web server.