Forum OpenACS Q&A: Response to Scaling, Redundancy, Multiple Machines...

Posted by Jeff Barrett on
Concerning the load balancer, I would suggest the use of an Alteon AD3, very nice product. Check out ebay and busted dot com sales for one real cheap. This type of load balancer works by grouping IP addresses behind one address, you can have several groups with hundreds of servers behind each group. The group consists of one public IP address and the IP addresses of machines that are load balanced for that address. Now you can do some very slick things with a load balancer more then just distributing the load for all HTTP requests to a pool of servers. You can do things like say "index.html is served by machine1 (or group of machines) while all adp pages are served by machine2 and all .jpg, .gif etc are servers by machine 3". You can even go so far as to send all people with a certain cookie value to one pool of servers or another based on their value (the better the customer the faster the service or better graphics etc). The load balancer can also do some primitive detection of machine failures and take a machine out of a pool. It will also allow for overspill (you know you can handle 100pps on the 4 load balanced servers) by taking the extra load and handing it off to a stripped down HTML server or even a backup group of servers. (Side note do not try and load balance firewalls with this machine, I heard it is incredibly difficult.)

I would suggest having two to three network cards in all machines that are to be load balanced. The first network would be the public side that is reached via the load balancer and the second would be for communication between servers and to the DB server. The third nic would be for a failover situation with the load balancer (but if you want to do failover I suggest you just pay the cash and hire a network guru to figure that all out, that is where my knowledge ends and I personally pick up the phone and start signing checks.) I would just use some cheap cisco 2950 switches between the alteon and each group of servers and something along the same strength between the balanced servers and the DB machine(s).

When I did this setup last I was not using aolserver but apache and PHP, be thankful you are runing aolserver, so much simpler to configure and streamline. With apache I had to add lingerd and SQLRelay to run more then four webservers in front of a modestly sized Postgres DB machine and that was only getting me 60-80pps with a minimum of one DB query per page.

One thing I would also recommend is getting a standard machine for your web servers so you can cookie cutter their installation and know how long it will take your supplier to get a new machine to your colocation facility. The biggest nightmare I have in scaling up is adding machines due to delays from manufaturers. Remember redundancy means redundant headaches so try and get some more people to help, the workload to implement these things needs load balancing as well.