Forum OpenACS Q&A: Open ACS handling heavy traffic

You might find the ASJ article on scaling helpful. They don't say what ACS version it deals with, although given the problems with aD 4.x permissioning that are fixed in OpenACS (see also this thread) I doubt it was 4.x.

So this will remove one bottleneck.  What other bottlenecks will we uncover?  Hard to say.  As Jonathan mentioned, more information on the actual number of pages to be served is needed in order to give you feedback, and as was mentioned later caching can help a lot, too.  You can cache directly in AOLserver and perhaps have more fine-grained control over when to expunge cached pages than through the use of Squid or similar tools (only your code knows when a dynamic page changes, after all).

One thing we are working on is the provision of automated testing tools, and this includes load testing using scripts that "know" about the structure and perhaps usage profiles of a site.  This will give users the ability to drive development sites with input that more accurately reflects reality (by posting to bboards, etc) than is possible with a simpler tool than (say) apache bench.

I realize none of this answers your question directly.  In part that's because I think the scalability issue for [Open]ACS 4 has yet to be answered - I'm outlining the steps we're taking to boost performance and to enhance our ability to test performance (and correctness) later.

One point of reference for OpenACS 3.2 ... this site right here withstood a slashdotting when running PostgreSQL 6.5 on a single dual-processor PII 400 box.

As an aside, I think 2 years ago a prof somewhere contacted a number of /.ed sites and graphed the common pattern of hits following the /. post -- and when his paper was /.ed, his site matched his numbers. If you grovel around in the /. search pages you might find it.

100 requests per second is what slashdot appears to serve at its best. This was mentioned by joe random user in a story there yesterday and it was also what I estimated back in August in a message to Roberto

Hi Roberto,

Thought you might be interested...

According to CommanderTaco, http://slashdot.org/~CmdrTaco/journal/ The new slashdot:

"We topped out over 55 pages/second for several minutes (Live Slashdot does only around half that much)."

Also we know that slashdot was: from http://slashdot.org/articles/00/05/18/1427203.shtml

• 5 load balanced Web servers dedicated to pages
• 3 load balanced Web servers dedicated to images
• 1 SQL server
• 1 NFS Server

So the old slashdot needed 5 webservers * 27.5 pages/second = 137.5 pages per second.

So those are 137.5 big pages (the homepage is right now about 45K). Outrageous BOTEC: all pages are 45K so 137.5 pages/second or about 6Mbytes/second throughput. Big, but not too big. Apache Bench tells me that I can do 98 45Kbyte pages per second on one machine (PIII 450) with AOLserver (system claiming to be 38% idle). What do they need 5 web servers for?

So for every user, how many db transactions are there *on average* to present a page? I think at most one and it's a read. Assuming every user is logged in, slashcode needs to get the preferences (what kind of content boxes (slashbox?) the user wants) and then find content to fill those boxes. But in general filling a content box should be a cached operation that changes only on the rare writes. I.e, I look up user id from cookie (3872), determine that you're the real bruce perens, determine you want these 20 boxes, and then look in the cache to see what's in those boxes and push the bytes out. Even the ordering and other various thresholds can probably be cached. Sort this box in highest order first, showing all messages, now cache it again with order by date, showing messages scored 2 or higher. You need some way to post a comment, commit to the db, and flush/fix up the cache.

I do think it's interesting to say that OpenACS is not a good fit for a Slashdot site. OpenACS as it is today is better suited to sites that need many more than one db transaction per page, and where there are significant chunks of writes, and not just reads. (In that sense, MySQL may not be such a bad fit for Slashdot.) Another way of saying this is that OpenACS is for "communities" of collaborating peers, not just mobs of lurkers.

Anyway, I don't know where I'm going with this....

(I'm using the present tense because I'm busy importing the openacs.org forums into my OpenACS 4 test platform so I can do some performance testing on the speed-up stuff I've been working on recently).

Also regarding Slashdotting.  AFAIK Ben did not track the slashdot volume.  However the trigger was a reference to the infamous "Why not MySQL?" paper and resulting comments, so undoubtably a lot of Slashdotters paid a visit.  After all, they're mostly rabid MySQL fans and get deeply offended whenever anyone criticized their favorite Open Source database (ignoring the fact that it wasn't Open Source at the time, as we don't want to fan the flames, do we? :)

Ben used it as a work machine at the time and did report that load barely topped "1" on the modest dual processor system.

They abstract out SQL differences by wrapping every query in a function, with all functions for each RDBMS being placed in the directory DB/[your RDBMS here].  There aren't very many queries in the system, either.  And, as I suspected, they're all very simple.  You see functions that call other functions to get values to use in the query, leading to the "execute lots of little queries" model of execution.  Also they don't seem to be wrapping inserts, updates and deletes in transactions even when using PG or Oracle though I can't be sure (in other words, have spent all of 15 minutes reading the code).

They template everything with templates drug from the database - of course this, like everything else, is heavily cached.

OK, time for someone else to spend 15 minutes reading the code in order to add to our understanding :)

I've been slashdoted and maxed out at about 6 hits/second over about 3 hours to the specific page that was mentioned.  The increase on the rest of the was not that substational.  We basically ended up creating a geocities site for the images, but the P133 with 128MB of memory and Linux 2.2 handled it well.  The only real problem was that the Apache/Jserv (at the time) combo got a little slow getting the pages going (all pages were .jsp's).  We ended up having to tune the kernel a bit and do a quick reboot.

I did have an OpenACS 3.2.5 development site running and (unknown to me) it was included in a E! documentary and received 5 hits/sec for 2 hours without a problem while the show was on TV.  I did move the site back to the pre-OpenACS code during the replays, but that was mainly because the site wasn't finished.

Hope that helps you a bit.

A big advantage of Oracle (and perhaps Postgres, but I'm not qualified to speak about it) over MySQL is that it scales better. Oracle is huge and somewhat bloated, but it is really intended to hold up even when stressed out. I could be wrong on this, but it doesn't look like MySQL even has a library cache (see here and here), which databases like Oracle use to cache parsed queries and their execution plans. A typical web application is going to generate many, many of the same queries (with different values plugged into the WHERE clause) over and over again. An application using Oracle bind variables (and shame on you if you aren't) will mean that the overhead associated with parsing queries is simply not there. Tweak the Oracle shared pool size so that it'll hold all the needed queries in cache (and use bind variables), and parsing overhead goes away. The queries will only be parsed the first time they're executed.

For applications without much load, the raw speed of its "lightweightness" makes MySQL look like a screamer compared to big, bloated Oracle. But increase the load, and Oracle simply ramps up while MySQL struggles under its own lack of sophistication. I believe similar results were shown by the big PostgreSQL versus MySQL web benchmark that was done a year or two ago. Under small loads, MySQL won. As traffic increased, PostgreSQL coped much better. Based on that, it looks like Slashdot needs a lot of hardware to support software that's not as fancy as it should be.

Of course PL/pgSQL only generates query plans the first time a function is run within a particular backend's context, and with much of OpenACS 4's logic tied up in PL/[pg]SQL we do get the benefit of that.

Caching query plans doesn't help when the basic queries suck, though (says Don as he's slowly making progress getting bboard to scale).

Put simply, each connection gets a process.  So if you have 50 connections hammering away, you have 50 context switches just waiting for attention.  Context switches really, really suck up the time.  In the case of Apache 1.x, each forked process takes up a connection.  So 5 servers times 30 processes equals 150 processes on the db server.  Aaagh.

AOLserver and (supposedly) Apache 2 deal with this with connection pooling. With pgsql it is a humongous deal.

Oracle, MySQL and Sybase don't fork processes for connections.  Sybase in particular uses select() to accept packets all in one thread.  Scary, but very effective.

Jonathan, I respectfully disagree that the parse time is negligible, if you're really concerned about scalability. Sure, 1/10 of a second is nothing when doing interactive queries. But if hundreds of users are descending on your site, you need all the help you can get. I think Tom Kyte's example (given here) does a nice job of showing this. I took his example and ran it on our own development database server:

SQL> -- the wrong way (no bind variables)
SQL> alter system flush shared_pool;

System altered.

Elapsed: 00:00:00.20
SQL> set serveroutput on size 1000000
SQL> declare
  2      type rc is ref cursor;
  3      l_rc rc;
  4      l_dummy all_objects.object_name%type;
  5      l_start number default dbms_utility.get_time;
  6  begin
  7      for i in 1 .. 1000
  8      loop
  9          open l_rc for
 10          'select object_name
 11             from all_objects
 12            where object_id = ' || i;
 13          fetch l_rc into l_dummy;
 14          close l_rc;
 15      end loop;
 16      dbms_output.put_line
 17      ( round( (dbms_utility.get_time-l_start)/100, 2 ) ||
 18        ' seconds...' );
 19  end;
 20  /
11 seconds...

PL/SQL procedure successfully completed.

Elapsed: 00:00:11.27
SQL> -- the right way (bind variables)
SQL> alter system flush shared_pool;

System altered.

Elapsed: 00:00:00.50
SQL> set serveroutput on size 1000000
SQL> declare
  2      type rc is ref cursor;
  3      l_rc rc;
  4      l_dummy all_objects.object_name%type;
  5      l_start number default dbms_utility.get_time;
  6  begin
  7      for i in 1 .. 1000
  8      loop
  9          open l_rc for
 10          'select object_name
 11             from all_objects
 12            where object_id = :x'
 13          using i;
 14          fetch l_rc into l_dummy;
 15          close l_rc;
 16      end loop;
 17      dbms_output.put_line
 18      ( round( (dbms_utility.get_time-l_start)/100, 2 ) ||
 19        ' seconds...' );
 20  end;
 21  /
1.06 seconds...

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.32

With PostgreSQL, you'll have to use techniques other than taking advantage of a library cache to get your speed. But if you're using Oracle, getting rid of those parse operations WILL give you better, measurable performance.

The parser itself should be fast ...

Experiments run by the guy who started work on the caching of query plans was seeing about a 30% improvement on simple queries.  On more complex queries parsing/optimization overhead's going ot depend a lot on the actual query.  If you join a bunch of tables but only return a few rows creating the plan will take more time than executing it, and caching the plan will be a big win.  On the other hand if you're doing an cartesian join of ten huge tables the optimizer will have few choices and run quickly, while the output will be so huge you'll probably run out of swap space before it is completed :)

As far as Oracle goes, I'm sure everyone here is aware of the fact that [Open]ACS 4 uses bind variables throughout.  AFAIK though, no one has done any measuring on an active site to see how much memory Oracle should be allowed to use for caching in order to see the benefits without eating too mucn into your shared buffer space used to cache database blocks.

Functions in PostgreSQL 7.2 (in 3rd beta right now) can return a cursor, which is equivalent to returning a relation. That should make working with Oracle <->PostgreSQL a lot easier.

7.2 also supports using the Oracle-style %TYPE in the declaration of parameters. Another win with working with Oracle.

Tom Lane did a nice job clarifying some obscure points of PL/pgSQL in the (now much better formatted) 7.2 documentation for PG's procedural languages: http://candle.pha.pa.us/main/writings/pgsql/sgml/plpgsql.html

What I'm also not sure of is how to deal with cookies and ACS - presumably a user who gets a cookie from webserver A has to be redirected back to webserver A rather than webserver B on their return. Am I right? Is this straightforward with a device lke the Cisco one above? How do people recommend such a thing is dealt with?

They do connection-based load balancing. The short story is that you give Big/IP's external interface the IP of your www. URL. You configure the Big/IP to do load balancing among a number of servers. It will rotate each incoming connection, following some algorithm or another for 'rotation', among the servers you've identified.

You can do this per-port, per-IP. So say you want to route port 80 to machines A, B, C, and port 443 to beefier machines D, E, F, and G. No problem. Need to ssh into your farm? Route port 22 somewhere else.

etc. You get the idea, I hope.

There are free + libre solutions for Linux that do the same thing. But they're also huge hassles. If you look at the Big/IP hardware and add the time you'd spend configuring a Linux load balancing and failover system, big/ip ends up more or less worth it. Particularly if you want to add failover features (by buying a second big/ip) then it gets even more worth it.

To directly answer your question, this is not an issue with cookies. The browsers all think they're connecting to the same machine. Unless, of course, you're keeping client state on the web server (tied with cookies). In that case, you have a severe architectural problem and you'll have to find another form of load balancing. My bank appears to have this problem, since every time I do web banking, I get sent to webbankingXX.tdaccess.com for random values of XX.