For anyone who is interested, I found out why both CPUs were not being utilized. FreeBSD uses userland threads which can't be shared across processors. AOLServer is a single process running multiple threads. One solution would have been to run another instance of AOLServer and do some sort of round-robin DNS. But I chose to use linuxthreads.
I compiled the linuxthreads port for FreeBSD and recompiled AOLServer to use linuxthreads. linuxthreads causes each thread to be treated as a process. After restarting AOLServer, I ran top and watched the threads spread to both CPUs during the benchmark. It looks like FreeBSD can put individual processes on separate CPUs, which was confirmed when I looked at the postmaster running on the other FreeBSD machine. Each connection was a separate process and FreeBSD spread the load across the CPUs.
As a result of the tweaking I doubled the connection/sec since I started benchmarking on this particular page (3 connections/sec to 7 connections/sec). PostgreSQL is not getting worked at all so I am convinced that another webserver will help greatly. For this server (dual PIII, 1GB RAM, OpenACS 4, PG 7.1.2), it looks like the maximum connections/sec is 17 (just serving the default aolserver index.html page).
I'm compiling a FreeBSD install/tuning guide which I will send to Roberto as soon as I finish.
