Forum OpenACS Q&A: Performance of the server, how can we help?

This is a question mainly for Mike, but maybe someone else can answer it as well.

It seems that the server is reaching it's limits. I heard that there are volunteers to move OpenACS to a new site.

Can you give us a rundown on what is happening, a timeframe (or at least an order of things happening in the next couple of months) and pointers on how more people in the community could help.

A trigger is fired on forum post which walks up the hierarchy of messages to find the parent and schedule it for the search indexer - it might be inefficient. If you have time investigate it in openacs.org-dev. We definitely need something better for the toolkit forums search (which openacs.org will benefit from of course).

As I see it, there are two problems with the site/machine:

1. The machine has a hardware problem of some sort. After a reboot it takes about 5-minutes for the machine to get past the SCSI-bios disk-spin up. It could be a bad SCSI terminator or a sticky disk in the array. Furthermore, when I tried to update the kernel with the newest RPM from Red Hat (this machine is running a Dell-specific version of Red Hat 7.1) it went into a kernel panic and locked up. Rather than debug the problem at the time (since we had just driven down to NYC to fetch the machine from Ben and returned to Boston) I just put the old kernel in place and started the site.

2. There is a performance problem with some code somewhere. This machine is a 1-GHz PIII with 1.5 GB of RAM and 2 18-GB SCSI3 disks on a hardware RAID 1 controller. It doesn't get enough traffic to bog it down like it's been. We're running much busier sites on  less hardware.

I've provided a new machine to move the site to while we fix the current machine and get it updated. A team of volunteers lead by Dave Bauer is working to move the site, but I'm afraid I've probably been the bottleneck -- I've been busy and haven't had as much time to work on this as I'd like. But it's getting there. Dave will have a better idea of the current status.

Once we get the site moved to a new machine I'll be upgrade the old one with a second CPU and new, bigger disks and debug whatever hardware problems the machine might be having.

I am grateful to whomever fixed the forums submission problem. I'm now seeing 2-3 second response when submitting a post. Compared to well over a minute, it's lightning-fast!

Does someone want to explain what was done to speed this up?  Was it server-specific, specific to the openacs.org code, or something generic in the PG version of forums?

I think it was the secret server gnomes since I didn't do
it and I don't think Dave did either (and I don't see
anyone else logged in long enough to really fix anything).
Maybe it was some sort of redhat up2date thing that fixed it?

Geez ... sometimes the simplist things help so much :)  Good catch, Dave.  A good audit project for a bored hacker would be to go through PL/pgSQL functions making sure those that can be declared isstrict and iscacheable are so declared.

More granularity's been added, too, I think in PG 7.3.  Another reason to revisit PL/pgSQL functions.

I did an audit of this sort on acs-kernel when isstrict and iscacheable were first implemented in PG, there were some noticable speed improvements as a result.

Yeah, I know what you're talking about but apparently the root message id calculation was far, far worse.  Because it's only taking a couple of seconds to insert a message now vs. a minute or two.

Thanks goes out to Dirk Gomez for helping me analyze the query plan and Jeff Davis for thinking of checking if the function was defined isstrict, iscachable. Sorry I forgot you guys when  I responded.

Thanks to Dave, Jeff and Dirk.