I think upgrading your memory would help. We had a similar problem some months ago and our first step was to upgrade RAM from 1 to 2 GB and make heavier use of util_memoize. It helped a bit, the webserver would support about 60-70 concurrent users instead of only 40-50 before getting awfully slow and unstable.
