This error message means that you have exceeded the maximum number of file handles (or maybe inode handlers) the kernel can allocate. How do you determine what these values are?
cat /proc/sys/fs/file-max
cat /proc/sys/fs/inode-max
There's a downloadable document (4MB! PDF) on the web called Securing and Optimizing Linux that discusses this. Older versions of RH Linux had file-max set to 4096 and inode-max set to 16384. For file-max, you're supposed to budget 256 per 4MB memory (i.e., 8192 per 128MB RAM). For inode-max, this should be 3-4 times the value of file-max.
You can change this on the fly:
echo "8192" > /proc/sys/fs/file-max
echo "32768" > /proc/sys/fs/inode-max
I change the defaults at boot by putting the following lines in /etc/sysctl.conf (I have 384MB RAM):
# Increase no. of file handles - 2.4.7 default, recommended 8192 per 128MB RAM
# fs.file-max = 8192
fs.file-max = 24576
# Increase no. of inode handlers - 2.4.7 default, recommended 3 x file-max
# fs.inode-max = 16376
fs.inode-max = 98304
Basically, the shrink-wrapped RH kernels suck. You should be A.) compiling your own, and B.) tune the kernel to your particular configuration. You can change stuff like maximum shared memory (SHMMAX), virtual memory behavior, etc. Check out the stuff in /usr/src/linux/Documentation/sysctl/ and read the aforementioned guide (it's rather old, but mostly pertinent even today).
This guide specifically points out that web servers and file servers run into the problems you have seen. Good luck!
