Do you have your limits set correctly? Max OS X sets them pretty low, to install Oracle you need to override the defaults. I've incorporated the ulimit values from Oracle's install directory in /etc/profile and have had no problems:
I'm using Postgres, though, but I guess this applies even with it. I put the lines in /etc/profile, but it didn't help, even after rebooting the whole box. The behaviour changed a bit, though: now I even get "(core dumped)" after the errors given above.
