Running two or more ACS instances on a site is very easy. Your last set of problems seem to be due to files going in different places due to mixing the RPM install with tarballs, as you point out.
You just need to remember to:
1. have the two instances listen on different IPs or different ports on the same IP (if you get tired of typing "8000" or whatever at your browser remember that it is very easy to multihost local IPs on a single NIC in Unixland).
2. two different pageroots
3. two different PG databases
4. remember that the database pools in your .tcl init file has to point to the right database for each instance
that's really just about all there is to it. Lately I've been routinely running an Oracle and PG instance of OpenACS 4, so I can check on stuff back-and-forth between the two (takes lots of memory to run Oracle and PG together in an efficient way, other than that it's no big deal).
