I remember reading an interview with some Oracle tech about running Oracle on newer Linux kernels and the code and advice they had/will be offering. Specifically, the 2.4 kernel supports raw IO, and the Oracle tech mentioned a speed up of something around 30%. Raw IO allows oracle to write staright to disk, bypassing the kernel buffer cache.
I haven't had a chance to try this myself so I don't know how easy this is in practice. One of the other nice things about the 2.4 kernel is the Logical Volume Manager. This would be useful in a situation where you wanted to run some journaled file systems, some raw partitions for the db, software RAID (which I've been very happy with, but my requirements don't match up to photo.net's...)
You can run more than one RAID level! Try a tablespace on a RAID 1 (mirrored) volume with raw IO for tables with data that must be preserved. Another tablespace for indexes (derived data, materialised views etc.) on a RAID 0 (striped) volume for extra speed and space efficiency. Use one of the journaling file systems for the rest, perhaps XFS considering the need for efficient io on large photos. At least RAID 1 for this, or RAID 5 if you have the disks and need the space.
You can play tricks with Oracle (9i comes with extra tricks...), for example if you carry through with the idea that indexes store non critical data and can be rebuilt, you can run that table space with the 'no logging' option. Can you even use a temporary tablespace? Are your multikey indexes compressed? Are you using newer features like index organised tables, bitmap indexes, bitmap join indexes...
I thought the concurrency problem solved by running 10 nsd processes was suspected to be in the Oracle client libraries. It might be worth re-checking as Don mentioned, with 9i. The prefetch option was added recently to the 2.6 oradriver, which helps a little. The driver currently doesn't support prepared statements, but (again...) a 30% speed improvement is mentioned in the Oracle docs when using this feature.
Keeping it all on one system would be less expensive, and easier from a management point of view. Modern PC harware should be way faster than your old Sun box...
