Hi,
Maybe the best way to do it is to take it into stages. Move aolserver to a front end box and still use the solaris box for Oracle. Test the load, tune it. Add RAM, raid maybe but maybe not since this will be front end boxes maybe just huge amount of RAM and get everything in the case. After moving the aolserver then the Oracle maybe then moved to another boxed tune for RDBS, RAID 1, 1+0, etc. Test everthing up... after a while upgrade Oracle. Why upgrade Oracle? Is there a benefit?
Since the images are on the file system maybe another web server may serve this up. Not sure though it will depend on photo.net's way of serving its images. But have a look at www.mathopd.org, its one of fastest http servers, secure and very low in resources. Maybe adding squid or somekind of a reverse proxy. Dont know if this will actually have benefits I haven't really done this. It must have.... hehehe.
Use Intel based servers based on ServerWorks chipset. Although I am not sure if LE based chipsets has this dual SDRAM channels. I am sure HE offers this, so its basically 2x the bandwidth than other Intel chipsets. Stay away for BX and GX they are getting old. For AMD based I think the Altus server mentioned earlier is good. I believe that the design was from VA Linux then made its way to PenguinC. There is an extensive review of it on www.anandtech.com. Anand uses a similar server for this Cold Fusion driven site. Also the FSB of AMD is better than Intel, I think its called EV6 that came from DEC Alpha. From what I remember Intel FSB is shared amoung all the processors, so adding more processors will reduce the bandwidth on each processor. AMD does not suffer from this problem as each processor has its own dedicated bus to the chipset.
Has anybody tried out XFS? I have been using it on my machine, its ok but not sure though I have not benchmarked it. I dont know if am fortunate or not. They say that Reiser if good for small files and XFS is good for big files. Anybody have numbers on Oracle + Linux + XFS? and Pg + Linux + XFS?