There's no problem mixing PC100 and PC133 RAM on a machine with a 100MHz FSB. Just don't try running a 133MHz FSB machine with PC100.
If you mix ECC and non-ECC memory in the same box, the ECC feature won't be used on a LX/BX-based motherboard. You have to have 100% ECC.
If your machine's based on a I810, I810E, I815E-based motherboard ECC isn't supported. The memory will work but you'll not get ECC in other words. Same if you got stuck with one of those I840 motherboards with the hub chip that lets you run with PC100. You probably don't have on e of these I840+MCH kludges though 'cause Intel had to recall them since they were unstable.
If you aren't sure what you've got e-mail me the brand and model of your motherboard and I should be able to help you.
If you find Dell RAM shockingly expensive make certain you're not looking at one of their servers that requires RDRAM, which is still about twice as expensive as SDRAM (no matter who you buy it from) giving nothing near an equivalent boost in performance.
There's nothing wrong with using a "regular computer" as a server as long as you have quality parts. The main advantage "real servers" like the PowerEdge get you are redundant hotswappable power supplies, hotswap disks, etc. If you don't mind your system being down while you replace broken parts a "regular computer" can be a much cheaper option. It really boils down to how important it is to keep your site up as close to 365x24x7 as possible.