Forum OpenACS Q&A: Distributed PostgreSQL - Clustgres, Postgres-R, etc.

Posted by Andrew Piskorski on
Btw, John Sequira recently pointed out this brief Yahoo article about Clusgres, a tool for clustering PostgreSQL for higher performance. In contrast to say, Backplane (which sounds like super-early-alpha code, not currently useful for anything real), Clustgres sounds like it might actually be useful now in some cases.

Aha, however, they require the (expensive, very high performance) Dolphin/SCI network interconnect. And they say they're using its shmem-style global shared memory hardware, so unfortunately, they are not just taking advantage of SCI's very low latency, they need that distributed shared memory support.

If they were using SCI just for the low latency, then presumably you could still run Clustgres and a cluster with Ethernet hardware (although with probably very bad performance), and could improve the performance somewhat adopting the GAMMA, M-VIA, or other codebases that let you come much closer to SCI latencies using cheap ethernet hardware, much closer than you can ever get using the typical Linux kernel and TCP/IP interface. But that isn't the case - oh well.

They say however that Clustgres currently allows writes, but is "optimized only for reads". Ah, and they're currently using NFS to share the filesystem. Ugh, how could that possibly support transactional and atomic writes? My guess is it can't, and that's why they're currently doing all writes only on the master node. Perhaps a real cluster distributed file-system with the right properties would let Clustgres scale for writes as well as reads, and maybe such a FS is do-able as long as they're already dependent on the SCI hardware.

Actually, I'm not sure how/why NFS would work for transactional reads either, but presumably they've made it do so. (I'm very foggy on what sort of actual locking and ACIDity primitives NFS provides, I just know they're supposed to suck.) Their "white paper" is pretty vague on how the magic "libOPUS" libraries used by Clustgres actually work.

Anyway, Clustgres seems interesting, but given the high cost of SCI hardware, and its current non-scalability for writes, I bet you'd have to have a large PostgreSQL database to make using Clustgres worthwhile. But, if you already had a one really big SMP server, as your database grows it might actually pay to offload the read requests to a bunch of (still expensive) smaller boxes with SCI via Clustgres, thus avoiding having to trade up to an even bigger (and probably much more expensive) SMP box.

I wonder how, if at all, this Clustgres stuff relates to Postgres-R. Since it depends on the SCI hardware I suspect it's unrelated. It would certainly be interesting to read a good compare and contrast of the two though...