Forum OpenACS Q&A: PostgreSQL vs Interbase- Future Direction of OpenACS?

Hi, I'm working with my team (of two!) to build a RDBMS backed
commercial-strength website. ACS is a great resource for us. OpenACS
is even better since it uses open source DB.

I have two questions.

1) Is PostgreSQL good enough (scalability, reliability, etc) vis a
vis Oracle, to build a website for a commercial strength deployment?
Are there any commercial-strength websites backed by OpenACS on
PostgreSQL in deployment today?

2) Is the future direction of OpenACS with PostgreSQL or Interbase?
Or is it both? In other words, if I choose to go with PostgeSQL
assuming that OpenACS will support it in the future, am I taking too
much risk?

Thanks.

Good to hear that OpenACS is of use! Remember to link your
site when it's up and running if you go with OpenACS!

(1) Postgres is not as scalable or reliable as Oracle. Although, I
have a friend who tells me that his Oracle 8i database crashes
(whole thing core dumps) consistently on a particular piece of
SQLJ, so who knows where Oracle is going by adding more
bloat to their core server. Regardless, Postgres should be able
to support a moderately popular site with reasonable hardware.
We're not talking yahoo.com, but certainly a site that takes a
couple million hits a day (again, given the right hardware).

(2) OpenACS has every plan of maintaining Postgres support for
the foreseeable future. We may attempt to separate out the SQL
well enough that we can also offer Interbase compatibility, but I
don't see us switching altogether.

We encourage you to use OpenACS! We don't have any huge
sites currently running it, but the technology is solid. Our biggest
site so far is http://community.aolserver.com

PostgreSQL has a larger user base than Interbase and we all like PG. Liken Ben said, we are not dropping PostgreSQL for it is improving constantly.

We are trying to make things easier for those that eventually would like to port OpenACS to Interbase. If anyone wants to take it over, go for it.

Thanks for the answers Ben and Roberto!

I have one more question. Does OpenACS have all of the modules (OR almost all, ie >90%) that ACS has? Or is it a subset of ACS?

OpenACS is a 95% port of ACS/Oracle and, excluding the Java-Oracle specific modules (e.g. webmail) we should reach 100% really soon now (R).

OpenACS is not a subset of ACS/Oracle. We intend to provide full capabilities, and we hope more and more modules are created by OpenACS users.

Even the Java modules are being worked on by Dan Wickstrom and we can expect them to be ported as well.

A slight addition to the above - Postgres is on a great trajectory.

18 months ago it was unusable for the kind of websites we want to use it for today, too slow (including unecessary disk writes after read-only selects, a killer for web sites using the db for personalization), table-level locking, and so flakey that I knocked it over during my first day of testing.

About a year ago, 6.5.3 came out.  Many bugs were fixed, MVCC (row-level lock granularity much like Oracle), no unecessary disk writes, etc all implemented.  This release convinced me that perhaps basing some web work on Postgres was practical after all.

Now we have 7.0.  Along with some new features such as referential integrity constraint checking, there's been tons of bug fixes, considerable improvements in the optimizer, improvements in disk management by the back end, the addition of to_char() for Oracle compatibility, and more.  A great step forward.

7.1 will be easily as impressive.  Write-ahead logging will be added for complete satifisfaction of durability needs (PG is slightly vulnerable in its current form, but much closer to the bullet-proofness of Oracle than to the cluelessness of MySQL), a really slick large-object facility (already partially implemented), and our (OpenACS) personal favorite, outer joins.

Ben's assessment's right on, with the future looking even better.

I'd love to see us support Interbase if we can abstract out SQL differences in a way that makes supporting two databases practical.
Having two good open source database engines available is at least four times as good as having just one available.

FYI. This is an excerpt of a press release announced today regarding the formation of "Great Bridge LLC" to provide commercial support for PostgreSQL. A $25 million bet on PostgreSQL!!

"Norfolk, Va, May 9, 2000 - Landmark Communications, a large privately held media company whose chairman bankrolled Linux leader Red Hat Inc., announced it has funded a wholly owned subsidiary with the intention of doing for databases what Red Hat did for Linux and operating systems - give business a formidable open source alternative to existing proprietary applications.

Newly formed Great Bridge LLC is backed by more than $25 million from Landmark Communications"

The full press release is at http://www.postgresql.org/news.html.

Definitely good news for OpenACS!!

Yes.  Great Bridge has signed up two of the developer's group thus far.  I think the odds have gone up greatly that PG 7.1 will have the features we need.
Sorry, I'm still working my way through the code and docs so this may be off the mark... But I noticed the XML module is not yet ported, presumably because it relies on Oracle's internal XML capability.

Also gathered that the decision to implement most stuff in Tcl is largely for "historical" reasons. If I haven't missed or misunderstood something fundamental, this suggests a way to kill several birds with one stone:

1) Work with Postgresql people to implement internal XML support similar to Oracle. Seems to be a critical requirement for database oriented web services and may be higher on ACS/pg priorities than on Postgresql priorities. But the Postgresql architecture as a true ORDBMS cf Oracle RDBMS seems ideally suited for easily adding that sort of functionality as just another of many Postgresql add ons. Are they working on it already?

2) That would really bring out the not just financial advantages of using a free DBMS cf a commercial one. Just porting RDBM features used by ACS for "historical" (Oracle) reasons tends to highlight the difficulties due to Postgresql not having yet got around to some things that are common in the commercial RDBMS world like outer joins. Specifically, Postgresql should end up being able to do a great deal more with XML than just importing and exporting it because of the way it is designed to treat fields as structured objects on which you can define operations.

3. Gradually eliminate the Tcl stuff that just converts between SQL and Tcl/AOL to use XML as "middleware", while at the same time adding enhancements that take full advantage of XML.

4. End up with an OpenACS that is:

a) More database independent (ports to specific SQL dialects fiddle only with the XML to SQL for their dialect, not with Tcl code intertwined with the rest of the logic, unlike entirely separate ports for Interbase and Postgresql).

b) Fully able to participate in the enormous momentum around XML.

c) Less dependent on Tcl. (Existing ACS specific functionality just disentangled from the direct SQL dependency, additional specific functionality can be added in whatever language the developer prefers - many do not prefer Tcl).

Some interesting ideas.  I don't see XML on the PG radar screen, I'm afraid.  If it weren't for us (OpenACS), I'm not sure the web would really be on their radar screen.  At times I wonder if they HAVE a radar screen!  Let's just say that the developer's group has grown from a fairly eclectic user community which seems to somewhat detatched from the real world.  Great Bridge should help repair this disconnect, forming a bridge to the real world.  The more I think about that corporate name the more I like it, hmmm!

aD has played with at least two open source XML parsers written in Tcl.  The use of XML you propose isn't dependent on the parser being embedded in the database engine, and in fact portability might be enhanced if it isn't.  On the other hand, since Postgres supports embedded Tcl procs burying a parser might be easy.  There are parsers available in Java, too, but you can't embed Java into Postgres.  One OpenACS guy is working on a "mod_java" for AOLserver, though, so making use of a Java XML parser might be possible before too long.

As far as using XML as a tool to provide an abstraction layer between the database and code dependent on query results, it's one way to go about it.  We've tossed around other ideas, as well. One thing for certain is that we'll be abstracting the SQL rather than maintain a separate port for InterBase.  Exactly how we'll attack this problem is  up in the air, still.

For the most part, the OpenACS team is content with Tcl as the underlying language.  Or more accurately, content with whatever arsDigita is doing as long as we can easily map it from Oracle to Postgres or Interbase.  At this point, we still see ourselves as leveraging aD's efforts rather than breaking off on our own, though at times forking seems like an inevitability.  We'd prefer not to, though, and if aD decides they are willing to work on abstracting out SQL rather than scattering Oracle-isms hither and thither throughout the code, we'll probably adopt whatever choice they make.  They pay real money to real staffers to work on the toolkit, so we should take advantage of this fact as much as possible.

Another inevitability - a joyful one! - is that this project will continue to attract thoughtful and intellegent people.  The time may come when we feel like we have sufficient resource to do whatever we want, paying little or no attention to aD.  If so, it's some time down the road (and personally, I hope OpenACS and aD's ACS Classic stay coupled).

Welcome aboard!

Thanks for the warm welcome.

Unfortunately I don't really qualify for a "welcome aboard" as not likely to contribute much useful code but only some thoughts based on browsing around lots of different projects. Hopefully at least some of that may be useful as "cross fertilization" for people who are actually doing the hard stuff. I'm sure what isn't useful will just be rejected anyway 😉

1. Re XML inside Postgresql or outside. Agree that it can be done either way. The interesting thing is that it looks (to me, from outside) as though it can be done so easily inside Postgresql (without even getting right inside) that an equivalent to the Oracle XML facility should be almost trivial for people who have been getting right inside (combined with people who have built XML parsers etc). Your description of the gap between Postgresql team concerns and current developments on the web seems a plausible explanation of why, if it is that easy, it might not have been done. Adds weight to my suggestion that it could be higher on OpenACS priorities than Postgresql priorities (as well as higher on OpenACS priorities than ACS classic priorities since Oracle already has it).

As has been pointed out (with some vehemence) elsewhere, many people doing interesting and useful things on the web haven't got a clue as to why one would need ACID transactions. This is probably true of much of the XML community. So OpenACS might well be the most logical place for an initiative to bring Postgresql people and XML people together on this, as it SHOULD be very high on OpenACS priorities. Unless I've misunderstood the ORDBMS concept it should be a perfect demonstration of Postgresql's advantages so really important to them too once the relevance is made clear. XML integration is far closer to the non-web related ORDBMS stuff Postgres was originally developed for than any of the "conventional" SQL stuff it is actually being used for. So it could be more interesting to them if their attention was drawn to it.

2) Re using XML between the database and application, whether or not inside the database. This is becoming pretty standard. Even Microsoft is recommending it in preference to their own proprietary ADO "databinding" and adapting the latter to XML.

A point not mentioned in stuff I've seen here re XML so far is that it will soon also be pretty standard for actual delivery to the browser. IE5 has already overtaken IE4 and Netscape is emphasizing that their next release is more standards compliant than IE4. So the majority of browsers will soon have proper XML parsers that can also handle XSL style sheets. It doesn't make sense to me to develop non-XML approaches to templates. With an XML approach you can mix XML and other stuff from the database with XML templates (using Tcl 😉 to produce intermediate XML which either gets just shoved to the browser or rendered into HTML at the server for non-XML browsers. This seems to fit well into the architecture behind ACS of the web server being the database client.

3. re Tcl and not forking from ACS classic. I agree that not forking is far more important than language preferences. The contribution made by ACS is in having a working community system where the web server is the database client and the ongoing work on that is what makes the OpenACS contribution of adding support for Postgresql instead of Oracle worthwhile.

However my reference to killing several birds with one stone still applies. By disentangling the Tcl application code from the database dialect so you can support multiple databases (whether using XML or not) and ALSO disentangling the Tcl application code from the presentation so you can install upgrades without wiping out customizations (again whether using XML or not), you gain the side effect of also being able to add new ACS application functionality that can talk to both the presentation and the database and need not necessarily be written in Tcl (while retaining the existing massive investment in Tcl code plus the additional investment in disentangling that code, which will HAVE to be done at some point).

By using XML for both, OpenAccess could leverage the Tcl/XML work common to both these separate problems and hook into the momentum behind XML and the inevitable consequences of imminent changes in what browsers people are using. (XML not being central to the concerns of a database oriented web application could be almost as out of touch as database team not paying much attention to the web).

I guess whether that is practical or not depends very much on ACS classic. I have no knowledge and therefore no opinion on whether they could be persuaded to move in that direction. But it seems pretty inevitable that everyone will have to eventually. Not making XML central will soon be a bit like not making HTML central would have been when the web got going. ("What's wrong with telnet and ftp, they work don't they?")