Weblog Page

A history of OpenACS

by Ben Adida

In the Beginnning ...

Back in the summer of 1995, Philip Greenspun, Brian Tivol and I spent a few weeks in New York working at Hearst Publishing developing the Multimedia Newsstand. The tools available at the time were pretty pathetic. Netscape was at version 1.1 (barely), and about 30% of our visitors were still using v0.9.

Philip had done his homework, though, and had chosen Naviserver as our web server platform. Naviserver was the brainchild of Jim and Doug, two amazing hackers who immediately understood how to build server-side web technology:

• Built-in, simple, string-oriented, scripting language: Tcl,
• Efficient multi-threading,
• Simple and abstracted database access and connection pooling.

We set out to create a web site with daily editorials, magazine sales and ordering, customer tracking, and quite a few more pieces. In about 12 weeks, it was up and running and happy. The Illustra RDBMS would crap out every now and then thanks to the Elan License Manager (among other problems), but overall Hearst was happy. The numerous utility procedures that we created and used that summer came together into what is now the utilities.tcl file in your ACS installation. The information so far should allow you to guess the reason why bt_mergepiece is called bt_mergepiece.

 

Driving the Idea

It was Philip who then started to use Naviserver (later GNNserver, now AOLserver, all the same product) and Illustra to create a host of services that became greenspun.com, in addition to the early versions of photo.net. Philip created the initial versions of bboard, classified, neighbor-to-neighbor, and many other pieces all in order to manage the growing photo.net community.

Philip continued to push AOLserver, eventually dropping Illustra by hiring Cotton Seed, another hard-core hacker, to write an Oracle driver. Oracle brought a whole new level of scalability and reliability to the thousands of lines of Tcl code already written. At that time, in early 1998, Philip officially created ArsDigita, LLC, in order to push the consulting work he was already doing. He brought on 6 people (Philip, Olin, Cotton, Terence, Ulla, and myself) to carry the initial ArsDigita flag (although I had almost nothing to do with the setup of ArsDigita to begin with).

Philip brought on Jin Choi, one of the only people I know who deserves the title of "monster hacker." Together they went ahead and built entirely new services based on AOLserver/Oracle. The most famous of these was, of course, scorecard.org, an amazing site that anyone claiming to understand web scalability should take a look at (30 db-backed hits/second on Earth day running on one Sun Ultra 2).

 

And There Was A Toolkit...

It was right around that time that Philip convinced me to come back and work full-time for ArsDigita. Jin, Eve, Tracy, Philip and I worked through the summer of 1998 on various projects (Levi Strauss, Cognet, ASME, Greentravel now away.com), while Philip kept talking about his grand-integration goal: combining all of these pieces into the ArsDigita Community System.

Soon enough, the ACS was real. The first release was posted on December 8th, 1998, after a huge packaging, debugging, and integration effort led by Philip. The ACS became the backbone of all ArsDigita projects, and many hackers around the world started using it.

With the rise of open-source software and the realization that good software can still be free, many people started wondering if the ACS could be made to run on some RDBMS other than Oracle. An Interbase port of the ACS v2.1 was created, but Interbase still cost money at the time (although it should be open-source and free by end of 2000). Many cheered for MySQL, but the lack of transaction and subselects makes it unacceptable for a true ACS (or for any critical system, for that matter).

 

PostgreSQL

In December 1999, a small group of ACS hackers came together on SourceForge to create the ACS port to PostgreSQL. Following true open-source methods, we gave write permissions to anyone who showed enough competence to help out. The group soon grew to more than 20 people, with about 5 active developers.

The initial name of the project, ACS/pg, was changed to OpenACS as the group realized that there was a need to push porting to possibly other databases than PostgreSQL (Interbase?). OpenACS further represents the importance of a fully open-sourced system that truly works in symbiosis with the Open-Source community.

Philip and his team have done a tremendous job creating the processes and data model necessary to build scalable, reliable online communities. OpenACS hopes to bring this tremendous contribution to the world of fully open-sourced systems, available to anyone interested in building their own online community.

http://openacs.org/about/history

I'm spearheading a project to develop project management software for OpenACS.

OpenACS.org Project Management Page

This is a page for miscellaneous notes on Postgres.

Example of a trigger in Postgres

create function modify_date() returns OPAQUE as '
 begin
        new.modified_date := ''now'';
        return new;
 end;
 ' language plpgsql;

create trigger modify_date_trig 
before insert or update on pan_type
for each row
execute procedure modify_date();

Thanks to Priti Sawant for his help on this.

Common errors

ERROR:  Relation "rl_resources_revisions" with OID 330733 no longer exists

This is Postgres' helpful way of telling you to recreate your view. Or it might mean that you should say "CREATE OR REPLACE" when you drop and create a function.

Hierarchical queries

See using tree_sortkey for OpenACS.

Getting around 16 argument limit

See how to update after an insert.

Or upgrade to Postgres 7.3.x, which doesn't have that limit.

Dates

To specify the present date and time, use CURRENT_TIMESTAMP. If you just want the date, use CURRENT_DATE.

Also, adding integers to dates in Postgres

Also, selecting a series with Postgres:

select current_date + s.a as dates from generate_series(0,14,7) as s(a);
   dates
------------
 2004-02-05
 2004-02-12
 2004-02-19

(3 rows)

to_date vs. to_timestamp (also info on coalesce and truncate vs. date_trunc)

Reserved words

• Postgres reserved words list

Upgrading from 7.2.x to 7.3.x

Upgrade script that handles the function and view truncation problems
Remember that psql 7.3 doesn't like tabs

Installation troubleshooting

• If you get an error message like this during ./configure :

  checking for readline... no
  configure: error: readline library not found
  If you have readline already installed, see config.log for details on the
  failure.  It is possible the compiler isn't looking in the proper directory.
  Use --without-readline to disable readline support.
  

  Then you need to install the libreadline4-dev package (which includes the header files you need). On Debian:

  apt-get install libreadline4-dev
  

Postgres query debugging

*.xql files are loaded into memory at start. They take precedence over queries quoted in *.tcl files.

Sometimes Oracle syntax or reserved words make it into postgresql queries. Here (is the start of a list) of equivalents

• Oracle > Postgres equivalent
• sysdate = current_timestamp

Querying multiple databases from OpenACS

http://openacs.org/forums/message-view?message_id=118550

Dealing with concurrent issues

How do you make sure that user A and user B don't clobber each other's changes?

 

http://developer.postgresql.org/docs/postgres/sql-select.html#SQL-FOR-UPDATE

Using Tcl Namespace

The main reason to use namespaces is to avoid namespace collision, where you have two variables called the same thing.

http://www.tcl.tk/man/tcl8.4/TclCmd/namespace.htm this reference really helped me understand how namespaces work.
http://mini.net/tcl/1489
http://www.collaboraid.biz/developer/package-developers-guide

 

namespace eval mypackage {}

Then create the procedures using their fully qualified name as such:

proc mypackage::get {} {}
proc mypackage::set {} {}

Acknowledgements

DaveB, Bart

Why would I want to use OpenACS's templating system?

This is a basic introduction to the templating system in OpenACS.

Why would you want to read this document? Well, if you want to put up a webpage using information from a database, the templating system it what you use to do it.

Background

If you've used HTML to build a webpage before, then you already undertstand a lot of the process. In OpenACS, the webpage that is most similar to the HTML page you're used to is an ADP page.

So, instead of index.html, we'll start with index.adp

What OpenACS adds to HTML is a couple of extentions which allow you to incorporate dynamic information from a database. The dynamic information comes from two other files with the same name as the ADP file, but a different suffix. So for index.adp, you'll also have index.tcl and index.xql (sometimes index-oracle.xql or index-postgresql-xql if the code is database specific. This allows for multiple database support)

OpenACS has a very powerful way of separating code and presentation. If you're working on a large team of programmers, this allows you to let the programmers concentrate on providing datasources for the graphic artists, and let the graphic artists concentrate on making the pages look good.

Overview

The basic idea is you have three files for each webpage:

• a .adp page - this is HTML with a few added markups to allow for dynamic, database data to be inserted in the page.
• a .tcl page - this handles permissions and other aspects of the code. It provides the data to the adp page.
• a .xql page - this is the SQL executed by the database. If the SQL code is oracle or postgres specific, it might be preceded by -oracle.xql or -postgresql.xql

Sometimes it may be confusing to see sql code embedded in the .tcl file. Ignore this, as the SQL code in the .xql file has higher precedence.

These are the three steps you need to take to have things properly abstracted:

Refer to the variables in the .adp file

<p>User 2700 has email:  @email@ and a url: @my_url@</p>

Let's list all the email addresses:
<ol>
<multiple name="foobar">
<li>@foobar.email@</li>
</multiple>
</ol>

Perform the queries in the .tcl file

ad_page_contract {
    Main view page for one project.

    @author jader@bread.com
    @creation-date 2003-05-15
    @cvs-id $Id$

} {
    one_user_id:integer,optional
} -properties {
    email:onevalue
    my_url:onevalue
    foobar:multiple
}

# note how there is no SQL here, just an empty statement {}
# and we use the query_name from the .xql file.

# single row query for num_parties
# sets the value for num_parties in Tcl. The -properties portion of this page
# then exports it to the .adp page
db_1row one_user_select {}

# now we have values for email and my_url

# multirow query for emails
db_multirow foobaremails_select {}

The -properties section that identifies variables that can be used in the .adp file.

Define queries in the .xql file:

<?xml version="1.0"?>
<queryset> 
  <fullquery name="one_user_select"> 
    <querytext> 
        SELECT email, url as my_url 
        FROM parties
        WHERE party_id = '2700'
    </querytext> 
  </fullquery> 

  <fullquery name="emails_select"> 
    <querytext> 
        SELECT email FROM parties
    </querytext> 
  </fullquery> 
</queryset> 

Troubleshooting

A few things to try:

• make sure your .xql files (and others for that matter), have been added to your package's list of files. Do this in /acs-admin/apm
• make sure your .xql files are being watched. You can set this in /acs-admin/apm/

Templating inheritance

to inherit site master, and yet have a local (package or section)
master template, do:

-> in package/section master:
    <master>
    blah
    <slave>

-> in individual adp:
    <master src="path_to_section_master">
    blah
    <slave>

Mapping procedures to sql queries

If you're writing a procedure, then when you refer to the .xql file, the fullquery name has to include the full procedure name, then period, then query name: spm::project::change_order.get_current

Thanks

Dave Hwang's posting http://openacs.org/forums/message-view?message_id=100950
Jim, on IRC
DaveB, on IRC
Alex Vorobiev

Other references

• Reuven describes in great detail how the templating system works, and how to use it. Written for beginners, and for those unfamiliar with OpenACS.
• Jim describes how to use templating in an IRC session: http://jam.sessionsnet.com/files/download/templating-intro.txt?version_id=3168

Advanced topics

• How to dynamically return different template appearances