View · Index

No registered users in community rubick
in last 10 minutes

Using Arch for source control management with OpenACS

Email me if you have corrections or additions to this document.

Why use Arch

I've used CVS for a couple of years, and even though you can't live without version control, and it's decent, it is not well-suited to web application development. You run into problems with branching, tracking local customizations and keeping track of a changing code stream.

Arch handles these much better than CVS:

Branching
Renaming files
Moving directories

Compared to the competition (Subversion for example):

Handles distributed development

Subversion is also a very worthy tool -- I use Subversion every day. But I chose to go with Arch for an earlier project. Arch is pretty interesting, and if I didn't occasionally run into annoying error messages I would recommend it more. I also find that the "user interface" is too focused on people who understand the mechanics of how Arch works, rather than on abstracting those details from developers.

Learning about Arch

The best place to start with is GNU Arch. It has instructions for installing and learning Arch. You should at least look over the tutorial there to make sure you understand how to get help.

How we'll set things up

First of all, we're not going to tackle the more complicated issue of setting up a CVS to Arch gateway. This is very possible, but not something we're going to get into at this point.

We are setting up a very rigorous setup, which will have a developmental server, a staging server, and the live server. This will allow you to have localized changes to your OpenACS installation, use the remote repository without doing a bunch of CVS chicanery, and in general allow you to test changes before going live with them.

We will set up an OpenACS Arch repository that will mirror the stable releases of OpenACS core (currently 5.1.3). The website we are building is safe4all.

User	Purpose	Archive	Category	Branch	Version	Ports
safe4all-dev	Mirror OpenACS core releases	jade@safe4all.org--2005	openacs	stable	5.1.0	None
safe4all-dev	Import packages, html pages	jade@safe4all.org--2005	openacs	dev	5.1.0	8000/8443
safe4all-staging	Local customization	jade@safe4all.org--2005	openacs	staging	5.1.0	8001/8444
safe4all	Live site	jade@safe4all.org--2005	openacs	staging	5.1.0	80/443

Initial setup

# groupadd web
# useradd -m -g web safe4all-dev
# useradd -m -g web safe4all-staging
# useradd -m -g web safe4all
# apt-get install tla tla-doc

See the OpenACS wiki guide to installing OpenACS on Debian. Set up the Arch IDs for all the users

# su - safe4all-dev
$ tla my-id "Jade Rubick <safe4all-dev@safe4all.org>"
$ exit
# su - safe4all-staging
$ tla my-id "Jade Rubick <safe4all-staging@safe4all.org>"
$ exit
# su - safe4all
$ tla my-id "Jade Rubick <safe4all@safe4all.org>"
$ exit

You should also edit your .bash_profile or equivalent to make your umask 0002 instead of 0022. This helps you with permissions issues later. Do this for safe4all-dev and safe4all-staging.

Setup OpenACS core release mirror (stable branch)

What we'll do is track the latest stable release of OpenACS from CVS. Currently, that is 5.1.5. What we will do is start with 5.1.0 (start later if you want to save yourself some time), and upgrade each version to get to the current one. This tests our upgrade process and makes sure we're on the right path.

The basic way we're setting things up is to setting up your own arch repository.

If you haven't already:

# su - safe4all-dev
$ mkdir ~/{archives}

Now set up your openacs-stable archive.

$ tla make-archive jade@safe4all.org--2005 ~/{archives}/2005
$ tla my-default-archive jade@safe4all.org--2005
$ tla archive-setup openacs--stable--5.1.0
* creating category jade@safe4all.org--2005/openacs
* creating branch jade@safe4all.org--2005/openacs--stable
* creating version jade@safe4all.org--2005/openacs--stable--5.1.0
$ mkdir ~/wd
$ cd ~/wd
$ cvs -z5 -d :pserver:anonymous@cvs.openacs.org:/cvsroot co -r openacs-5-1-0-final acs-core  
(see http://openacs.org/4/checkout)
$ mv openacs-4 openacs-5-1-0
$ cd openacs-5-1-0
$ tla init-tree openacs--stable--5.1.0

We have to get all the source into Arch. You should look up the inventory and tree-lint commands if you don't understand what they do. They essentially help us make sure that we're importing all the files into Arch the way we want to.

$tla inventory --names --both --source |xargs tla add
$tla tree-lint

We have to get to a point where tla tree-lint doesn't report any issues. Here are some hints:

We now need to edit the =tagging-method file, to ensure that log files aren't stored in Arch, that the daemontools' supervise directory isn't stored in Arch, etc..

emacs {arch}/=tagging-method

Change this line:

precious ^(\+.*|\.gdbinit|\.#ckpts-lock|=build\.*|=install\.*|CVS|CVS\.adm|RCS|RCSLOG|SCCS|TAGS|\.svn)$

precious ^(\+.*|\.gdbinit|\.#ckpts-lock|=build\.*|=install\.*|CVS|CVS\.adm|RCS|RCSLOG|SCCS|TAGS|\.svn|supervise|content-repository-content-files)$

You also will need to add a .arch-inventory file to prevent Arch from storing database backups, server certificates, etc... See: https://www.gnu.org/software/gnu-arch/

echo "precious ^log|apm-workspace|rss|database-backup$" > .arch-inventory
echo "precious ^certs$" > etc/.arch-inventory
echo "precious ^down$" > etc/daemontools/.arch-inventory

Add in code that hasn't been added in yet (check without xargs first)

$ tla tree-lint -t |xargs tla add

Do this until tree-lint doesn't complain.

Now we commit the new stable branch:

$ tla import
* imported jade@safe4all.org--2005/openacs--stable--5.1.0
$ tla commit --summary "Imported OpenACS 5.1.0"
* update pristine tree (jade@safe4all.org--2005/openacs--stable--5.1.0--base-0 => openacs--stable--5.1.0--patch-1)
* committed jade@safe4all.org--2005/openacs--stable--5.1.0--patch-1

Set up a dev branch

At this point, we need to set up a dev branch, where we'll use to keep track of new packages we install, and HTML code we're given. At my company, the graphics department works on the HTML code, but there often are modifications I have to make after they have given it to me. For example, I may change a link after installing a package. The dev branch is where I upload their code, and the code for new packages I install via the remote repository.

We want to create a website called safe4all. You'll base this dev instance on the openacs stable version, by creating a branch:

Set up a staging branch

At this point, we need to set up a staging branch, where we'll use to keep track of our local customizations to OpenACS and the HTML code we have. For example, I may change the appearance of the OpenACS pages or packages, or change the wording of something in the OpenACS package.

We want to create a website called safe4all. We base this website on the safe4all-dev branch, but create another branch:

Set up the live server server

The live server then is based on the staging branch: openacs--staging--5.1.0

$ cd 
$ tla register-archive /home/safe4all-dev/\{archives\}/2005/
$ tla my-default-archive jade@safe4all.org--2005

Now set up the live server:

$ cd
$ tla get openacs--staging--5.1.0 safe4all
$ ln -s /home/safe4all/safe4all /var/lib/aolserver/safe4all
$ emacs ~/safe4all/etc/config.tcl
$ emacs ~/safe4all/etc/daemontools/run

Now set up the database. If you are starting from scratch:

Set up the database as per Prepare PostgreSQL for an OpenACS Service on the OpenACS install docs.

If you are importing an existing database, then, if you have a database dump to import, you can use Perl to make sure you are loading the information correctly into the dev and staging instances:

# su - postgres
$ createuser -a -d safe4all
$ exit
# su - safe4all
$ createdb -E UNICODE safe4all
$ perl -pi -e "s/^\\connect olduser$/\\connect safe4all/" dumpfilename.dmp
$ psql -f dumpfilename.dmp safe4all

Add to the crontab entries for safe4all:

$ export EDITOR=emacs;crontab -e

#30 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all > /dev/null 2>&1
30 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all > /dev/null 2>&1

Now let's make sure everything is set up correctly. Run the etc/daemontools/run script by hand.

$ su - 
# cd /home/safe4all/safe4all/etc/daemontools
# ./run

Check that the server actually starts up, look at the error.log files to troubleshoot. Once it works, then:

$ su -
# ln -s /var/lib/aolserver/safe4all/etc/daemontools /service/safe4all
# svgroup web /service/safe4all
# exit
$ svc -u /service/safe4all

When a new version of OpenACS comes out

When updates are made to OpenACS core (such as 5.1.3 coming out), you update it in the oacs archive.

Since we started with OpenACS 5.1.0, I'll now bring in 5.1.1 into the stable branch:

# su - safe4all-dev
$ cd wd
$ cvs -z5 -d :pserver:anonymous@cvs.openacs.org:/cvsroot export -r openacs-5-1-1-final acs-core
$ mv openacs-4 openacs-5-1-1

The old version of OpenACS is at ~/wd/openacs-5-1-0, and the new version is at ~/wd/openacs-5-1-1.

$ rsync -av --delete-after --force --exclude='{arch}' --exclude='**/.arch-ids**' openacs-5-1-1/* openacs-5-1-0
$ cd openacs-5-1-0
$ find . \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | \
   perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_'
$ find . \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | \
   perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_' | xargs rm -rf
$ tla tree-lint -m     (if you get anything do the next line)
$ tla tree-lint -m | xargs rm 
$ tla tree-lint -t | xargs tla add-id
$ tla tree-lint

Make sure tree-lint doesn't complain before continuing.

$ tla commit --summary "Imported OpenACS 5.1.1"

We made the ~/wd/openacs-5-1-0 directory have all the changes from openacs-5-1-1, so just to avoid confusion, let's clean up the working directory:

$ cd ~/wd
$ mv openacs-5-1-1 /tmp
$ mv openacs-5-1-0 openacs-5-1-1

Updating dev with stable changes

First make sure you don't have any local changes to commit:

$ cd safe4all-dev
$ tla changes

If you do have changes to commit, then commit them first, using the directions below.

# su - safe4all-dev
$ cd safe4all-dev
$ tla missing --summary openacs--stable--5.1.0
$ tla replay openacs--stable--5.1.0
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add
$ tla changes 
$ tla log-for-merge >> $(tla make-log)
$ emacs +(TAB)    (edit the log entry)
$ tla commit

Installing and upgrading OpenACS packages

First make sure you don't have any local changes to commit:

$ cd safe4all-dev
$ tla changes

If you do have changes to commit, then commit them first, using the directions below.

I usually check which packages are available using the acs-admin/install page. I don't actually install from there. Here is an example using calendar:

# su - safe4all-dev
$ cd
$ cvs -d:pserver:anonymous@cvs.openacs.org:/cvsroot export -r openacs-5-1-compat calendar 
$ rsync -av --delete-after --force --exclude='{arch}' --exclude='**/.arch-ids**' calendar/* dev-server/packages/calendar
$ find dev-server \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_'

If you see any results from the find command, you can delete those directories as appropriate.

$ cd dev-server
$ tla tree-lint 
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add-id   (may have to do multiple times)
$ tla changes 
$ tla make-log
$ emacs +(TAB)    (edit the log entry)
$ tla commit
$ cd
$ mv calendar /tmp

Updating staging with dev changes

First make sure you don't have any local changes to commit:

$ cd safe4all-staging
$ tla changes

If you do have changes to commit, then commit them first, using the directions below.

# su - safe4all-staging
$ cd safe4all-staging
$ tla missing --summary openacs--dev--5.1.0
$ tla replay openacs--dev--5.1.0

If you get conflicts during the replay, see how to deal with .rej files, which are the files that you get when there are conflicts.

Now commit the changes to the staging branch

$ tla changes 
$ tla log-for-merge >> $(tla make-log)
$ emacs +(TAB)    (edit the log entry)
$ tla commit

Updating the live server

This part is really easy.

$ su - safe4all
$ cd safe4all
$ tla missing -s
$ tla replay

You can make local changes to files, and updates will not overwrite them. So you can edit the etc/config.tcl file for example. If there are conflicts, you will have to resolve them, however.

Making changes on the dev branch

As you recall, the dev branch is for adding new OpenACS packages. You may also want to make local changes such as the change to the etc/config.tcl and etc/daemontools/run file. When you have added code or made any changes, you simply:

$ tla changes --diffs
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add      # (may need to do this multiple times)
$ tla commit --summary "Description of my changes"

Making customizations

When you want to make a customization, you do it on the safe4all-staging server. So first, we bring up the server (leave it off most times):

$ svc -u /service/safe4all-staging

Make your changes.

Commit your changes

$ tla changes 
$ tla tree-lint 
$ tla tree-lint -t
$ tla tree-lint -t |xargs tla add
$ tla commit --summary "Describe my customization here"

You'll have to resolve any conflicts if you find any.

Now that we're done, we'll take down the staging server, then update the live server

$ svc -d /service/safe4all-staging

Resetting the database

If you ever want to load more current data into your staging or dev branches. The Perl is to change the username in the backup dump.

$ su - safe4all-staging
$ svc -d /service/safe4all-staging
$ dropdb safe4all-staging

$ createdb -E UNICODE safe4all-staging
$ cp ~safe4all/safe4all/database-backup/backup.dmp ~/safe4all-staging/database-backup/
$ perl -pi -e "s/^\\connect safe4all$/\\connect safe4all-staging/" backup.dmp
$ psql < backup.dmp safe4all-staging
$ svc -u /service/safe4all-staging

Making upstream contributions from your changes

This section is still being worked on.

If you have a changeset that you'd like to commit to OpenACS, here's how I currently I am doing it:

$ su - safe4all-dev
$ ls -1

You'll see this:

dev-server
oacs-5-1
oacs-HEAD
wd
{archives}

The directory: oacs-5-1 is a cvs checkout of OpenACS, from the oacs-5-1 branch.

Let's say the change I want to commit is patch-8.

$ tla get openacs--dev--5.1.0--patch-7 patch-7
$ tla get openacs--dev--5.1.0--patch-8 patch-8
$ diff -ur patch-7 patch-8 -x '{arch}' > patch-file

Then you can use that patch-file to patch oacs-5-1, using the patch command

$ man patch

Changes to this document

If you want to keep up to date with the changes in this document, you shouldn't have to read every bit of the page and scan it for changes. Just look in this section, and I'll detail what you have to do to get up to the latest changes. For example, if you last used this document on January 12, 2005, then you'll need to make the January 13, 2005 changes. Then you'll be exactly on the same page as everything else in this document.

January 13, 2005:
By default, the dev and staging servers should not start up, and only the production server should. With daemontools, this is accomplished by putting a file called down in the daemontools directory. We want Arch to ignore this.
# su - safe4all-dev $ touch ~safe4all-dev/dev-server/etc/daemontools/down # exit # su - safe4all-staging $ cd staging-server $ touch etc/daemontools/down $ echo "precious ^down$" > etc/daemontools/.arch-inventory $ tla tree-lint $ tla tree-lint -t |xargs tla add (will need to do this several times) $ tla tree-lint -m |xargs rm $ tla make-log (edit the log file) $ tla commit
- December 12, 2005:
  As of Postgres 7.4, you do not need to explicitly run vacuum all the time. So you may need to update your crontab entries:
  # 0 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all-dev > /dev/null 2>&1 0 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all-dev > /dev/null 2>&1
```
This can be done for dev, staging, and production.
```

References to this Page: en:index.html

Using Arch for source control management with OpenACS

Table of contents

Why use Arch

Learning about Arch

How we'll set things up

Initial setup

Setup OpenACS core release mirror (stable branch)

Set up a dev branch

Set up a staging branch

Set up the live server server

When a new version of OpenACS comes out

Updating dev with stable changes

Installing and upgrading OpenACS packages

Updating staging with dev changes

Updating the live server

Making changes on the dev branch

Making customizations

Resetting the database

Making upstream contributions from your changes

Changes to this document