Table of contents
Email me if you have corrections or additions to this document.
Why use Arch
I've used
CVS for a couple of years, and even though you can't live without version control, and it's decent, it is not well-suited to web application development. You run into problems with branching, tracking local customizations and keeping track of a changing code stream.
Arch handles these much better than CVS:
- Branching
- Renaming files
- Moving directories
Compared to the competition (Subversion for example):
- Handles distributed development
Subversion is also a very worthy tool -- I use Subversion every day. But I chose to go with Arch for an earlier project. Arch is pretty interesting, and if I didn't occasionally run into annoying error messages I would recommend it more. I also find that the "user interface" is too focused on people who understand the mechanics of how Arch works, rather than on abstracting those details from developers.
Learning about Arch
The best place to start with is
GNU Arch. It has instructions for installing and learning Arch. You should at least look over the tutorial there to make sure you understand how to get help.
How we'll set things up
First of all, we're not going to tackle the more complicated issue of setting up a CVS to Arch gateway. This is very possible, but not something we're going to get into at this point.
We are setting up a very rigorous setup, which will have a
developmental server, a staging server, and the live server. This will allow you to have localized changes to your OpenACS installation, use the remote repository without doing a bunch of CVS chicanery, and in general allow you to test changes before going live with them.
We will set up an OpenACS Arch repository that will mirror the stable releases of OpenACS core (currently 5.1.3). The website we are building is
safe4all.
| User |
Purpose |
Archive |
Category |
Branch |
Version |
Ports |
| safe4all-dev |
Mirror OpenACS core releases |
jade@safe4all.org--2005 |
openacs |
stable |
5.1.0 |
None |
| safe4all-dev |
Import packages, html pages |
jade@safe4all.org--2005 |
openacs |
dev |
5.1.0 |
8000/8443 |
| safe4all-staging |
Local customization |
jade@safe4all.org--2005 |
openacs |
staging |
5.1.0 |
8001/8444 |
| safe4all |
Live site |
jade@safe4all.org--2005 |
openacs |
staging |
5.1.0 |
80/443 |
Initial setup
# groupadd web
# useradd -m -g web safe4all-dev
# useradd -m -g web safe4all-staging
# useradd -m -g web safe4all
# apt-get install tla tla-doc
See the
OpenACS wiki guide to installing OpenACS on Debian. Set up the Arch IDs for all the users
# su - safe4all-dev
$ tla my-id "Jade Rubick <safe4all-dev@safe4all.org>"
$ exit
# su - safe4all-staging
$ tla my-id "Jade Rubick <safe4all-staging@safe4all.org>"
$ exit
# su - safe4all
$ tla my-id "Jade Rubick <safe4all@safe4all.org>"
$ exit
You should also edit your .bash_profile or equivalent to make your umask 0002 instead of 0022. This helps you with permissions issues later. Do this for safe4all-dev and safe4all-staging.
Setup OpenACS core release mirror (stable branch)
What we'll do is track the latest stable release of OpenACS from CVS. Currently, that is 5.1.5. What we will do is start with 5.1.0 (start later if you want to save yourself some time), and upgrade each version to get to the current one. This tests our upgrade process and makes sure we're on the right path.
The basic way we're setting things up is to
setting up your own arch repository.
If you haven't already:
# su - safe4all-dev
$ mkdir ~/{archives}
Now set up your openacs-stable archive.
$ tla make-archive jade@safe4all.org--2005 ~/{archives}/2005
$ tla my-default-archive jade@safe4all.org--2005
$ tla archive-setup openacs--stable--5.1.0
* creating category jade@safe4all.org--2005/openacs
* creating branch jade@safe4all.org--2005/openacs--stable
* creating version jade@safe4all.org--2005/openacs--stable--5.1.0
$ mkdir ~/wd
$ cd ~/wd
$ cvs -z5 -d :pserver:anonymous@cvs.openacs.org:/cvsroot co -r openacs-5-1-0-final acs-core
(see http://openacs.org/4/checkout)
$ mv openacs-4 openacs-5-1-0
$ cd openacs-5-1-0
$ tla init-tree openacs--stable--5.1.0
We have to get all the source into Arch. You should look up the inventory and tree-lint commands if you don't understand what they do. They essentially help us make sure that we're importing all the files into Arch the way we want to.
$tla inventory --names --both --source |xargs tla add
$tla tree-lint
We have to get to a point where tla tree-lint doesn't report any issues. Here are some hints:
We now need to edit the =tagging-method file, to ensure that log files aren't stored in Arch, that the daemontools' supervise directory isn't stored in Arch, etc..
emacs {arch}/=tagging-method
Change this line:
precious ^(\+.*|\.gdbinit|\.#ckpts-lock|=build\.*|=install\.*|CVS|CVS\.adm|RCS|RCSLOG|SCCS|TAGS|\.svn)$
to
precious ^(\+.*|\.gdbinit|\.#ckpts-lock|=build\.*|=install\.*|CVS|CVS\.adm|RCS|RCSLOG|SCCS|TAGS|\.svn|supervise|content-repository-content-files)$
You also will need to add a .arch-inventory file to prevent Arch from storing database backups, server certificates, etc... See:
https://www.gnu.org/software/gnu-arch/
echo "precious ^log|apm-workspace|rss|database-backup$" > .arch-inventory
echo "precious ^certs$" > etc/.arch-inventory
echo "precious ^down$" > etc/daemontools/.arch-inventory
Add in code that hasn't been added in yet (check without xargs first)
$ tla tree-lint -t |xargs tla add
Do this until tree-lint doesn't complain.
Now we commit the new stable branch:
$ tla import
* imported jade@safe4all.org--2005/openacs--stable--5.1.0
$ tla commit --summary "Imported OpenACS 5.1.0"
* update pristine tree (jade@safe4all.org--2005/openacs--stable--5.1.0--base-0 => openacs--stable--5.1.0--patch-1)
* committed jade@safe4all.org--2005/openacs--stable--5.1.0--patch-1
Set up a dev branch
At this point, we need to set up a dev branch, where we'll use to keep track of new packages we install, and HTML code we're given. At my company, the graphics department works on the HTML code, but there often are modifications I have to make
after they have given it to me. For example, I may change a link after installing a package. The dev branch is where I upload their code, and the code for new packages I install via the remote repository.
We want to create a website called
safe4all. You'll base this dev instance on the openacs stable version, by creating a
branch:
See also
Elementary branches.
$ cd ~wd
$ tla archive-setup openacs--dev--5.1.0
* creating branch jade@safe4all.org--2005/openacs--dev
* creating version jade@safe4all.org--2005/openacs--dev--5.1.0
$ tla tag jade@safe4all.org--2005/openacs--stable--5.1.0 openacs--dev--5.1.0
Now we have a dev branch of the openacs-stable archive.
Now you can set up a dev server. This step is not strictly necessary. When you edit your config.tcl file, I recommend setting up the email redirecting, so email isn't sent from your staging server. See "Configure an AOLserver Service for OpenACS" on
the OpenACS documentation.
$ cd
$ tla get openacs--dev--5.1.0 safe4all-dev
* from import revision: jade@safe4all.org--2005/openacs--stable--5.1.0--base-0
* patching for revision: jade@safe4all.org--2005/openacs--stable--5.1.0--patch-1
* patching for revision: jade@safe4all.org--2005/openacs--dev--5.1.0--base-0
* making pristine copy
* tree version set jade@safe4all.org--2005/openacs--dev--5.1.0
$ ln -s /home/safe4all-dev/safe4all-dev /var/lib/aolserver/safe4all-dev
$ mkdir ~/safe4all-dev/log
$ emacs ~/safe4all-dev/etc/config.tcl
$ emacs ~/safe4all-dev/etc/daemontools/run
$ touch ~/safe4all-dev/etc/daemontools/down
Now set up the database. If you are starting from scratch:
Set up the database as per Prepare PostgreSQL for an OpenACS Service on the OpenACS install docs.
If you are importing an existing database, then, if you have a database dump to import, you can use Perl to make sure you are loading the information correctly into the dev and staging instances:
# su - postgres
$ createuser -a -d safe4all-dev
$ exit
# su - safe4all-dev
$ createdb -E UNICODE safe4all-dev
$ perl -pi -e "s/^\\connect olduser$/\\connect safe4all-dev/" dumpfilename.dmp
$ psql -f dumpfilename.dmp safe4all-dev
$ locate vacuum |grep vacuumdb
$ export EDITOR=emacs;crontab -e
Put this in the crontab, changing the directory to pgsql depending on what which said for vacuum. Note the first line is commented out -- it is no longer necessary as of Postgres 7.4.
# 0 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all-dev > /dev/null 2>&1
0 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all-dev > /dev/null 2>&1
Now let's make sure everything is set up correctly. Run the etc/daemontools/run script by hand.
$ su -
# cd /home/safe4all-dev/safe4all-dev/etc/daemontools
# ./run
If you get this error message:
/usr/local/aolserver/bin/nsd: error while loading shared libraries: libnsd.so: cannot open shared object file: No such file or directory
See:
error while loading shared libraries: libnsd.so
Check that the server actually starts up, look at the ~safe4all-dev/log/error.log files to troubleshoot. Post on the OpenACS Forums if you have problems. Once it works, then link it in.
$ su -
# ln -s /var/lib/aolserver/safe4all-dev/etc/daemontools /service/safe4all-dev
# svgroup web /service/safe4all-dev
# exit
$ svc -u /service/safe4all-dev
If the svgroup command doesn't work, you need to install it (see the OpenACS installation guide (
here).
Once you get it right, commit the changes you've made. You can go through the web interface now and set up OpenACS if you didn't import anything.
Set up a staging branch
At this point, we need to set up a staging branch, where we'll use to keep track of our local customizations to OpenACS and the HTML code we have. For example, I may change the appearance of the OpenACS pages or packages, or change the wording of something in the OpenACS package.
We want to create a website called
safe4all. We base this website on the safe4all-dev branch, but create another
branch:
See also
Elementary branches.
# su - safe4all-staging
$ cd
$ tla register-archive /home/safe4all-dev/\{archives\}/2005/
$ tla my-default-archive jade@safe4all.org--2005
$ tla archive-setup openacs--staging--5.1.0
* creating branch jade@safe4all.org--2005/openacs--staging
* creating version jade@safe4all.org--2005/openacs--staging--5.1.0
$ tla tag jade@safe4all.org--2005/openacs--dev--5.1.0 openacs--staging--5.1.0
Now we have a staging branch of the dev branch.
Now set up the staging server:
$ cd
$ tla get openacs--staging--5.1.0 safe4all-staging
* from pristine cache: jade@safe4all.org--2005/openacs--dev--5.1.0--base-0
* patching for revision: jade@safe4all.org--2005/openacs--staging--5.1.0--base-0
* making pristine copy
* tree version set jade@safe4all.org--2005/openacs--staging--5.1.0
$ ln -s /home/safe4all-staging/safe4all-staging /var/lib/aolserver/safe4all-staging
$ emacs ~/safe4all-staging/etc/config.tcl
$ emacs ~/safe4all-staging/etc/daemontools/run
$ touch ~/safe4all-staging/etc/daemontools/down
When you edit your config.tcl file, I recommend setting up the email redirecting, so email isn't sent from your staging server.
Now set up the database. If you are starting from scratch:
Set up the database as per Prepare PostgreSQL for an OpenACS Service on the OpenACS install docs.
If you are importing an existing database, then, if you have a database dump to import, you can use Perl to make sure you are loading the information correctly into the dev and staging instances:
# su - postgres
$ createuser -a -d safe4all-staging
$ exit
# su - safe4all-staging
$ createdb -E UNICODE safe4all-staging
$ perl -pi -e "s/^\\connect olduser$/\\connect safe4all-staging/" dumpfilename.dmp
$ psql -f dumpfilename.dmp safe4all-staging
$ export EDITOR=emacs;crontab -e
Add to the crontab entries for safe4all-staging:
#15 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all-staging > /dev/null 2>&1
15 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all-staging > /dev/null 2>&1
Now let's make sure everything is set up correctly. Run the etc/daemontools/run script by hand.
$ su -
# cd /home/safe4all-staging/safe4all-staging/etc/daemontools
# ./run
Check that the server actually starts up, look at the error.log files to troubleshoot. Once it works, then:
$ su -
# ln -s /var/lib/aolserver/safe4all-staging/etc/daemontools /service/safe4all-staging
# svgroup web /service/safe4all-staging
# exit
$ svc -u /service/safe4all-staging
Set up the live server server
The live server then is based on the staging branch: openacs--staging--5.1.0
$ cd
$ tla register-archive /home/safe4all-dev/\{archives\}/2005/
$ tla my-default-archive jade@safe4all.org--2005
Now set up the live server:
$ cd
$ tla get openacs--staging--5.1.0 safe4all
$ ln -s /home/safe4all/safe4all /var/lib/aolserver/safe4all
$ emacs ~/safe4all/etc/config.tcl
$ emacs ~/safe4all/etc/daemontools/run
Now set up the database. If you are starting from scratch:
Set up the database as per Prepare PostgreSQL for an OpenACS Service on the OpenACS install docs.
If you are importing an existing database, then, if you have a database dump to import, you can use Perl to make sure you are loading the information correctly into the dev and staging instances:
# su - postgres
$ createuser -a -d safe4all
$ exit
# su - safe4all
$ createdb -E UNICODE safe4all
$ perl -pi -e "s/^\\connect olduser$/\\connect safe4all/" dumpfilename.dmp
$ psql -f dumpfilename.dmp safe4all
Add to the crontab entries for safe4all:
$ export EDITOR=emacs;crontab -e
#30 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all > /dev/null 2>&1
30 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all > /dev/null 2>&1
Now let's make sure everything is set up correctly. Run the etc/daemontools/run script by hand.
$ su -
# cd /home/safe4all/safe4all/etc/daemontools
# ./run
Check that the server actually starts up, look at the error.log files to troubleshoot. Once it works, then:
$ su -
# ln -s /var/lib/aolserver/safe4all/etc/daemontools /service/safe4all
# svgroup web /service/safe4all
# exit
$ svc -u /service/safe4all
When a new version of OpenACS comes out
When updates are made to OpenACS core (such as 5.1.3 coming out), you update it in the oacs archive.
Since we started with OpenACS 5.1.0, I'll now bring in 5.1.1 into the stable branch:
# su - safe4all-dev
$ cd wd
$ cvs -z5 -d :pserver:anonymous@cvs.openacs.org:/cvsroot export -r openacs-5-1-1-final acs-core
$ mv openacs-4 openacs-5-1-1
The old version of OpenACS is at ~/wd/openacs-5-1-0, and the new version is at ~/wd/openacs-5-1-1.
$ rsync -av --delete-after --force --exclude='{arch}' --exclude='**/.arch-ids**' openacs-5-1-1/* openacs-5-1-0
$ cd openacs-5-1-0
$ find . \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | \
perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_'
$ find . \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | \
perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_' | xargs rm -rf
$ tla tree-lint -m (if you get anything do the next line)
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t | xargs tla add-id
$ tla tree-lint
Make sure tree-lint doesn't complain before continuing.
$ tla commit --summary "Imported OpenACS 5.1.1"
We made the ~/wd/openacs-5-1-0 directory have all the changes from openacs-5-1-1, so just to avoid confusion, let's clean up the working directory:
$ cd ~/wd
$ mv openacs-5-1-1 /tmp
$ mv openacs-5-1-0 openacs-5-1-1
Updating dev with stable changes
First make sure you don't have any local changes to commit:
$ cd safe4all-dev
$ tla changes
If you do have changes to commit, then commit them first, using the
directions below.
# su - safe4all-dev
$ cd safe4all-dev
$ tla missing --summary openacs--stable--5.1.0
$ tla replay openacs--stable--5.1.0
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add
$ tla changes
$ tla log-for-merge >> $(tla make-log)
$ emacs +(TAB) (edit the log entry)
$ tla commit
Installing and upgrading OpenACS packages
First make sure you don't have any local changes to commit:
$ cd safe4all-dev
$ tla changes
If you do have changes to commit, then commit them first, using the
directions below.
I usually check which packages are available using the acs-admin/install page. I don't actually install from there. Here is an example using calendar:
# su - safe4all-dev
$ cd
$ cvs -d:pserver:anonymous@cvs.openacs.org:/cvsroot export -r openacs-5-1-compat calendar
$ rsync -av --delete-after --force --exclude='{arch}' --exclude='**/.arch-ids**' calendar/* dev-server/packages/calendar
$ find dev-server \( -name {arch} -o -name ".arch-ids" \) -prune -o -print | sort | perl -ne 'chomp; print "$o\n" if -d $o && !/^\Q$o\E/; $o=$_'
If you see any results from the find command, you can delete those directories as appropriate.
$ cd dev-server
$ tla tree-lint
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add-id (may have to do multiple times)
$ tla changes
$ tla make-log
$ emacs +(TAB) (edit the log entry)
$ tla commit
$ cd
$ mv calendar /tmp
Updating staging with dev changes
First make sure you don't have any local changes to commit:
$ cd safe4all-staging
$ tla changes
If you do have changes to commit, then commit them first, using the
directions below.
# su - safe4all-staging
$ cd safe4all-staging
$ tla missing --summary openacs--dev--5.1.0
$ tla replay openacs--dev--5.1.0
If you get conflicts during the replay, see how to
deal with .rej files, which are the files that you get when there are conflicts.
Now commit the changes to the staging branch
$ tla changes
$ tla log-for-merge >> $(tla make-log)
$ emacs +(TAB) (edit the log entry)
$ tla commit
Updating the live server
This part is really easy.
$ su - safe4all
$ cd safe4all
$ tla missing -s
$ tla replay
You can make local changes to files, and updates will not overwrite them. So you can edit the etc/config.tcl file for example. If there are conflicts, you will have to resolve them, however.
Making changes on the dev branch
As you recall, the dev branch is for adding new OpenACS packages. You may also want to make local changes such as the change to the etc/config.tcl and etc/daemontools/run file. When you have added code or made any changes, you simply:
$ tla changes --diffs
$ tla tree-lint -m | xargs rm
$ tla tree-lint -t |xargs tla add # (may need to do this multiple times)
$ tla commit --summary "Description of my changes"
Making customizations
When you want to make a customization, you do it on the safe4all-staging server. So first, we bring up the server (leave it off most times):
$ svc -u /service/safe4all-staging
Make your changes.
Commit your changes
$ tla changes
$ tla tree-lint
$ tla tree-lint -t
$ tla tree-lint -t |xargs tla add
$ tla commit --summary "Describe my customization here"
You'll have to resolve any conflicts if you find any.
Now that we're done, we'll take down the staging server, then update the live server
$ svc -d /service/safe4all-staging
Resetting the database
If you ever want to load more current data into your staging or dev branches. The Perl is to change the username in the backup dump.
$ su - safe4all-staging
$ svc -d /service/safe4all-staging
$ dropdb safe4all-staging
$ createdb -E UNICODE safe4all-staging
$ cp ~safe4all/safe4all/database-backup/backup.dmp ~/safe4all-staging/database-backup/
$ perl -pi -e "s/^\\connect safe4all$/\\connect safe4all-staging/" backup.dmp
$ psql < backup.dmp safe4all-staging
$ svc -u /service/safe4all-staging
Making upstream contributions from your changes
This section is still being worked on.
If you have a changeset that you'd like to commit to OpenACS, here's how I currently I am doing it:
$ su - safe4all-dev
$ ls -1
You'll see this:
dev-server
oacs-5-1
oacs-HEAD
wd
{archives}
The directory: oacs-5-1 is a cvs checkout of OpenACS, from the oacs-5-1 branch.
Let's say the change I want to commit is patch-8.
$ tla get openacs--dev--5.1.0--patch-7 patch-7
$ tla get openacs--dev--5.1.0--patch-8 patch-8
$ diff -ur patch-7 patch-8 -x '{arch}' > patch-file
Then you can use that patch-file to patch oacs-5-1, using the patch command
$ man patch
Changes to this document
If you want to keep up to date with the changes in this document, you shouldn't have to read every bit of the page and scan it for changes. Just look in this section, and I'll detail what you have to do to get up to the latest changes. For example, if you last used this document on January 12, 2005, then you'll need to make the January 13, 2005 changes. Then you'll be exactly on the same page as everything else in this document.
- January 13, 2005:
By default, the dev and staging servers should not start up, and only the production server should. With daemontools, this is accomplished by putting a file called down in the daemontools directory. We want Arch to ignore this.
# su - safe4all-dev $ touch ~safe4all-dev/dev-server/etc/daemontools/down # exit # su - safe4all-staging $ cd staging-server $ touch etc/daemontools/down $ echo "precious ^down$" > etc/daemontools/.arch-inventory $ tla tree-lint $ tla tree-lint -t |xargs tla add (will need to do this several times) $ tla tree-lint -m |xargs rm $ tla make-log (edit the log file) $ tla commit
- December 12, 2005:
As of Postgres 7.4, you do not need to explicitly run vacuum all the time. So you may need to update your crontab entries:
# 0 1-23 * * * /usr/lib/postgresql/bin/vacuumdb --analyze safe4all-dev > /dev/null 2>&1 0 0 * * * /usr/lib/postgresql/bin/vacuumdb --full --analyze safe4all-dev > /dev/null 2>&1
This can be done for dev, staging, and production.
