Imagine that you are the technical manager responsible for delivering a dotLRN-backed application. From your initial gap analysis, you find that dotLRN covers an impressive proportion of the requirements for your project. But you also realize that you cannot simply run dotLRN "out of the box": you will need to customize your dotLRN installation.
dotLRN provides you with a number of customization options. Where the out of the box installation does not match your requirements, dotLRN's behavior can often be changed through the OpenACS parameter system. Where dotLRN is missing a key feature area, its clean modular boundaries make it easy to develop collaborating, dotLRN-compatible modules. And in some cases, perhaps, the core dotLRN packages simply do not match your requirements. In this case, because dotLRN is free software, you can modify its source code to meet your needs.
When it comes time to add and modify files, for the sake of your sanity and productivity you should use a source code control system to manage your project's code. The Concurrent Versions System, or CVS, is a good choice. It is free, widely-used and well-documented. The rest of this document covers the details of using CVS for dotLRN development.
For the purposes of this discussion we will assume a fairly specific case, namely that
- You are working from the head of the OpenACS and dotLRN development trees,
- You expect to both add new code and make local edits to base toolkit files,
- You plan to frequently merge the latest OpenACS and dotLRN changes into your tree (ideally replacing your local edits with cleaner, more general solutions as they become available in the base toolkits).
Overview of OpenACS-based code management with CVS
Unlike earlier versions, the OpenACS 4x series---and particularly the dotLRN extensions---present a real possiblility of deployment without editing base project files (that is, the OpenACS and dotLRN distribution code). In this case, development would proceed along the following track:- Untar the distribution code.
- Configure the site parameters through the web interface or a programmatic equivalent
- Load data such as users, departments, classes
- Add presentation-oriented files (master templates, graphics, etc) to base modules.
- Create new, collaborating modules.
We recommend that you use CVS anyway, if only because it allows you produce a complete copy of your project code with a single shell command (useful for setting up new servers). This way, you will also be prepared to track changes to core project files if the need arises (to fix a spelling error, for example). As explained in detail below, the base project files are stored in CVS on a "vendor branch". Updates to the base project, such as new releases, are periodically committed to these branches and integrated into the main development trunk. If no changes have been made to the base files, the process is equivalent to untarring the new code over the old code, with the additional benefit that the changes are being tracked in the local repository. If the base project files have been edited, a code review and conflict resolution step will need to be undertaken to merge the two sets of changes.
The decision of whether or not to edit base project files is a matter of engineering judgement and the answer will vary depending on the project. Be aware that managing multiple, moving code branches is not for the faint of heart. To be successful you will need to have a very good understanding of CVS and a deep understanding of your project's code and its dependencies. You should allocate time for careful review and testing of merge results. If at all possible, you should consider working from QA'ed release code that you will add to but not modify.
CVS Concepts
- Repository. Code management systems like CVS typically distinguish between a central, canonical file database and working copies of the files. The files in the central database, or "repository", are never modified directly. To make changes to the repository, developers must first perform a "checkout".
- Checkout. A checkout is a special kind of copy of the repository files. The checkout carries extra information (in subdirectories named "CVS") that helps CVS track the state of the checked out code relative to the repository. Changes in the checked out files can be saved to the filesystem and tested without affecting the repository. To save to the repository, one must perform the additional step of "committing" the file. In the ACS world, CVS checkouts are most commonly used to run live webservers. However, it is perfectly reasonable to perform CVS checkouts purely for code management tasks such as tagging and merging. You will see examples of this below.
- Tagging. Each time a changed file is committed to the repository, it is assigned a new version number that can be used to track changes. As a project matures, the various files in the project will tend to have different version numbers reflecting different change histories. The need often arises to take a "snaphot" of the code at a given moment in time. This is accomplished by tagging the tree. Tagging amounts to saving a collection of the version numbers for the files in the checkout at the time the tag was applied.
- Branch. It is sometimes useful to maintain related copies of a project. For example, suppose you have just finished release 1.0 of the project and are planning a major refactoring for v2.0. The refactoring involves deep changes that will break the code for a while. In the meantime, you will also need the ability to apply small bugfixes to v1.0, and want to use CVS to track the bugfixes. In short, you need two related, but not identical, committable instances of the project files. In CVS lingo these instances are referred to as "branches".
Step 1: Export and Import
The first step is to get an export of the OpenACS and dotLRN toolkits. An export is a named or dated snapshot of a CVS repository. It is different from a checkout in that you will not be getting a copy of the CVS accounting files associated with the base projects, just the code files themselves. This is what you want, because as a third-party developer you will not be able to commit against the OpenACS or dotLRN trees. Instead, you will manage the core code plus your changes in a local CVS repository, into which you will import the base toolkits shortly. Later, we will show how the CVS concept of vendor branches helps you to merge changes to the base toolkits into your tree.You will prompted for a password. Hit Enter.bash-2.05$ mkdir cvs-workspace bash-2.05$ cd cvs-workspace bash-2.05$ cvs -d :pserver:anonymous@openacs.org:/cvsroot login
You can alternatively usebash-2.05$ cvs -z3 -d :pserver:anonymous@openacs.org:/cvsroot \ export -r HEAD acs-core
-r tagName for a snapshot
matching a particular tag, or -D "YYYY-MM-DD" for a
snapshot of the tree on a particular date.
You should now have an export of the openacs core in the
openacs-4 subdirectory. Now you will export the OpenACS
modules required by dotLRN (check the dotLRN install doc for an up-to-date list).
Now fetch an export of the dotLRN packages.bash-2.05$ ls openacs-4 bash-2.05$ cd openacs-4/packages bash-2.05$ cvs -z3 -d :pserver:anonymous@openacs.org:/cvsroot \ export -r HEAD acs-datetime acs-developer-support acs-events acs-mail-lite \ attachments bulk-mail calendar faq file-storage forums general-comments news \ notifications ref-timezones user-preferences
Now that you have a snapshot of OpenACS and dotLRN, you need to import it into a local CVS repository. If you haven't already, create one (may require root depending on where you want to put it).bash-2.05$ cvs -d :pserver:anonymous@dotlrn.openacs.org:/dotlrn-cvsroot login (hit enter) bash-2.05$ cvs -z3 -d :pserver:anonymous@dotlrn.openacs.org:/dotlrn-cvsroot \ export -r HEAD dotlrn-core
Now import the project.bash-2.05$ sudo cvs -d /usr/local/cvsroot init
You have just created a project calledbash-2.05$ cd .. bash-2.05$ pwd /home/aegrumet/cvs-workspace/openacs-4 bash-2.05$ cvs -d /usr/local/cvsroot import \ -m "OpenACS and dotLRN heads, 2002-06-25" \ dotlrn-backed-proj dotLRN Head_2002_06_25
dotlrn-backed-proj. The initial code was brought in with
vendor tag dotLRN and release tag
Head_2002_06_25.
The following diagram depicts the CVS version information for a hypothetical file in the import:
1.1
\_
\_
\
(1.1.1) dotLRN vendor branch
\
1.1.1.1 tags: Head_2002_06_25
The project should now show up in your CVS repository.
You can now remove the export files.bash-2.05$ ls /usr/local/cvsroot CVSROOT dotlrn-backed-proj
bash-2.05$ cd .. bash-2.05$ pwd /home/aegrumet/cvs-workspace bash-2.05$ rm -rf openacs-4
Step 2: Checkout
In order to modify files and track changes, you will need at least one CVS checkout.bash-2.05$ cvs -d /usr/local/cvsroot co -d /web/dev dotlrn-backed-proj
Step 3: Develop and Commit
Once you've run through the OpenACS and dotLRN install actions (covered elsewhere) you will be ready to add files, edit files and commit against your local checkout.Returning to our example drawing above, commits to a base project file in the local checkout are represented by versions 1.2 and 1.3 below:
| 1.1
| | \_
Your | 1.2 \_
changes | | \
V 1.3 (1.1.1) dotLRN vendor branch
\
1.1.1.1 tags: Head_2002_06_25
Step 4: Reference Checkout (optional)
It can be very useful to have a local checkout available to track activity on the OpenACS and dotLRN code trees backing your project. These commands will create that checkout.bash-2.05$ cvs -z3 -d :pserver:anonymous@openacs.org:/cvsroot co acs-core bash-2.05$ mv openacs-4 cvs-anon bash-2.05$ cd cvs-anon/packages bash-2.05$ cvs -z3 -d :pserver:anonymous@openacs.org:/cvsroot \ co acs-datetime acs-developer-support acs-events acs-mail-lite \ attachments bulk-mail calendar faq file-storage forums general-comments news \ notifications ref-timezones user-preferences bash-2.05$ cvs -z3 -d :pserver:anonymous@dotlrn.openacs.org:/dotlrn-cvsroot \ co dotlrn-core
Step 5: Merge in Third-party Changes (Whole Tree)
From time to time, and perhaps quite often, you will want to merge changes from the OpenACS and dotLRN trees back into your project. To do this you will first need to get an export, just as you did when you started the project.Now import the tree. Note: before conducting this step, you may want to tag the tree or make a note of the date and time of day, should you need to revert the tree to its pre-import state.bash-2.05$ cd ~/cvs-workspace bash-2.05$ rm -rf openacs-4 bash-2.05$ cvs -z3 -d :pserver:anonymous@openacs.org:/cvsroot \ export -r HEAD acs-core bash-2.05$ cd openacs-4/packages bash-2.05$ cvs -z3 -d :pserver:anonymous@openacs.org:/cvsroot \ export -r HEAD acs-datetime acs-developer-support acs-events acs-mail-lite \ bulk-mail calendar faq file-storage forums general-comments news \ notifications ref-timezones user-preferences bash-2.05$ cvs -z3 -d :pserver:anonymous@dotlrn.openacs.org:/dotlrn-cvsroot \ export -r HEAD dotlrn-core
bash-2.05$ cd .. bash-2.05$ pwd /home/aegrumet/cvs-workspace/openacs-4 bash-2.05$ cvs -d /usr/local/cvsroot import \ -m "OpenACS and dotLRN heads, 2002-07-25" \ dotlrn-backed-proj dotLRN Head_2002_07_25
After the second import, the CVS structure will vary depending on the change history by you and the vendor. Here are a few likely possibilities:
1.1
| \_
1.2 \_
| \
1.3 (1.1.1) dotLRN vendor branch
\
1.1.1.1 tags: Head_2002_06_25 |
| | Vendor changes
1.1.1.2 tags: Head_2002_07_25 V
1.1
\_
\_
\
(1.1.1) dotLRN vendor branch
\
1.1.1.1 tags: Head_2002_06_25 |
| | Vendor changes
1.1.1.2 tags: Head_2002_07_25 V
1.1
| \_
1.2 \_
| \
1.3 (1.1.1) dotLRN vendor branch
\
1.1.1.1 tags: Head_2002_06_25
Head_2002_07_25
1.1
\_
\_
\
(1.1.1) dotLRN vendor branch
\
1.1.1.1 tags: Head_2002_06_25
Head_2002_07_25
1.1
\_
\_
\
(1.1.1) dotLRN vendor branch
\
1.1.1.1 tags: Head_2002_07_25
1.1
\_
\_
\
(1.1.1) dotLRN vendor branch
\
1.1.1.1 tags: Head_2002_06_25
Assuming that some of your project's files have been changed by both you and the vendor, the CVS import will return a message similar to the following:
CVS is warning you here that some of your changes conflict with the vendor's changes. To resolve the conflicts, we need to perform a merge. Instead of using cvs' suggestion above, we will use the variation from Karl Fogel's book:14 conflicts created by this import. Use the following command to help the merge: cvs -d /usr/local/cvsroot checkout -jdotLRN:yesterday -jdotLRN dotlrn-backed-proj bash-2.05$
At this point, you have a checkout with conflicts in yourbash-2.05$ cd .. bash-2.05$ pwd /home/aegrumet/cvs-workspace bash-2.05$ cvs -d /usr/local/cvsroot co -kk -j Head_2002_06_25 \ -j Head_2002_07_25 dotlrn-backed-proj
cvs-workspace directory. Before continuing on, let's
pause for a minute and think about what just happened. Because no
-r or -D option was given, CVS has checked
out the HEAD version of our code. In a bizarre twist, "HEAD" in CVS
(v1.11) appears to mean
the tip of the vendor branch (i.e. 1.1.1.x) if you haven't committed anything, otherwise the tip of the trunk (1.x).Also, according to the "double j" options, CVS has computed the changes from
Head_2002_06_25 to
Head_2002_07_25, and applied these changes to the HEAD.
You can now begin to fathom what has changed by switching to the new checkout directory and issuing the command
Here is a more detailed description of what is in the checkout directory:bash-2.05$ cvs -qn update
| Description | Prefix from cvs -qn update |
|---|---|
| File was changed by both you and the vendor (Figure 1 above) | C or M |
| File was changed by the vendor but not you (Figure 2 above) | (file won't be listed) |
| File was changed by you but not the vendor (Figure 3 above) | (file won't be listed) |
| File was changed by neither you nor the vendor (Figure 4 above) | (file won't be listed) |
| File was added by the vendor (Figure 5 above) | (file won't be listed) |
| File was not changed by you and was removed by the vendor (Figure 6 above) | R |
Pause for a moment and note that there may be a significant number of files that have changed as a result of the merge (corresponding to Figures 2, 3 and 5) but aren't listed as "modified" (C or M) by CVS.
Successful completion of the merge requires code review of all the cases listed above, but your first and foremost concern will be to resolve files changed by both you and the vendor (marked C or M). The CVS structure for these files looks like this:
You can inspect the changes to the M files using1.1 | \_ 1.2 \_ | \ 1.3 (1.1.1) dotLRN vendor branch | \ | 1.1.1.1 --- | | | delta | 1.1.1.2 --- | | | checkout<---------------
cvs
diff, which will output changes from your last checked-in
version (1.x).
To inspect changes to the C files, it is important to understand the significance of the conflict markers. The conflict markers represent changes you've made since the last import that do not match the differences applied via the j tags. The markers are inserted into files with the following format:
(adapted from the "Detecting and Resolving Conflicts" section of Karl Fogel's book).<<<<<<< (filename) local (non-vendor) changes in the main development trunk blah blah blah ======= the new changes that came from the merge blah blah blah and so on >>>>>>> (latest revision number in the repository)
Usually, resolving these conflicts will require very careful thought by someone who is intimate with the code. Furthermore, note that CVS is not particularly intelligent. It generates conflicts based on pattern-matching rules and not AI or any notion of how computer languages work. Hence it is possible (and likely) that auto-merged code in "M" files and inside conflict markers in "C" files will not be logically grouped.
To illustrate, consider the following true to life example:
line 1: <li>
2: <if @forums.new_p@><b></if>
3: <a href="@forums.url@forum-view?forum_id=@forums.forum_id@">@forums.name@</a>
4: <<<<<<< forums-portlet.adp
5: <if @forums.new_p@ eq t><img src="/doc/acs-datetime/pics/new.gif" align="absmiddle" border="0" alt="New!" align="baseline"></if>
6: =======
7: <if @forums.new_p@></b></if>
8: >>>>>>> 1.1.1.2
9: </li>
In this example, we have replaced this code
<li>
<if @forums.new_p@><b></if>
<a href="@forums.url@forum-view?forum_id=@forums.forum_id@">@forums.name@</a>
<if @forums.new_p@></b></if>
</li>
<li>
<if @forums.new_p@ eq t><img src="/doc/acs-datetime/pics/new.gif" align="absmiddle" border="0" alt="New!" align="baseline"></if>
</li>
Note from the previous point that C files can also contain
modifications that did not generate conflicts. These changes should
be inspected with cvs diff.
The last things to do are: commit your changes, and update any other checkouts.
A final note: the code merging steps described above do not address the issue of database changes. Such changes must be addressed by examining changes in the SQL data model files and determining what steps must be taken to bring an existing database into sync with the expectations of the latest code.1.1 | \_ 1.2 \_ | \ 1.3 (1.1.1) dotLRN vendor branch | \ | 1.1.1.1 | | | 1.1.1.2 | 1.4 <-- merge conflicts resolved.
Step 6: Merge in Third-party Changes (Partial)
Third-party changes to specific subdirectories may be merged by producing skeleton directories matching the project structure above the subdirectory, placing the updated files in the subdirectory, performing an import as described above, merging and committing.How do I...?
This section will contain a series of questions and the CVS incantations that answer them.More
- Open Source Development with CVS, an excellent, free reference by Karl Fogel
- Subversion, a free alternative to CVS
- Perforce, alternative to CVS that is no-cost for educational and open source projects
- dotLRN installation doc
aegrumet@alum.mit.edu
