Forum OpenACS Q&A: Re: How can I find the code that picks up packages from cvs?

A useful contribution to the would be work on the preliminaries, such as: Currently the git structure is composed of 305 separate git repositories (if i counted correctly). In order to generate the .apm files for all packages in all release channels (oacs terminology), one has to clone all these repositories and bring it to an appropriate structure. This leads us to the "git packaging" question (modules and wrappers, branches, tagging).

Maybe one option is build a wrapper git module, which contains e.g. the better maintained 78 packages currently in the oacs-5-8 branch, using maybe git submodules [1]. This cloud be useful for setting up a small website with OpenACS common packages. But is this for such sites any better than using the acs-core + "install from repository"?

For installed sites, the 78 packages are just a partial help. We have e.g. on our learn@wu system 148 OpenACS packages, including most of the 78 packages above, but many of these are modified, and rest are own development. All of these packages are in git since 2008, with their own history, branches, tags etc. We want to use feature branches etc. I am pretty sure, other large installed sites have a similar situation. Since we have to rethink packaging with git, we should come up with some kind of recommendation, guidelines, development workflows.

What is the best way to setup the git repository structure for large installations? Are submodules the best way? Is this performance-wise good enough? What to do with cross-submodules commits, maintenance work (cross submodules tagging, branching) etc. Although submodules look from some distance perfectly suited for packages, they are in many respects inconvenient (see e.g. [2, 3]), alternatives are e.g. git subtree [4] or subrepo [5]. It would be usefule to evaluate the options to some depth to come up with a founded recommendation.

As for apm-generation: we need a way to get all packages from the 305 repos, which require a consistent tagging / branching structure to offer the right packages/package upgrades to site admins using different OpenACS releases. Is this feasible to enforce this in a highly decentralized way?

The big question is, what we do want to achieve.
What are the important use-cases?
- first install of OpenACS? scm does not matter.
- site admins: install from repo should be sufficient
- sites developing/contributing own packages: the workflow based on the package manager is a first step
- users who want to adapt the code: it would be nice to switch from a "tar checkout" to a "git checkout" via a click in the package manager and to contribute code back also via package manager.
- sites maintaining modified versions of core packages: this happens and will happen, but from our own experience i would be more happy, when improvement would go more often into the common source base than in local modifications
- regular openacs-development: this is my least concern.

Probably there are much more aspects to this.

-g

[1] http://bast.fr/talks/git/submodules/
[2] http://somethingsinistral.net/blog/git-submodules-are-probably-not-the-answer/
[3] https://codingkilledthecat.wordpress.com/2012/04/28/why-your-company-shouldnt-use-git-submodules/
[4] http://blogs.atlassian.com/2013/05/alternatives-to-git-submodule-git-subtree/
[5] https://github.com/ingydotnet/git-subrepo/

- There is the main repository which includes the directories of ACS_ROOT, but no packages.
- Packages are configured using a pkgs-list.txt file, which tells checkout-modules to get the packages as submodules
- I usually have a branch for the main repository for each client / installation. This way I can point to special commits in the packages, depending on the client.
- If I make changes to a package for a client, I usually create a quick branch for that feature in the package and point to it from the clients main repository branch. Once fully tested and in production I merge the branch back to master for the package.
- Whenever I work for a client on a package, I make a pull from master for that package to incorporate all the latest changes.
- Any change to a package which is client specific is a very bad idea and will result in myself kicking myself and moving this code into a client package, utilizing any of the methods for customization (Paramaters, Callbacks).
- There is an update-code.sh package which upgrades to the latest version for the client (e.g. If I released code from my dev system and want to update the demo system). This is actually a link from the admin pages.
- there is an addition update-submodules.sh code which updates my dev system to the latest version of the submodules.

This workflow pretty much covers what Gustaf mentioned under the first 5 bullet points.

In addition to that I maintain one installation where I have scripts that pull the remote code from the ]project-open[ CVS repository clone into my package repositories, something I dread once per quarter (or so). This basically is updating my (heavily changed) code base with the latest changes done in ]project-open[. This last part though is more of interest if you keep your own Cloned repositories from the OpenACS repositories if you don't have the ability to commit all your code back (for whatever the reason).

If you have questions on the workflow, let me know. The update-code / update-submodules would be the equivalent to "update from repository" which the APM packages try to do.

In theory though adding the link to check out the master for each submodule should be fairly easy, yet it isn't really the switch from tar checkout to git checkout that is mentioned above. It would require a git submodule to be in place.

I only use branches to be honest as I import the tags from CVS (the code which moves CVS code to git adds branches and TAGS) and I don't need additional tags for my workflow, especially if you work with submodules and point to actual commits for each package.

As for subtree, I actually started with subtree (and still have a version with subtree in place). Yet I ran into issues with custom packages with client specific code and the main repository which contains all the subtrees and keep that in sync with the various branches for development, testing and production. Ultimately I gave up and switched to submodules, never looking back. But I do have code which transforms ]project-open[ and OpenACS into ONE repository using subtree as well (if that is of interest).

What you suggest would be the GIT subtree approach and maybe a mix and match is the correct answer (aka. GIT subtree for "Distributions" like OpenACS Core, dotLRN, various project open installations and the GIT sumbodules for anything installed in addition to the distribution.

Downside: a distribution can only be updated and branched as a whole, disallowing the independent package by package modifications or package based branches.