Forum OpenACS Q&A: Design of an "Automatic Software Update Service"

Hi,

did somebody of you consider to implement an automatic software update service for OpenACS? Or is there already some code around?

Our idea is to provide relatively "non-technical users" with a software update service very much like "Yast Online Update" that automatically checks for available updates and automates the task of patching the system.

Such a service would go beyond the (great!) APM service, because it would automatically check for new software versions, download them and update the /packages/* folder. Then it would redirect to /acs-admin/apm/.

There are several considerations:

1. Updates using CVS or via .apm download: Both could be an option. Updates via CVS would have the advantage that (small) user modifications might be preserved. APMs are already somehow integrated into the current APM. => I think CVS would be easier to implement, because a "cvs update -r xxx" would be sufficient.

2. File format to store version information: XML is probably best suited, but are there already standard formats (DTDs) for this type of information? => We'd probably go for a simple format with a list of versions composed of version name, release date, description and some additional fields.

3. Security and Authentication: Is there a (relatively) easy way to introduce a server authentication so that the updating user can be sure he doesn't download any spoofed stuff?

4. Backup and Roll-Back: Such an update service should have the option to roll-back the changes. Should it? The code could be rolled back by issuing another "cvs udpate -r xxx", so we basicly would have to remind the user to perform a database backup.

5. Different Update Sources or CVS Servers: We are hosting the Project/Open module on a CVS different from OpenACS, because we have some proprietary modules ("ISO 9000 Translation Quality" and an automatic "Translation RFQ/RFC Marketplace"). So there would be updates from two different CVS servers. Is this P/O specific or is it the generic case? => We'd probably include the "URL" of the CVS server for each module to be updated.

Just some ideas... I think the main question is whether to use .APM files or CVS.

Bests,
Frank

Collapse
Posted by Malte Sussdorff on
Your are forgetting some issues:

6) What happens if there is custom code. Currently the APM approach will overwrite any custom code and it's gone. As for CVS, it might generate conflicts. You could do a check if any files in the package you are about to update have been changed and if yes send the administrator a notification that the package is available for upgrade, but can't be upgraded due to changes made locally (similar to how apt-get works).

7) We have a tendency of not being very good with releasing packages, as it requires to retag the package every now and then. So the APM approach would require the developers to tag packages on a regular basis. On the other hand, if you go with the CVS approach, you might break your site even in the "stable" tree, because you are updating without dependencies (as you might depend on a function in the kernel that has been added but not resulted in a new release of the kernel package).

My recommendation therefore is this:

- Add a "sources" repository to the APM which is a list of repositories where APM can retrieve the files from.
- Use APM files instead of CVS
- Check before the upgrade if it is safe to install (we should be doing this anyway)
- Write a collector that retrieves the CVS messages between the previous release and the current one and stores them in a file "latest_release.changes" that is displayed to the user before upgrade.
- Write a TIP for it.
- Collect gratitude :).

I don't think a rollback will work, especially not an automated one. For sure you could execute a database backup in advance, but do you really want the users to loose their data because of an upgrade they did not like?

So yes, I do like your idea, but it is not the piece of cake as it might sound.

Collapse
Posted by Torben Brosten on
I think user-level automatic software updating would be most practical for sites where significant customizations only occur in the package/package-key/www and siteroot/www sections, mainly with the adp pages. Otherwise there are too many factors that need to be considered before an update, so that cvs / arch or whatever becomes the most practical tool.

I am considering writing a "global" etp (G-ETP) package that edits pages by first locating the adp file from within the siteroot/www tree, such as /subsite1/foo.adp. If no file exists, then the editor copies the files (necessary to make the page work) from the package-key/www/foo.adp associated with the url, and places them at siteroot/www/subsite1/foo.* for editing in a large textarea (like ETP).

An editor like this would be really convenient for end-user-admins because general templating can be modified to specific instances. A side benefit of using the siteroot/www tree is that APM updates will not affect the pages. The G-ETP could diff changes on request, validate etc.

Modifying sites this way would help developers to know when to put business-logic in package-key/tcl or /package-key/lib while retaining the features of templating for end-user-admins in the package-key/www area.

This is sort of how mac.com handles template editing (with the addition of webdav).

The main problem I see is that there is no current provision for the APM to "watch" or "load changed" xql files in the siteroot/www tree. I have added a feature request in the bugtracker.

Collapse
Posted by Torben Brosten on

Digressing on the G-ETP idea a bit further..

..instead of copying the .tcl from the package-key/www, a tcl file in the serverroot/www tree could probably (untested) source the original, and include new variable calculations etc so that most changes via apm will automatically update the tcl, too, while retaining customized tcl code.

# foo.tcl
# calculations before sourcing original file (if any)

# source original file
source /var/lib/aolserver/service1/packages/package-key/www/foo.tcl

# calculations after sourcing original file
Collapse
Posted by Frank Bergmann on
Hi,

user-level automatic software updating [...] where
significant customizations only occur in the
package/package-key/www and siteroot/www sections

The Project/Open case is relatively simple, because users probably won't modify much of the code. That's possible because the P/O "core" architecture has a lot of flexibility built-in, such as dynamic views and the option to aggregate data to an existing object.

Our main focus is to distribute small bug-fixes.

not the piece of cake as it might sound

Right. I see that this is definitely not going to be a solution for generic OpenACS software updates. Maybe the "Arch" version management system is the better solution if you have heavy customization?

I'm still unsure...

Thanks for the feedback,
Frank

Collapse
Posted by Frank Bergmann on
Hi,

just to let you know: We've implemented two new packages "intranet-update-server" and "intranet-update-client". This combination is really only a convenient frontend for CVS. The server sends a list of update options the client together with some suitable CVS commands that are executed on the client side.

- The server part provides a XML file with information about available updates.
- The client part parses the XML file, shows the content as a HTML table and does a "CVS update -r Xxxx" or whatever as specified in the

That's it. Using CVS allows to have some(!) changes/ customizations on the client side that might survive an update. That's not perfect but it's already something.

Somebody interested in the code? The code is going to be part of Project/Open V3.0.0 which is going to come out hopefully(!) this weekend.

Bests,
Frank

Collapse
Posted by Malte Sussdorff on
Wouldn't it make sense to use cvs update based on the date of the last update? I assume your intranet-cvs-client stores when a package has last been updated and you could use this information to retrieve the changelogs. Then the user could decide if he wants to upgrade. Obviously the latest release tag should show up in the history as well, so the user could decide only to go to the last release.
Collapse
Posted by Frank Bergmann on
Hi Malte,

well, we send an "hand made" XML file to the intranet-update-client which is rendered as a HTML table.
So the user can actually decide whether to upgrade his system.

Also, we want to use the system to announce the availability of new modules (such as the wiki module recently), so that we woudln't always want to update every user to every available update.

So the process is basicly manual and gives the decision power to the user. That simplifies the code and is sufficiently flexible to deal with all kind of strange exceptions etc.

Bests,
Frank

Collapse
Posted by Malte Sussdorff on
How is the user going to make an informed decision. If you have a hand made XML this implies that you will store some information on the intranet-update-server for every new code bit. This is a lot of maintenance.

That's why I mentioned that cvs dates. All entries to the CVS logs between the dates should be available as a comment the user can see before he decides whether he wants to upgrade. This is considerably less maintenance than releasing packages and storing release notes with them. Furthermore it gives you the flexibility of not having to release so often :).

Collapse
Posted by Frank Bergmann on
Hi Malte,

I understand your points, believe me.

How is the user going to make an informed decision

Every update option has a link to a specific Sourceforge forum where users can discuss the upgrade amongst themselves or give their opinion after a (hopefully!) successful update.

This is considerably less maintenance than releasing
packages and storing release notes with them. Furthermore
it gives you the flexibility of not having to release so
often :).

Good thinking. I had some very similar thinking ... 😊

Check it out:
http://www.project-open.com/product/services/software-update/.

You will see that our design is very much covering your points. However, we can offer the user to perform "major updates" of the entire OpenACS system (like: "cvs update -r openacs-5-1-5") AND we can offer the user to install a new package (like: "cvs checkout intranet-wiki").

Obviously, any Linux admin could execute the same commands manually via CVS. However, we at Project/Open have to deal with "technically interested users" who generally don't even know about the existence of CVS... So this is also why we are going to ask for money for this service.

Checkitout man, checkitout,
Frank

Collapse
Posted by Malte Sussdorff on
You are funny. How should I check it out ?

No seriously, I will only look at it in more detail if there is a chance that we can add it to the OpenACS repository so users could easily upgrade OpenACS from the repository. Which involves looking at the code, which you may or may not provide under the GPL, but I doubt it at the moment as you want to make money of this service. Furthermore, what is important to us in that regard is the ability to upgrade based on DIFFS between the last upgrade and now. I don't see this with your system (you are release dependend).

So yes, your work is a valuable addition to Project/Open and will make live easier for the normal user. If it can be valuable for OpenACS depends on the release of the code and commitment by one or more users to adapt the code with the OpenACS repository.

Collapse
Posted by Frank Bergmann on
Hi Malte,

How should I check it out?

I meant: Just check the link out, which will lead you to our marketing blurb about the service, which includes a screenshot. This screenshot should explain everything.

provide under the GPL

No problem with the GPL. You can grap the current code and release it under the GPL. We are going to sell the _service_ (i.e. the service of providing an updated "hand made" XML file), not the code. Both modules ("intranet-update-client" and "intranet-update-server") are available on our CVS:

cvs -d :pserver:mailto:anonymous@berlin.dnsalias.com:/home/cvsroot checkout intranet-update-client

upgrade based on DIFFS

Just add this to the existing stuff. Our code is pretty small and straight forward, some 1200 lines in total for both modules. We are not going to use this for our customers, because "diff lines" would just scare them off ... 😊

I hope this could become a reasonable contribution from us to the OpenACS community, together with the Windows installer.

Bests,
Frank