Forum OpenACS Development: ACS Roadmap from aD

1: ACS Roadmap from aD - worth a look for OpenACSers

Posted by Don Baccus on 04/06/01 06:25 PM

Actually, I'll post two convenient links.

The first a wimpypoint presentation that was given out as background during yesterday's conference call with aD. It is the definitive word on aD's current plans regarding ACS Tcl and ACS Java. It doesn't contain any surprises for those who've been following the situation. It's goes into considerable detail about the internal allocation of resources made to make those plans reality.

Embedded within this is a link to an internal paper regarding a change in how HTML quoting will be handled in the upcoming (and last) ACS Tcl 4.x release.

The changes should be easy to incorporate (and are important). I don't know if we can get them quickly but even if we have to wait diff and patch should be able to handle most of them.

2: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Kjell Wooding on 04/08/01 10:20 PM

One of the more disturbing things I took from that presentation was the move away from CVS for the core engineering team. CVS tends to encourage more openness in the core, as the CVS tree tends to be available between formal releases.

(As a side not, I'm not sure I understand the comment that CVS is suitable mainly to small, geographically close teams. Most of the projects I'm involved with that use CVS are exactly the opposite)

Does this mean that only formal releases are going to be available
to us pesky outsiders? Wait and see, I suppose...

3: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Don Baccus on 04/08/01 10:22 PM

Why not ask them directly in web/db?

4: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Lars Pind on 04/09/01 11:34 PM

The move to Perforce was for technical reasons only. Perforce lets you rename and move files without loosing the history of the file, it remembers what merges have been made, so you don't as easily screw up by repeating or leaving out a merge, it commits atomically and it remembers what changes were committed together, so it's easier to undo a change that affects multiple files, and, probably the most important thing in daily life, "p4 sync", the analogous to "cvs update", usually only takes fractions of a second, compared to the minutes it can take with a heavily branched cvs tree.

The issue of perforce vs. cvs and the issue of anonymous access to the source code tree are not related. Tehcnically, it's not hard to get the soruce code from p4 and into cvs, so the tree can be accessed via anonymous read-only cvs access.

Ask Richard Buck about actually making that happen.

5: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Don Baccus on 04/09/01 11:37 PM

Thanks, Lars ...

6: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Roberto Mello on 04/09/01 11:55 PM

Frankly Lars, it looks like aD needs to buy a CVS book and learn how to use it. Most of the things you are saying you are using Perforce for are perfectly doable with CVS. It looks like the "technical reasons" you mention are more brochure-related than anything else.

And the comment (and rationale) that "CVS is an excellent open-source SCM that works very well for relatively small, geographically close development teams i.e., teams of less than ten people who work in the same office" is, excuse me for the language, a load of crap.

There are hundreds (maybe thousands) of projects in the world that are _exactly the opposite_ of what is in the above comment, and hence prove that is very wrong. Take Samba for example. The team is spread all over the globe, from Australia to the U.S. The PostgreSQL project has developers everywhere too, from the U.S to Japan. Unless my geography is really screwed up, and the world has been remapped, I don't think that classifies as "geographically close".

And I have never experienced such long times to update my copy of the repository. For me to sync the entire OpenACS tree with its over 7000 files it takes just a few seconds. Never minutes.

I have nothing against aD using Perforce or whatever it feels like. It's their business and they put their trust in the code they want. My comment here is related to the inaccurate, misleading comments about CVS. Use the tools you want, but be fare about other tools and the reasons for the switch.

7: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Albert Langer on 04/10/01 01:08 AM

Since only the aD internal developers are using perforce to commit
anyway, seems to me all that's needed is a gateway that mirrors
from aD's perforce to something that loads it into cvs (one-way)
so that people interested don't need to wait for major releases to see what's going on.

Perforce licensing is free for open source projects anyway.

http://www.perforce.com/perforce/price.html

Are there any technical problems with OpenACS setting up such a gateway, or policy problems with read only access from aD?

If no such problems, not much point discussing the merits of the decision rather than just setting up the mirror/gateway.

8: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Ron Henderson on 04/10/01 02:28 AM

We had a lot of internal debate about switching from CVS to Perforce within the engineering group (NB: only a relatively small sub-group within ArsDigita is using Perforce). However, the arguments that Lars outlined above are accurate. It's true that virtually anything you can do in CVS is possible in Perforce, and vice versa. However, atomic commits are considered a real advantage, as are the way that Perforce handles branches.

Note that "atomic" here refers to atomicity across changes to multiple files. If I commit changes to 15 files at once then p4 treats the entire set of changes as one atomic unit, while CVS handles this file by file.

Branches are advertised as a real advantage of p4. Personally, I don't like their branching model that much, but it is definitely more efficient. CVS only stores the head trunk revision as a complete copy of the file so that working on the trunk is fast; it stores everything else (including branch revisions) as a diff, and gets painfully slow when you accumulate a large number of changes on a branch. From what I can tell, p4 treats each branch as a separate little trunk and introduces a different model for tracking branches. This means that working on a p4 branch is as fast as working on the CVS trunk, you just have to learn their new semantics. P4 also tracks which changes have been merged to which branches so that it's harder to generate spurious conflicts by merging the same thing twice.

I think most other features are a toss-up, and I still have a slight preference for CVS just because I've used it longer.

However, this should all be irrelevant because we hope to mirror the p4 depot in a CVS repository in the near future so that it's available for browsing and for anonymous checkouts. We have technical and non-technical issues to work out first, but hopefully this will happen soon.

9: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Don Baccus on 04/10/01 02:30 AM

If aD sets up a CVS tree for current sources for ACS 5 we'd probably just point people to it - they'd provide CVS browsing via the web and all that just like they've done before, I imagine. No reason for us to duplicate that...

ACS 4 sources are a bit more problematic ... we've forked the source so they're no longer and synch and it is unlikely they'll be in synch ever again. It will be up to us to import any improvements or bug fixes.

This isn't a big deal - we knew that very little future work would be done by aD on the Tcl version when we decided to fork, so we knew we weren't setting ourselves up for the hairy administration problem that one might think would result from the decision.

"folks in the know" are going to be more interested in OpenACS 4.x than aD 4.x - multi-db support's a popular idea.

10: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Roberto Mello on 04/10/01 03:46 AM

On my previous post to this thread, I'd just like to say that I meant nothing bad to Lars, or to bad mouth him, or anything like that. Lars has been a responsive person in aD and it's nice to have him around here.

Ron, thanks for the clarification. I see the slowness point now. I wonder if there's a way to setup CVS to keep a copy of branches. There probably is. I just found out that you can do ACLs with CVS a few minutes ago.

I guess the harshness of my post comes from frustration with some of aD's actions (and that "cvs in good for small teams" blew me away).

11: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Albert Langer on 04/10/01 06:52 AM

I'm confused about OpenACS Roadmap and relation to
ACS Roadmap. Still catching up with bboard postings. Could
anyone point me to specific messages/threads that would
get me clear faster on OpenACS Roadmap?

Re Don's comment on having "forked" already and:

""folks in the know" are going to be more interested in OpenACS 4.x than aD 4.x - multi-db support's a popular idea."

and related comments under "current short term happenings".

My impression was that the Tcl could just as well be described
as "picked up and continued" rather than "forked" in the light
of the ACS Roadmap confirming further development is effectively
discontinued at aD. Either way it will obviously diverge since one will continue developing and the other won't, so that's just a matter of terminology.

But I also thought the initial OpenACS 4 Oracle DDL would be largely identical to the ACS 4 Tcl Oracle DDL and ACS 4 java Oracle DDL, so no "fork" there yet. Have I got that wrong?

If that's true about Oracle DDL then current short term situation looks to me like OpenACS 4 PostgreSQL DDL is still a "port" of the SQL rather than a "fork" of the SQL. Same for DQL and DML seems implied by concept of Query Dispatcher. That isn't just a difference in terminology.

The SQL and Tcl or java are after all running in separate processes connected only via protocol adaptor interfaces, so at least in *theory*, one *could* fork without the other doing so if they still maintained a common abstract data model that had implementations for different RDBMS SQL dialects on one side of the interface and different web application platforms on the other side (plus relatively minor interface adaptations on each side for the purpose of maintaining that common abstract data model given the specifics of each implementation).

It may turn out that will prove impracticable or undesirable. But as far as I can see it is the *current* situation for Oracle SQL that can be used with either Tcl or java and could possibly become the situation with PostgreSQL SQL that could be used with either Tcl or java (or python 😉 *if* a deliberate choice was made to maintain a common data model, whereas it will diverge if a deliberate choice is made to fork the data model rather than port it in both directions.

I would have thought that if aD is interested in possible future
database independence from Oracle, it would be likely to roll the Query Dispatcher concept and the PostgreSQL port of SQL back into ACS5 or earlier when doing the more abstract database API for java. The macro processor might help with that, just as it helps with migration from ACS 4 Tcl to ACS 4 java.

Even if aD is more interested in MS Sqlserver than PostgreSQL in the short term, a general approach that can extend to PostgreSQL once they do go for database independence, would seem just as logical for ACS as for OpenACS. After all PostgreSQL was far less suitable for commercial contracts before than it is becoming now.

So if aD *is* aiming for (eventual) DBMS independence with a more abstract database API for java, I don't see why people wanting
multi-db support would necessarily find OpenACS 4.x more interesting than ACS 4.x. Developers wanting AOLserver or OpenNSD Tcl support would, but why should users be more interested in that, than in the range of modules and features available from each?

The Roadmap for ACS says:

"Oracle (with SQL abstraction support *shortly*)" (emphasis added) in a column that is headed "ACS 4.x, 5 Java". It doesn't seem to imply that this won't happen until ACS 5.

Is there some thread I should read confirming that, (and explaining why), a definate decision to fork the SQL (as opposed to Tcl) has already been taken?

12: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Richard Li on 04/10/01 05:55 PM

To follow up on Ron's response to Roberto, you might want to check out Perforce's CVS to Perforce comparison. I also want to reiterate that the change to Perforce is motivated by purely technical reasons. I was one of the managers of the CVS tree, and I can say that the lack of atomic checkins is a huge headache. In order to keep the tree relatively clean, we had to resort to things like developer-based lazy branching, do very careful checking after landing branches to prevent silent merge deaths, and do poor man's atomicity with CVS tags. In CVS, before committing a major change, you tag before and you tag after so you can rollback, but in Perforce, it's just one checkin. Moreover, you can apply that checkin to multiple branches, which is something you can't do in CVS. I'm not saying that CVS is a bad system, and that it's not workable (as Roberto points out, Samba and Postgres are two examples where people are using it successfully). We just decided that the cost of maintaining a CVS tree is too high given the number of developers that we have. We have 40+ developers with commit rights on the tree. Samba has 20; Postgres appears to have 18. And yes, anon checkout via either CVS or Perforce is coming soon :).

13: Response to ACS Roadmap from aD - worth a look for OpenACSers (response to 1)

Posted by Don Baccus on 04/10/01 06:08 PM

My impression was that the Tcl could just as well be described as "picked up and continued" rather than "forked" in the light of the ACS Roadmap

Almost right but slightly premature. aD *will* release a Tcl 4.3, and one aspect is fairly significant - changing of how HTML is quoted. I've read the doc (from the German office) and I agree with their analysis and changes. We'll want to track them. Hopefully we can get this rolled in by the OpenACS 4.x release time.

That's going to be most of the change for 4.3, it rates a new version number because customized pages may have to be changed to work with the new quoting rules.

Other than that, your comment's substantially correct. Ad will fix bugs that hit client sites, and do nothing more.

But I also thought the initial OpenACS 4 Oracle DDL would be largely identical to the ACS 4 Tcl Oracle DDL and ACS 4 java Oracle DDL, so no "fork" there yet. Have I got that wrong?

Nope, you're right. Some details may change, i.e. the APM datamodel's different because it knows about package support for different databases now (the XML .info file is changed, too). We'll be enhancing the content repository to allow in-filesystem as well as in-database storage of binary content, and that will change the datamodel. There may be other slight enhancements and of course we'll fix bugs if we find them (which are usually in .tcl code, not the DDL, though).

"Oracle (with SQL abstraction support *shortly*)" (emphasis added) in a column that is headed "ACS 4.x, 5 Java". It doesn't seem to imply that this won't happen until ACS 5.

All they mean by this is that their class design for the Java version will abstract out the interface to persistent storage, and operations on stuff stored there. They won't actually be supporting another database for many, many months. They don't plan to roll a full-featured ACS Java 4 release but rather work towards ACS Java 5. This may not be entirely clear from the roadmap but seemed very clear in the conference call.

One terminology clarification, the only ACS 4.x that is going to exist in full-blossomed form is the Tcl version. Since that's a recent decision there's still a lot of stuff lying around talking about an ACS Java 4.x full-featured release before ACS 5, and there's the existing core release of course.

So when I say that folks will find OpenACS 4.x more interesting than ACS 4.x, there's an implied "Tcl" adjective in each of those labels. I'm not saying that more folks will necessarily be interested in OpenACS Tcl than ACS Java ("OpenACS 4.x vs. ACS 5").

aD's change in path isn't that huge, they're just accelerating ACS 5 development by cutting ACS Tcl work off a bit more quickly than folks thought would happen a few months ago, and skipping the building of a fully fleshed-out ACS Java 4.x. It's left us with confusing terminology, though, when you go off and read threads, docs, talk on the phone, etc.

14: About the CVS fuss (response to 1)

Posted by Pierre Phaneuf on 09/20/01 08:05 PM

We just decided that the cost of maintaining a CVS tree is too high given the number of developers that we have. We have 40+ developers with commit rights on the tree. Samba has 20; Postgres appears to have 18.

Don't forget to tell that to the Mozilla folks: there's hundreds of people with CVS commit rights on a tree that I think is sizeably larger than the ACS!

Forum OpenACS Development: ACS Roadmap from aD - worth a look for OpenACSers