Forum OpenACS Q&A: What *should* a good architecture Look Like?

1: What *should* a good architecture Look Like?

Posted by Brent Fulgham on 04/13/00 10:16 PM

Since OO techniques seem unpopular, what *would* a more modular ACS
look like?

From my perspective (as someone trying to make this easily
installable/upgradable) I want to see the following:

1. The ability to get bug-fixes inserted into my working website.
This implies a separation between the presentation and the ACS
framework (data paths, transactions, queries, etc.)

2. The ability to add new features to a working website. This
implies that functionality be provided in modules that can be used
independently, or in concert, to achieve certain goals.

And a final goal (requirement) I would have is:

3. The ability to provide a consistent look-and-feel throughout the
site without having to hand-edit every page in the system. This
implies a way of templating the presentation, or providing site-wide
customizations that are registered/recognized throughout the site by
the disparate pieces of the system.

We might have one lemma implied by the above, which would be that:

4. The existance of "Release" programs to move an existing data set
from one form to another. I.e., if a data model changes, it should
be possible to craft programs/scripts that would reorganize or
correct the data as required to move from one known state of the ACS
to another.

So, let's discuss *these* issues, and leave the language wars for
comp.lang.perl.misc. 😊

2: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Brent Fulgham on 04/14/00 12:33 AM

After writing the above, I was thinking that one possible way of managing the ACS's complexity would be to involve a packaging system. This could be something like the Emacs packaging system, or the Debian *.deb, or the Red Hat *.rpm. Internally this would be a tarfile containing some kind of install script that could be used to place the components in the right places. It could also modify some kind of registry to indicate that module XYZ is now present and available for use.

This would allow different modules of the ACS to be inserted at a later time. However, this doesn't deal with the troublesome issue of cross-linkage between many pieces of the ACS.

3: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Kevin Dangoor on 04/14/00 12:40 AM

Cross-linkage seems inevitable to me, because a big system, especially an open source one, is likely to be built from many smaller components written and maintained by different people. But, if you can at least tell the user what the dependencies are, that would be a big plus.

4: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Michael Feldstein on 05/02/00 02:59 PM

There's an interested document on aD's developer site regarding their own progress toward a packaging system:

http://dev.arsdigita.com/doc/packaging

5: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Don Baccus on 05/02/00 05:17 PM

Thanks, Michael. It looks like there's going to be a 3.3, rather than nothing until 4.0. The beginnings of the package manager will be included in this. We'll have to take a look to see whether it's worth the effort to incorporate this, or whether perhaps we should simply wait for 4.0. 4.0 will include a total revamping of the directory layout of the ACS (by package, rather than functionality) so our semi-automated approach of using diff and patch isn't going to help much.

4.0 will be a big effort, we may be better off trying to solidify 3.2.2 as much as possible and do what we can to set aside a fairly large chunk of time to port 4.0 over.

Then again, 3.3 may be a simple upgrade. We'll just have to see.

6: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Ben Adida on 05/02/00 05:21 PM

3.3 will not be a simple upgrade, unfortunately. Just found out
that the new templating scheme (karl.arsdigita.com) will be
included in 3.3.

I agree with solidifying 3.2.2. Let's make it a final version. We'll
see about 3.3 when it comes out.

7: How 'bout that templating system? (response to 1)

Posted by Janine Ohmer on 05/03/00 04:35 PM

This is a bit of a digression, but I'm curious what folks think of that new templating system?

8: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Don Baccus on 05/04/00 02:26 AM

Janine, if you're talking about the Karl Goldstein one, the last document I saw by Philip (on Feb 28) made it sound as though this wouldn't be adopted ACS-wide. There would be support for it in the overall design "because Karl and Cynthia seem to like it", but templating per se will be handled differently.

I only read this about three-four days ago, had meant to drop you a line and a pointer. Poke around here, Michael Feldstein posted a pointer to it in one of these forums (yes, we know we need to get site-wide searches working). I think it was him. He spends some of his time poking around the aD site looking for interesting stuff.

9: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Michael Feldstein on 05/04/00 02:31 AM

Yes, I do poke around aD when I'm not busy poking around here. 😉

It looks like Karl's system is going to be rolled into ACS 3.3. Here'ss the latest:
http://karl.arsdigita.com/doc/publish/content/index.adp

10: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Michael Feldstein on 05/04/00 03:00 AM

Actually, that last link was just to the content repository component. The description of the whole system is here:

http://karl.arsdigita.com/doc/publish/index.adp

It looks pretty extensive.

11: Templating system (response to 1)

Posted by Michael A. Cleverly on 05/04/00 06:35 AM

If Karl's templating system is going into 3.3, but won't be *the* templating system for 4.0, does anyone have any pointers as to what *the* 4.0 system will be?

From looking at Karl's system, I'm quite impressed. I haven't tried it yet, though I'm tempted to tomorrow, time permitting... I've got a couple legacy AOLserver (non-OpenACS) sites that could really benefit from it.

12: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Don Baccus on 05/04/00 06:43 AM

Heck, Michael did have a pointer to the Grander Scheme (as I mentioned earlier). It was a Philip piece talking about the entire templating problem and was quite comprehensive. Karl's scheme's pretty complex, perhaps this is why aD is looking for a less ornate way to deliver the functionality they need.

Michael, do you have the link I'm thinking of handy?

BTW, let's all thank Michael for his digging up this stuff for us. Digging through the aD site isn't on my daily agenda, though perhaps it should be. Feldstein seems to be finding everything worth knowing about. That's useful.

13: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Janine Ohmer on 05/04/00 04:30 PM

Yea Don, "complex" and "ornate" are two good words to describe it. I read through the docs a while back before deciding to go with the Ybos CM system (Karl has a CM system built on his templating code in there too). It was a beautiful design, and would make a great master's thesis or something. :) But I had a hard time imagining myself using it in a real production project. Too many files, too much extra typing. Maybe I'm just old-fashioned, but I'd rather deal with page layout and Tcl code in one file than that!

I do think we need a good templating scheme but it needs to be simple, and not much more time-consuming than the current all-in-one approach, or it's just not going to get used.

14: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Michael Feldstein on 05/04/00 05:08 PM

Don, I'm not sure which reference you mean, but it might be http://dev.arsdigita.com/doc/templating-etc.

Janine, I think there's a need for both approaches. Some of my clients would find the more "ornate" system very useful. They develop large quantities of fairly detailed (and interactive) online training/educational material with folks collaborating across (world-wide) organizations. Their workflow process is very messy and the technological savvy of the contributors varies enormously. The content needs to be easily localized/internationalized. In this sort of a situation, Karl's system would be just the ticket. Admittedly, most of these clients would want to implement this in ACS Classic behind their fire walls. However, I foresee the day when they will go to an ASP for this sort of a thing, in which case running it on OpenACS would work just great (provided that Postgres continues to scale up).

15: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Michael Feldstein on 05/04/00 11:52 PM

OK, here's a better link to a document that describes the system-wide templating system for ACS. It sounds like they're already beginning to implement it.

http://photo.net/doc/style.html

16: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Michael A. Cleverly on 05/05/00 12:03 AM

http://dev.arsdigita.com/doc/templating-etc points to http://photo.net/doc/style.html as "the June 1999 system" which ACS 4.0 is supposed to improve upon. Only a few modules actually attempt to make any use of the existing templating system.

17: Yet another link to aD templating stuff (response to 1)

Posted by Michael Feldstein on 05/05/00 02:14 AM

Here's yet another one:

http://dev.arsdigita.com/acs40/tagged-content.html

Apparently, there will be an effort to at least partly implement this in ACS 3.3.

18: Response to What *should* a good architecture Look Like? (response to 1)

Posted by David Richards on 05/31/00 10:35 PM

Last Saturday I hired Roberto Mello to give me a swift kick in the butt and get me going in the right direction with the openACS. He's incredibly good at what he does. I was very impressed with his work and preparation. If I were his boss, I'd be sure he was making at least 6 figures. I had previously read hundreds of web pages on the ACS and philip's book. I'm starting to get really comfortable with the system and I'm glad to be participating in this endeavor with the rest of you.

In other words, I'm kind of a newbie at some of this, although I've done what I can to this point to make a meaningful contribution.

One issue that seems to plague me as I try to sleep and wakes me up in the morning is the issue you've been discussing on this forum--site-wide maintenance of presentation. I know that at least three of the possibilities have been discussed on this thread, and a fourth (introducing Python to the openACS) is mentioned on another thread.

I don't give the Python idea too much consideration because I believe there are at least three other possibilities that would be more practical, maintainable, etc. Python may have some wonderful redeeming features. In my mind, however, I can't justify the extra effort of a major conversion, and then the extra effort of keeping the openACS in sync with ACS. I understand that you can encapsulate Tcl in Python, but when the ACS is upgraded, the upgrade can be more than just code changes. It can also be file system changes, and other things that would be harder to port to the openACS if the openACS were encapsulated with Python.

Sorry for the long digression.

The following is a list of what I think I can do to build a unique look and feel on an ACS-based site:

1) I can copy or edit each page as built by aD to your own specifications. With this approach, I'd need to keep these changes current manually as bug fixes and new realeases are implemented.

2) I can create content sections to edit the basics--background color, top section, and a side bar. I haven't really tested this much, but I'm assuming this would be the easiest way to implement a moderate divergence from the plain-vanilla site.

3) I can create my own ad_header and ad_footer procs. ad_header can cover the top and left-hand side of the page. ad_footer can cover the bottom and right-hand side of the page. I can get tricky with the Tcl procs I build and adjust the type of look and feel based on user, user group, content section, etc. Naturally, I'd be starting from scratch with this approach, so I'd have to think up my own architecture and maintenance.

4) I can mess with the templating stuff in Beta 3. I've just glossed over some of this stuff from the admin pages. It seems that this builds a bunch of adp files for each template. There's something with forms also. I'm really drawing a blank right now. (I'm far from my working system...enough excuses. Sorry.)

5) I could use Karl Goldstein's templating system as has been mentioned on this discussion board. I understand that I'll have access to a partial implementation of Karl's ideas with ACS 3.3. From one or two comments on this thread, it sounds like I can actually get his code today. I haven't dug deep enough into this to confirm or deny this. (anybody feeling authoratative on this issue?)

6) I can work with XML as defined in http://dev.arsdigita.com/doc/xml.html. (A doc by Karl Goldstein). I think this is different than Karl's templating system. I believe that the templating solution doesn't use XML, although it seems that he's shared a lot of underlying ideas between the two ideas. This XML approach requires the implementation of an XML parser (which I'm assuming isn't readily available in postgres). It sounds like this solution would require some effort in building a generalized architecture and administrative views of the system. (Is there any more work in this direction? Am I wrong in any of my assumptions? If so, it wouldn't be the first time!)

7) I could wait for ACS 4.0 and its subsequent port to the openACS, whatever that turns out looking like. I know that the major links for this approach are probably http://photo.net/doc/style.html, http://dev.arsdigita.com/doc/templating-etc. This would mean you build style tags and use them in your adp files. Also, it looks like it uses some or all of Karl's templating ideas. I guess everything from 3.3 and later will be pushed into packages instead of modules, offering a hierarchal presentation of a site.

My main concern is do I have these options clear in my mind? Are there more options than this?

If I've got a clear idea of my options, it seems that the only truely feasable options are 5, 6, and 7. They seem to be the only comprehensive approach to the problem that might have future support somewhere. The safest move seems to be to wait for 4.0. Since I'm impatient, I'll probably be building something with either of Karl Goldstein's ideas and then see if I can port my site into 4.0--if 4.0 offers all the functionality I'm going to require.

So, am I completely off base? Do you disagree? If I surf around Karl's stuff, can I actually find Tcl code and data models that will implement some or all of his ideas? Or do I need to build based on the specs he writes about? (I've only had time to read the overview of his templating solution.)

Sorry about the length of this post. It's a brain dump of all my thoughts and concerns on the issue.

19: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Don Baccus on 05/31/00 11:04 PM

Someone posted in web/db that they easily moved Karl's stuff to Solid, so perhaps moving it to Postgres won't be difficult.

It would be nice to have it up so we can play with it and get some experience with it. Hint hint :)

It would be nicer to find out what aD plans to do in this area long-term. I've not dug around in a few weeks, but last time I did it appeared that they've not settled on a single approach.

20: Response to What *should* a good architecture Look Like? (Ask and ye shall receive...) (response to 1)

Posted by Michael Feldstein on 05/31/00 11:10 PM

The source code to Karl's system is here:

http://karl.arsdigita.com/projects/template/doc/code.adp

21: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Michael A. Cleverly on 05/31/00 11:18 PM

I was the one who posted about getting Karl's stuff to work on Solid (in this other OpenACS forum thread). Getting it to work on Solid consisted of:

Downloading the source (each file individually rather than the tarball)
Installing 00-ad-preload.tcl and the *.preload files from the ACS/OpenACS Tcl library
Naming my database pools main, subquery, and log
Following the rest of Karl's installation instructions (i.e. .ini file settings)

Since OpenACS already has the right files in the Tcl private directory, I bet you'll find you can simply download and the drop the source right into place and be off and running. I only had to make one change to his code for Solid (where he gets todays date by querying dual). I just changed it to return [ns_fmttime [ns_time] "%Y-%m-%d"] instead. I've made one or two other tweaks to add a bit of functionality, but nothing much.

I'm guessing, from looking at Karl's site, that the full-fledged DPS 1.0 will eventually *not* be database agnostic, but to date, the templating ascpets at least are. I'm tied up with work today, but if nobody beats me to it, I'll grab the latest OpenACS source from sourceforge and see how well Karl's DPS 0.4 drops in to place this weekend.

I'm interested in hearing what other people have to say about David's post above. Also, Ben, do you (or does anyone else) have any insight into where aD will actually end up going with 4.0?

22: Response to What *should* a good architecture Look Like? (response to 1)

Posted by David Richards on 06/01/00 08:27 AM

That sounds great! I think I'll get a bit of sleep, and get an early crack at that code. If I'm lucky, I might actually get something working by the time you guys wake up. I'll let you know where I'm at before the weekend so we can pass the baton if that makes sense.

Thanks you guys.

Also, just something on my mind. There's some doctor, I can't remember his name, who has done research into sleep cycles. He says the typical person's sleep cycle is about an hour and a half. (when you finish rem and are ready to start stage one again.) So, if you give yourself sleep in increments of an hour and a half (if that's your cycle--it is mine) then you won't be fighting your body so much. Also, Thomas Jefferson did some experimenting with this. If I remember right, he found something similar. So, I'll be working on that code in either 3 or 4 1/2 hours. Good night.

23: Response to What *should* a good architecture Look Like? (response to 1)

Posted by David Richards on 06/02/00 08:43 AM

I've been working on a conversion of Karl Goldstein's 0.4 version of his DPS (Dynamic Publishing System, or templates--he's not consistent with his naming conventions). I've got a few things to report and ask.

First, I must not have been there when I first read the overview to the DPS--because he certainly uses XML. So, my post above seems a little bird brained.

Second, I've got most of the module implemented on my test system. There is actually a bit more to it than the 4 step process mentioned in the Solid post above. Those manual downloads are just the Tcl libraries. He's got an installation directory that gives you more code, and more complete instructions (there's seven or eight steps to the installation).

Third, I'm stuck. Well, not really. I'm tired, and if I can't get things going better in the morning, then I'm stuck. I'm converting the data model tonight, and I've got an outer join that looks too hairy for me tonight. It is as follows (I've messed with the formatting a little so this is more readable, I'm big on using white space for readability, but this messge is getting formatted funny):

create or replace view ad_site_node_page_map_ext as

select

p.page_id, p.page_title, p.page_url, p.description as page_description,

npm.node_id, npm.prev_page_id,

sn.parent_id, sn.asset_stub, sn.description, sn.nav_title

from ad_site_pages p, ad_site_node_page_map npm, ad_site_nodes sn

where p.page_id = npm.page_id (+)

and sn.node_id = npm.node_id;

I'm sure I could figure it out eventually, but maybe someone more keen on these things can offer some advice. I'll work on it in the morning if nobody responds.

Also, I did some diffs between ACS code and openACS code. It seems that with the openACS views, you don't use create or replace. You just use replace. Is that an oversite, or is that required?

I read the conversion document, and it didn't mention varchar2 data types. As far as I know, that's an Oracle idea, right?

Is connect by on the PostgreSQL to do list? Just checking. That sure would be nice if it were there.

Anyway, to sum things up. There were only a couple of changes to the data model for the DPS. 2 outer joins, one connect by, and two clobs. I've just touched the actual code working on my server a little. Without the data model, you can't really go too far. If anyone else is working on this right now, let me know.

Finally, can I request a code review from someone more senior with these things? I would just like someone to take a look at an outer join conversion. Without any data in the data model, it might take me a little time to prove my join does what I think it should be doing. Maybe a quick glance by someone accustomed to this stuff would be valuable.

Here's the original join:

create or replace view ad_site_node_index_pages as

select sn.node_id, sn.parent_id, sn.nav_title, sn.asset_stub,

p.page_title, p.page_url

from ad_site_nodes sn, ad_site_pages p

where sn.index_page_id = p.page_id (+);

And here's my join:

create or replace view ad_site_node_index_pages as

select sn.node_id, sn.parent_id, sn.nav_title, sn.asset_stub,

p.page_title, p.page_url

from ad_site_nodes sn, ad_site_pages p

where sn.index_page_id = p.page_id

union select sn.node_id, sn.parent_id, sn.nav_title, sn.asset_stub, null as page_title, null as page_url

from ad_site_nodes sn

where 0= (select count(*) from ad_site_pages p where p.page_id = sn.index_page_id);

Thanks

24: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Janine Ohmer on 06/02/00 02:39 PM

We used David's option #3 (customize ad_header and ad_footer) in a site we're just finishing. Since we renamed the procs and added a parameter to the header, it required editing every user-visible file. Yow, was that a lot of work, and it's still not quite done.

In short - it works, but I don't recommend it. I knew that going in but as tedious as it is it's all we had time for. The client insisted he wanted the plain ACS look until very late in the project, then we had to jam something in when he changed his mind. I was glad he changed his mind, I think the ACS is *too* plain for a commercial site, but I wish he'd done it earlier!

25: Correction to my steps for getting DPS 0.4 working on Solid (response to 1)

Posted by Michael A. Cleverly on 06/02/00 04:18 PM

Yes, I did leave out a step... I untarred the tarball (which gives all the non-Tcl stuff under /template), and then replaced each .tcl file in the Tcl lib individually with the then-current sources from Karl's site, since I couldn't get the 0.4 tarball ones to work my first pass with them.

The templating and form systems seem to work well. I've not attempted to make the site map stuff work at all on Solid (yet).

26: Response to What *should* a good architecture Look Like? (response to 1)

Posted by Don Baccus on 06/02/00 05:12 PM

Regarding outer joins, "not exists (select ... " should be marginally more efficient than "0 = (select count(*) ... "

Clobs should become lztext, which is used just like varchar(). varchar2(n) should become either varchar(n) for small strings or lztext for bit strings where you don't want to state a limit.

Connect by is not on the PostgreSQL "todo" list, sorry :(

27: Response to What *should* a good architecture Look Like? (response to 1)

Posted by David Richards on 06/05/00 06:14 PM

I just thought I'd give you guys some information I got this morning from Karl Goldstein. This is a copy of an email he returned.

Hi David,

<blockquote>>>>> "David" == David Richards <mailto:geni100@yahoo.com> writes:
</blockquote>

David> Hello, I'm kind of a newbie in the ACS world. I've spent 5
David> or 6 months reading Philip's book, loading and configuring
David> the toolkit, and doing this and that with it. I've kind of
David> migrated over to the openACS partially for now--I'm just
David> small fries today, and so are my clients.

David> Over at the openACS, there's a lot of discussion about
David> adding an abstract presentation layer. Your solution seems
David> to be the favorite among all informed. However, we have
David> some questions regarding what will actually be implemented
David> in ACS 4.0. Since we need to convert those parts of the
David> toolkit that are Oracle specific, we are a little wary
David> about which ACS versions are converted, which add-on
David> modules are converted, etc.

Yea, this business with Oracle/postgres is agonizing. I admire you
guys for trying to deal with it.

So, as far as you know, will
David> your Dynamic Publishing System 1.0 be a package in ACS 4.0?

Yes, DPS will be two modules starting with ACS 3.3: templates and
form manager.

David> Or will 4.0 take another aproach to this abstraction layer?
David> Right now, the best we can do is just search the Arsdigita
David> pages for any clues as to where this part of the ACS is
David> headed.

David> Also, I've been doing a conversion of your 0.4 code because
David> I've got to decide on a working model to use really quickly
David> here. I know that on your installation page you warned us
David> of using the DPS with AOLserver 3.0--that it's buggy, etc.
David> Is this still the case? Any specific areas I could take a
David> look at?

I've written a patch for the ADP parser that fixes the specific
problems I was encountering. It's part of the AOLserver binary
release and also available from

http://karl.arsdigita.com/doc/publish/install.adp

David> Finally, after following your installation instructions, I
David> was left with 2 files in the /template directory of your
David> source. index.tcl and template.el. I haven't had time to
David> look at those yet. Do they belong somewhere?

template.el is a sort of abortive attempt to write some macros to
make it easier to write the spec files in emacs. index.tcl is
just a redirect to the doc directory...

David> Thanks,

David> David Richards