Forum OpenACS Improvement Proposals (TIPs): TIP #34 (Expired): Application package ui separation

1: TIP #34 (Expired): Application package ui separation

Posted by Lee Denison on 12/03/03 07:55 PM

Separation of an applications user interface from its functionality has two main advantages:

Flexibility in allowing the functionality to be presented in several ways; eg. using forums as a news and announcements list.
Encapsulating local installation ui customisations in a separate package from the functionality providing clear separation and lower maintenance overhead.

I propose that applications should in general be split into two which would provide the advantages above:

The first package containing the datamodel, tcl api and reusable includes in package-key/lib.
The second containing mainly user viewable pages which form the ui but allowing small extension datamodel and tcl functionality.

The main issues with doing this that I have thought of so far are:

Converting the ui of existing packages to re-usable includes according to some conventions that amount to a design pattern.
Proposed solution: This would have to be a gradual process with packages being updated as people found time.
Making package installation work and the mount points make sense. Application functionality packages would not be able to rely on the fact that they are mounted and would need some way to find out where their ui is mounted.
Proposed solution: A mapping service would relate package instances to their ui instances (maybe allowing more than one?)
Conversely a ui package would not be able to rely on ad_conn in its current form and would need some way for getting it's functionality package.
Proposed solution: The mapping service above.
Upgradability
Proposed solution: Any package converted to this approach would provide a default ui in its own www directory.

I'm working on a project which ideally would like to use the forums package with two different customised ui's so I'm looking for ways to keep the long term maintenance overhead as low as possible.

2: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Benjamin Bytheway on 12/04/03 12:12 AM

I would like this solution as well. I have a need for a much more customized forums solution that what the current package provides, both in functionality and UI. I had thought about ways to provide different look-and-feel options, but I like this the best.

If this gets approved, I'd love to help make it happen for forums.

3: Re: TIP #34 (Proposed): Application package ui separation (response to 2)

Posted by Jeff Davis on 12/04/03 12:21 AM

I think it's worth looking at what Jun has done with the bcms stuff since I think it follows this model. Also, this is in essence what happens when something is portalized in dotlrn (and if we make things like admin portals and cms portals it would happen there too). We end up needing page fragments that can be included in other packages and designing with this in mind from the start would be a big win.

4: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Dave Bauer on 12/04/03 04:28 AM

I don't think that the service package should be tied to the site-node. I can't see any reason to do that. To mount an application with your model, one would mount an instance of the user interface package.

Why wouldn't an application be able to rely on being mounted? I am not sure what you mean here.

Why would a UI package not use ad_conn? Without a connection there is no web interface.

I'd like to see more detail on how this would work. I agree with the concept.

One related idea being worked on it relating site-nodes and content folders. If content under a site node is related to a folder, it is not as important which package created the content.

5: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Lee Denison on 12/04/03 12:31 PM

Dave,

I had the functionality package tied to a site-nodes because I was thinking of a scenario in which several ui's pointed to the same instance of a forum; ie. the forums functionality package is not a singleton. However given, as you point out, that most of the problems I was thinking about disappear if you have the functionality package installed as a singleton and only have one ui per instance of forums, your approach seems a better one.

This also seems to be the approach taken in the bcms package although its functionality package isn't actually declared as a singleton and doesn't contain any reusable chunks for the ui packages to make use of.

The issues for using a singleton functionality package that I can think of are:

Converting the ui of existing packages to re-usable includes according to some conventions that amount to a design pattern.
Proposed solution: This would have to be a gradual process with packages being updated as people found time.
UI packages would have to know which apm callbacks to implement and what api calls to make in them.
Proposed solution: these would be documented and the core package would probably provide default callback implementations.
Functionality packages would not be able to provide default ui's in their www directory so the upgrade process would be trickier.
Proposed solution: Any application package converted to this approach would become the default ui package which depends on the <package-key>-core package. All of the principal functionality would be moved to the <package-key>-core module.

Incidentally I was also thinking the reusable includes should make use of template::set_file to allow them to be used as datasource only includes... is this function kosher?

6: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Jun Yamog on 12/04/03 12:54 PM

Hi Lee,

As of yesterday bcms is now singleton and service package. I removed reusuable UI from bcms as of now. But they do exists on bcms-ui-base/resources which should move to /lib according to the new docs.

I do agree a good design pattern and may helper procs for this tip. On thing is needed is to enforce includes like what ad_page_contract does.

7: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Robert Denison on 12/04/03 02:40 PM

Jun,

I think that a discussion about design patterns for helper procs, whilst being totally necessary, would quickly go beyond the scope of this discussion.

A new TIP to discuss the design pattern for writing good reusable includes seems like a good idea. It would also help keep this discussion focused on separating ui from functionality. I will create this new TIP.

I like the idea of bcms-ui-base/resources (or soon to be lib), but apart from clean conceptual separation, is there a concrete advantage over simply putting them in <package-key>/lib, as it is not much of an overhead and keeps the number of seperate packages down?

8: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Lee Denison on 12/04/03 04:53 PM

To be more concrete here is what I propose to do to forums:

Modify template::util::url_to_file (used in include tag lookups) so that if you specify an absolute path it no longer assumes it is absolute from the server root, instead if there is a connection it first checks from the current packages www directory, then it's lib directory, then from the root of packages and only then from the server root.
Write ad_include_contract which has similar functionality to ad_page_contract but validates variables passed to an include instead of those passed in form variables.
Go through every page in forums www and factor includable chunks that ui packages would want to use, put them in forums/lib and update the current forum ui pages in www to use them.
Forums doesn't have any apm callbacks but if it did I would make sure they were factored into functions that a ui package could use.
Make the package a singleton.
Rename the package to forums-core.
Make a new non-singleton package called forums which depends on forums-core.
Move the www directory from forums-core to forums.
Write upgrade scripts for forums which install forums-core.
Write up the conventions used in creating the includes in forums-core/lib.

Comments?

9: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Jun Yamog on 12/04/03 05:23 PM

don't mind my post above :) Anyway go on with the tip.

I do like the idea of the tip.

10: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Don Baccus on 12/04/03 11:35 PM

I need to think about this proposal a bit before responding, but one question I have ... what is the benefit of moving datamodel and tcl API code from their traditional directories in the forums-core package?

If everything other than the /www/* default UI lives under /lib then ... why have /lib? :) Why not just /tcl and /sql as we do now reserving /lib for reusable includable templates as we also do now?

I'm not seeing the need for this part of the TIP, which would require APM modifications to support and which would leave some packages using the old style and others the new style directory structure, confusing newbies and probably old farts like me alike.

Splitting of functionality and UI is useful and as mentioned above has been partially done in .LRN though not with the notion of providing multiple UIs to a single functional package base.

You might study some of the .LRN portlet/package pairs if you've not done so already to see how this is done. It doesn't solve your problem but might be source of more ideas.

11: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Lee Denison on 12/05/03 08:52 AM

Don,

My intention is that only reusable include chunks would live in the forum-core/lib (which, as you point out, is already the case in some packages)... I didn't mean to give the impression that datamodel or tcl api would move there as well!

.LRN does use some aspects of this approach and does benefit from pages being factored into reusable includes. One difference though is that .LRN portlets require the original package interface to be mounted somewhere.

In many ways this TIP is about officially endorsing an approach that is already becoming the norm before I undertake a significant refactoring of forums.

12: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Don Baccus on 12/05/03 09:59 PM

Well, it was this statement that gave me that impression:

"The first package containing the datamodel, tcl api and reusable includes in package-key/lib."

I guess if you'd said "with resuable includes placed in package-key/lib" or something like that I would've understood your meaning ...

So far my major concern is the note that using ad_conn would be impossible. In the include chunks, yes, and this is true of portlet code as well. But I would imagine the UI packages themselves would be normally mounted and would have package_id available. There are issues involving URL generation from the functionality package ... if it's not mounted how can it call pages within itself? It might be possible to drive the UI package with an index.vuh file that can resolve links into the functionality package by doing includes or something like that, haven't really thought it out.

You may be interested to hear that my original thinking for the portals package, and something I urged Open Force to consider, was to make portlets renderable in a stand-alone fashion rather than only via calls from the portals package.

You can actually do that with some work by dummying up the right array values to pass into the portlet but it's not necessarily straightforward.

If we are going to adopt a TIP along these lines I'd like us to look into being able to share as many UI chunks as possible between portlets and standalone pages. In .LRN some refactoring was done in packages to do some sharing but it's all ad hoc.

13: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Caroline Meeks on 12/10/03 03:19 AM

Approve.

This is clearly a good idea and a direction we want to go. I also agree with Don that portlets should move in the same direction.

Once the coding is done we will definitely want to evaluate the results before we take it into to the official release.

To my thinking that now, at the concept stage and then secondly when it is done, are the two points that OCT should be officially involved.

We should discuss, brainstorm and debate implementation choices and ideas as a community, but we don't need to get an official OCT vote on each choice. The first time for anything is always an iterative experiment. Hopefully this will be based on BCMS and thus a second time, but that is no where close to Pind's Rule of Five.

I suggest we approve the basic idea continue this discussion outside the TIP process.

14: Re: TIP #34 (Proposed): Application package ui separation (response to 13)

Posted by Lee Denison on 12/10/03 11:41 AM

Here is my attempt to clarify my intention:

Firstly, my original suggestion was that for each instantiation of an application there would be two instantiated packages; a functionality package and a ui package. This approach had the ad_conn problems. However Dave pointed out that there are no ad_conn problems if the funtionality package is a singleton - which is also the approach taken in bcms. So everything after Dave's post is with the singleton functionality package in mind, ad_conn is no longer an issue and there is no need for a mapping between functionality packages and ui packages.

The work I'm proposing can be thought of in two main stages:

Factor existing ui pages into useful, reusable include files that live in the lib directory and maybe move the odd bit of code into a function. This isn't terribly contraversial and there is no implied change in thinking about how the package works.
Split the package in two; a singleton functionality package (sql, lib and tcl dirs) and multiple instances of ui packages (www dir). This requires work to deal with issues like url generation, apm callbacks and package parameters as well as recognition that code in the functionality package must satisfy all ui packages.

In my post above I also considered a couple of 'nice to have's before creating a whole bunch of includes. Obviously I would like these modifications to be part of OACS rather than specific to my project but they are neither required nor central to this tip:

A modification to the way templates files are resolved to make it easier to use includes placed in the lib directory - for my project I have modified ad_return_template and the <include> and <master> tags to call a new function which resolves relative files the same way as the previous call to template::url_to_file. For absolute paths it checks the lib directory of the current package before checking from the server root. It also accepts paths of the form '<package-key>://path/to/template' which check the specified packages lib directory before resorting to the server root.
Secondly creation of an ad_include_contract. For my project I've broken ad_page_contract into functions which are now also used by ad_include_contract. ad_include_contract examines variables passed to the include rather than form variables. It basically works but current doesn't handle arrays passed to the include by reference. Also the filters currently registered for ad_page_contract need to be divided into those appropriate for page contracts, those appropriate for include contracts and those common to both.

15: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Walter McGinnis on 12/19/03 09:59 PM

I agree that this TIP is headed in the right direction, but I have some points I would like clarified.

Where would documentation for the singleton functionality package go?
I believe a package-key/resources/package-key directory is still useful to the functionality package for things like supporting gifs, css, and js files.

A diagram (ascii is probably fine) of the proposed relationship and directories would be nice. In some ways, I think this TIP should be an addendum to https://openacs.org/forums/message-view?message_id=120221.

I'm encouraged by how large a group of people we have working on the various problems related to reusibility. Thanks for your work Lee.

16: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Walter McGinnis on 12/19/03 10:28 PM

Thinking about my last post...

If we do need a package-key-core/resources, we probably want a way for package-key-ui/resources to trump it.

Maybe I'm not clear on the details of the current state of the proposal.

17: Re: TIP #34 (Proposed): Application package ui separation (response to 1)

Posted by Peter Marklund on 03/26/04 10:08 AM

Allowing template customizations in a controlled way that doesn't require a fork would be a great improvement. Maybe splitting up packages like you describe is a good approach to achieving this, I don't know. I think I would need a working example or a more detailed description to have an opinion on whether this is the direction for OpenACS to move in. If you could draft the change in one package and show the advantages that would be very helpful.

A different, and more radical, approach to solving the presentation customizability problem is the work that Lars and Branimir are doing, labeled .KUL, aka data soup. Their approach is that you specify in an admin interface which object types you want, which attributes and templates they should use, and which pages your data should show up on. The idea is that you should be able to combine object types with views in a multitude of ways, i.e. show forum posts in a blog view etc.

18: Re: TIP #34 (Proposed): Application package ui separation (response to 16)

Posted by Tilmann Singer on 03/27/04 06:41 PM

I am against deciding now that this approach should be adopted in general for all packages in the future, without having it seen work in practice first. It sounds to me as it would add at least another complexity layer to package development, which might be desired for some frequently used very important packages that need different UIs, but maybe other packages don't need it and developing on them would suffer from the complexity.

Lee, didn't you say somewhere else that the separation of forums is finished - is that code available?

19: Re: TIP #34 (Proposed): Application package ui separation (response to 18)

Posted by Tom Jackson on 03/27/04 07:34 PM

The only pattern I see here is that the TIP forum is being used as a general discussion forum. Was this ever discussed on either of the other two more appropriate forums? Not everyone subscribes to the TIP forum, and this discussion is very far from becoming a concrete recommendation. One TIP would be changing template::util::url_to_file. This one idea may cause changes, or at least review of every public and private package. Anyway my main point is not for or against the idea, but with the use of the TIP forum before any discussion on one of the other forums. When the TIP forum is used it divides the participants into two classes: those that can approve and those who cannot.

20: Re: TIP #34 (Proposed): Application package ui separation (response to 19)

Posted by Joel Aufrecht on 03/28/04 01:38 PM

As per TIP rules, I am (belatedly) expiring this TIP.

21: Re: TIP #34 (Expired): Application package ui separation (response to 1)

Posted by Robert Denison on 03/29/04 12:42 PM

Tilmann, I committed some changes to Forums which separated some of the more reusable ui components into includes in the lib directory. This would make it easier to create an alternate ui for forums as a separate package - which I went on to do for my current project. But I haven't committed anything else which explicitly supports or uses this approach.

22: Re: TIP #34 (Expired): Application package ui separation (response to 21)

Posted by Randy O'Meara on 03/30/04 04:17 AM

Robert,

Are your changes in 5.1? Also, would you happen to have doc, notes, or examples of how to use the newly refactored forums code?

Thanks,

/Randy

23: Re: TIP #34 (Expired): Application package ui separation (response to 22)

Posted by Jeff Davis on 03/30/04 09:39 AM

The changes went in for 5.0, the code works the same, it's just been refactored to have most of the page fragments generated from includes in forums/lib/. So for example forum-view.adp includes /packages/forums/lib/message/threads-chunk rather than having the forum list directly generated in the .adp.

The advantage here is the portlet then can simply reuse this code rather than having to have a chunk largely copied from the regular page .adp.

The ultimate goal here is I think to further refactor forums so there would be a forums-base which had the tcl libraries, lib includes, etc, and multiple forums-ui packages which implemented front ends with possibly quite different security policy, navigation, etc.

It's not terrible to have a default interface in forums but I find having a service/application pair is easier to understand and cleaner in implementation.

Also, to be clear, the consensus on this TIP was that it was a good idea, and the refactoring of page components into includes was something we wanted to see, but changes to specific modules should be tipped individually.