Forum OpenACS Development: Some ideas about package structure ... comments?

1: Some ideas about package structure ... comments?

Posted by Don Baccus on 07/01/03 07:00 PM

Today packages generally have this directory structure, which is understood by the bootstrap code and the request processor:

/tcl (read by bootstrap code)
/www ("/package-key/foo" is mapped to "/package-key/www/foo" rp requires user have read priv)
/www/admin (request processor requires user have admin privs)

One issue is that currently packages put .gif, .jpg, .png and other image elements under various /www directories. An issue for performance is the fact that references are then permission checked by the request processor, which typically isn't necessary.

Sloan created a global /graphics directory which is then not checked by the request processor, but this doesn't allow APMs packages by third parties to take advantage (as their files are all supposed to be in their package directory tree)

My proposal to fix this is simple: define "/www/resources" to be a directory that isn't permission checked by the request processor. We already have the "/www/admin" precedent for treating a subdirectory of "/www" specially. Package could then stash their little left and right arrow .gifs and the like there knowing that they'd be available whether or not the user's logged in and without any permission checking overhead.

The other issue concerns templates that are meant to be included, rather than referenced by URL. For instance file-storage has a template named "folder-chunk" under its /www directory. Now ... if permission checking is done by the caller of an includable template rather than the template itself, a security hole is introduced (potentially at least.) Outsiders can call folder-chunk directly, in this example (though I think it does the proper permission checking, perhaps I should check!) One way around this: put the template someplace where the request processor won't see it.

My suggestion is simple: /templates under the package key. If an includable template is stuck there no one can reference it with an HTTP request, so relying on the caller doing security checking is no longer a security issue (if it is done properly, of course!)

So my suggestion is that we support the following structure:

/tcl
/www
/www/admin
/www/resources
/templates

What do others think? I'd make the simple request processor change needed for /www/resources shortly. Of course current packages would still work fine with files stored as they are, but I'd anticipate people would start taking advantage of the /www/resources directory to avoid permission checking quite soon, particularly in .LRN ...

2: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Bart Teeuwisse on 07/01/03 07:33 PM

Don,

I'm all for it. I'm already using the /templates directory in my packages.

How about moving the master template from the global /www directory to a global /templates directory. For sites that I have customized I usually end up with several other global templates.

We could do the same for cascading style sheets. Place those in /www/resources (either package specific or global).

/Bart

8: Re: Some ideas about package structure ... comments? (response to 2)

Posted by Sean Redmond on 07/02/03 04:41 AM

We could do the same for cascading style sheets. Place those in /www/resources (either package specific or global).

I think this is the first half of what needs to be done for really useful CSS on a package by package basis -- which is the essence of "cascading". The second half (since the <link> elements have to go in the <head>, which is probably handled by a global template) is providing a mechanism to pass stylesheet references as properties up to more general templates or have global templates look up package stylesheets (by package id?). The former is probably be more useful since it would allow more freedom to specify particular stylesheets when necessary.

4: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Tom Jackson on 07/01/03 08:15 PM

Those are some pretty common problems, good to address them at this level. What will be the structure of the /www/resources and the /templates directories, I'm assuming you need to duplicate directories under /www?

I wonder why the choice of .adp was used for template files in the first place. It seems too late to do anything about that, otherwise the rp could just refuse to serve files ending in something like .tpl, or whatever.

3: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Jun Yamog on 07/01/03 08:16 PM

Hi,

I think the proposal is good. I guess I am a offender of this security problem you have mentioned. Since I normally put my templates, gif, css, etc. on a single folder. Like www/templates/subsite1.

I think we should however use /includes rather than /templates since that is what we are particularly trying to secure. The includes that do not do any checking. /includes is more explicit to tell the developer to place his includes in /includes.

I guess given this structure I would not have to structure my templates as:

/www/resources/subsite1
/includes/subsite1.

Maybe /templates is better since will put also masters there. Anyway just suggesting that the name should be more obvious to the developer.

5: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Don Baccus on 07/02/03 12:07 AM

Right master templates ought to go there too, hadn't thought of that. If no one had brought it up I would agree with Jun's suggestion of "/includes" but now that's no good, sorry :)

I think it will be clear enough ...

".adp" was originally chosen by the AOLserver team, and the acronym is AOLserver Dynamic Pages I believe.

6: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Randy O'Meara on 07/02/03 01:02 AM

I like the idea. I guess I can undo what I've done by placing only an index.vuh file in the package /www directory and having the index.vuh map requests to /pages/* files.

Will this go into the 4.6 branch?

Randy

7: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Don Baccus on 07/02/03 01:47 AM

I'm just putting it into HEAD. I'd already sold Sloan on this idea so we could undo the global /graphics directory kludge ...

There will be a 4.6.4 but probably no .LRN 1.0.2 because it's full steam ahead on 5.0/2.0 ...

If folks feel strongly about backporting we could do it but taking advantage involves a fair amount of CVS munging that's not all that much fun.

Actually I've just put this into HEAD and munged the dotlrn package (not all of dotlrn, just the main dotlrn package!) to take advantage. One immediate benefit is that we can now use the real .LRN logo when displaying the login page to visitors who aren't logged in ...

10: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Lars Pind on 07/02/03 11:17 AM

Don, you forgot to mention the www/doc special directory.

In fact, how about making the /packages/package-keywww/resources directory available as /resources/package-key/, just like we do with doc, so that you can rely on a fixed URL for your graphic, regardless of whether and where any packages are mounted.

That would be really useful for master templates, portlets, and other renderings, that don't necessarily know where the package is mounted.

/Lars

11: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Timo Hentschel on 07/02/03 11:21 AM

Excellent suggestions! But I would also propose moving the default-master from /www to packages/acs-subsite/templates/ since it really belongs to acs-subsite. Another idea (since it just came up in a client project) is to move the parameter DefaultFormStyle to acs-subsite to be able to not only change the master-template on a subsite-level, but also the form-templates, which would make more sense from my point of view.

9: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Matthew Geddert on 07/02/03 11:29 AM

Are the security checks not going to take place in /resources even if the connection is secure via https? (my vote is that it never check permissions even if it is a secure connection). Currently RP treats permissions on secure connections differently then insecure ones. i.e. the public could not see the graphics in my /graphics directory on the https://site.com/register page without a hack to the ad_login_page proc (found in packages/acs-tcl/tcl/security-procs.tcl file), though they could see it when on straight http:// ... is there a reason why the request processor treats permissions differently if you are connected via https?

12: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Lars Pind on 07/02/03 12:01 PM

Timo,

I committed the subsite DefaultFormStyle parameter to oacs-4-6 branch on June 6.

Feel free to merge it over to HEAD, something which I haven't done yet.

/Lars

13: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Don Baccus on 07/03/03 02:55 AM

Timo - I agree about putting the default template in acs-subsite.

Lars - after looking into things after our discussion earlier today why can't you just use "resources/foo.gif" and ignore the url to the mounted package? If for some reason you *really* need a URL to the package that is guaranteed to work, why not use [ad_conn package_url]? I don't know why the dotlrn package was written to reference /dotlrn in hardwired fashion, but changing to "resources/foo.gif" works just fine...and packages shouldn't be cross-referencing small graphics buried in other packages (other than our current kludge of using acs-subsite/www/shared, something we can fix as we get time)

I'm also adding a suggestion made by Jeff Davis: disallow execution of scripts in the unchecked resources directory. Actually I'm only allowing one to reference .png, .gif, .jpg and .css files - I suppose we could parameterize this but is it necessary? Packages need to be able to depend on a set of extensions being guaranteed to work and parameterizing the list would allow the site admin to break everything.

We also discussed using ns_returnfile to quickly return such content without going through a bunch of request processor codef. I've reviewed the code that's executed when the request processor is in performance mode, and the path through the code to the ns_returnfile used to return non-script files is quite efficient and I don't think any special treatment is necessary.

14: Re: Some ideas about package structure ... comments? (response to 13)

Posted by Jeff Davis on 07/03/03 08:49 AM

The reason lars would like a /resources/package-key/ is to make the things like folder-chunk lars-blogger/www/entry-chunk etc easier to include in other packages (and particularly in portlets) where the [ad_conn package_url] would be /dotlrn/ or / for example. You probably still need to pull out the package url for links so it's not a huge win.

One big win with this is that since the image path does not change accross mounted versions of the package, browsers will cache things better (think for example of dotlrn where fs forums etc are mounted under each class/club instance. As it stands, the browser can't cache the associated images since the urls are all different, but if a /resources/file-storage/x.gif url was used for all instances then the images would be cached much more often.

15: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Mark Aufflick on 07/03/03 09:34 AM

i echo matthew's question about authentication under https - Don?

If it is the case, that's great, because it's getting annoying having to code in the kludge every time i setup a secure site.

16: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Don Baccus on 07/03/03 07:36 PM

The template include system doesn't go through the request processor so whatever we do for resources won't impact includes at all. You can use [ad_package_root_dir package_key]/includes (or /lib or /templates or whatever we end up with) to get to your includes.

I'm nearly done with the /resources/package_key implementation and will write a simple package called acs-resources to mount at /resources which will make them appear in the site map, too. This will make them seem a lot less "magic" ... we'll be shortcutting processing of a particular node rather than inventing it out of thin air.

17: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Joel Aufrecht on 07/04/03 04:41 AM

Two points:

1) Since the special feature about /resources is that it's not permission-checked, could the name reflect that? What about /public-resources or /public?

2) Can we make this the first TIP? (Or OIP?) One quick-and-dirty way to support the OIP process technologically might be a new forum. Anybody can post, first post in a thread must be a full, formal OIP proposal; then the thread has a week or two (or whatever is in the OIP plan - we should link that in the forum title) to accumulate yes and no votes from the Core Team. I'm sure this process has many drawbacks compared to a new custom package or a special version of Bugs, but are any of those drawbacks fatal? If not, I hope we do this or something similar to get momentum in a formal OIP process.

18: Re: Some ideas about package structure ... comments? (response to 1)

Posted by xx xx on 07/04/03 05:12 PM

https://openacs.org/bugtracker/openacs/bug?bug%5fnumber=194

While at it, can we store xql files in /templates too or add a parameter that will exclude xql-files from http:// requests in production servers?

19: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Don Baccus on 07/04/03 08:48 PM

Good point about .xql files - it is easy to get the request processor to not serve them and you're right, we shouldn't. I don't want to move them all, though, it's very convenient having them in the same directory when developing/bug fixing. And there are a lot of them, moving them would be a PITA.

As far as the TIP process goes I guess informally we've kinda done this as Jeff, Lars and I talked this over in private and we're also talking about it in public.

We really do need to get moving on the governance stuff, everyone seemed to run out of steam ...

20: Re: Some ideas about package structure ... comments? (response to 19)

Posted by xx xx on 07/05/03 01:29 PM

Maybe some people like to serve the xql files on development servers to see which queries are used on a page. So would it be possible to make the request processor listen to config.tcl (just like 'set directoryfile') ?

22: Re: Some ideas about package structure ... comments? (response to 20)

Posted by Jarkko Laine on 07/05/03 04:31 PM

Aldert,

Why would someone want to see the queries in browser? If he knows the name of the xql file (he has to, otherwise he can't load it with the browser), he can as well open it in emacs.

25: Re: Some ideas about package structure ... comments? (response to 22)

Posted by xx xx on 07/06/03 12:26 PM

Never mind, I hardly use it myself, but at times it is faster to look at the queries in the browser than to use ssh. ...just add (-oracle/postgresql).xql to the page (file) your looking at to find out if there is a related xql in /www...

21: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Don Baccus on 07/05/03 04:04 PM

Realistically, is there any time a developer won't have ssh access to their server? I mean ... not only do you *look* at your query files, you edit them...

23: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Jarkko Laine on 07/05/03 04:36 PM

...and if he doesn't, he should probably use developer support.

Which reminds me: Why is the db query analysis disabled by default in dev-support? Isn't it the single most useful feature in the package? Quoting Peter: "Why would anyone use the package if not for that?".

24: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Don Baccus on 07/06/03 12:32 AM

Yes, I agree, it is rather useless without it ... I don't know why the original aDer(s) implementing this package turned it off by default.

But I know how to get it enabled by default: submit a patch!

I think I'll adopt Joel's suggestion of "public-resources" rather than "resources" to make clear the implications of puttng stuff there ...

26: Re: Some ideas about package structure ... comments? (response to 1)

Posted by xx xx on 07/06/03 06:37 PM

Thinking about it, it is probably not a good idea to allow this illegal use of the browser.

It reminds me, however, of another posting in bugtracker:
https://openacs.org/bugtracker/openacs/bug?bug%5fnumber=118

Don, do you see any hooks in the request processor for the .dat extension?
Should it be there or should the documentation be removed?
https://openacs.org/doc/acs-templating/guide/document.html

27: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Joel Aufrecht on 07/07/03 12:27 AM

I really think we should do this as a TIP. It's simple and not very controversial, so we can focus our argument energy on process instead of content. :) We're all waiting for one of us to get around to implemented a more complete system. Meanwhile, we could go ahead and do it in a dedicated forum. It wouldn't be perfect, but it would let us practice the organizational issues. It would also uncover hidden requirements for a more complete technological solution. If someone on the core team agrees, would you please set up the new forum? The last available draft of the OpenACS TIP thing should be linked from the description or charter, and a summary of this thread could be the first TIP.

28: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Dave Bauer on 07/07/03 02:05 AM

Joel,

Why not use bugtracker? It seems the correct place to request features, make comments, and report completion. The TIP can be assigned to the person who is working on it.

29: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Talli Somekh on 07/07/03 05:25 AM

Dave. I think that's a good approach but not necessarily Joel's point, which I think is that we should take this oppotunity to implement or flesh out the governance proposals.

I think everyone has trust in Lars, Don and Jeff, but the fact much of the discussion took place in private is because we hadn't worked out a lightweight governance bureacracy for deciding these issues.

Don called for reigniting the governance discussion, so I'll do so in another thread.

talli

30: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Tom Jackson on 08/28/03 06:07 PM

So what was the final decision on the package structure? I like the /templates idea, but what is the logic going to be on searching, if any?

The original proposal was:

/tcl
/www
/www/admin
/www/resources
/templates

31: Re: Some ideas about package structure ... comments? (response to 30)

Posted by Joel Aufrecht on 08/29/03 09:09 AM

Revised proposal: /tcl /www /www/admin /www/public-resources /template /test

resources renamed to public-resources
automated tests move out of /tcl to reflect the fact that they may be broader and more general.

32: Re: Some ideas about package structure ... comments? (response to 31)

Posted by Randy O'Meara on 08/30/03 02:53 AM

Joel: it sounds as if you're planning to incorporate some interface in addition to acs-automated-testing? Would that be tclwebtest?

33: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Don Baccus on 08/30/03 03:37 AM

We want to incorporate tclwebtest into the automated test framework so we can do page testing as well as API testing.

34: Re: Some ideas about package structure ... comments? (response to 33)

Posted by Randy O'Meara on 11/20/03 07:48 AM

Regarding resources and templates...

I reread this thread several times and I am still not certain of the final outcome. So what is the final configuration in 5.0?

And, what paths are used when searching for resources and templates?

Thanks.

Randy

35: Re: Some ideas about package structure ... comments? (response to 33)

Posted by Randy O'Meara on 11/20/03 07:49 AM

One more quick question. Don, did you create the acs-resources package mentioned above?

Thanks.

Randy

36: Re: Some ideas about package structure ... comments? (response to 35)

Posted by Lars Pind on 11/20/03 10:33 AM

This became TIP #12:

https://openacs.org/forums/message-view?message_id=120221

37: Re: Some ideas about package structure ... comments? (response to 1)

Posted by Randy O'Meara on 11/20/03 09:13 PM

Thank you, Lars.

38: Re: Some ideas about package structure ... comments? (response to 37)

Posted by Malte Sussdorff on 11/21/03 08:00 AM

Can this be put somewhere in the docs, most likely the developer tutorial under "What is the typical structure of a package" ?