Forum OpenACS Development: Edit This Page 2.0 project

Collapse
Posted by Luke Pond on
Jun, Dave, and I have been working on plans for a rewrite of the Edit This Page package. Here they are:

https://openacs.org/new-file- storage/download/ETP2.html?version_id=523

If you're interested in content management or the acs-content- repository data model, please read this and post your comments on this thread. Particularly if you've checked out ETP and decided it wasn't for you - I want to know if there are any bugs, limitations, or evil hacks that I haven't explicitly addressed in the plans for the new package. By asking everyone for a design review at the level of detail of how ETP integrates with the rest of OpenACS, I hope to avoid making some of the mistakes I made with the first version of ETP 😊

In a few weeks we're going to need beta testers (both oracle and postgres). Testers will get the opportunity to contribute demonstration templates and "applications" to the final version of the package. Please let me know if you're interested in helping out with this.

Collapse
Posted by Jowell Sabino on
Since acs-content-repository allows you to store content in the database (as lob or text) or in the filesystem, how about a parameter that lets you specify how content is stored? For performance reasons, some sites may want to store ETP content on the filesystem.

3.4 Likewise, when a subtopic is deleted, ETP must delete the package and site node for you.

Will this also allow deletion of the package instance from the site map (i.e., not from the ETP admin interface)? ( See this). When a package is deleted from the site map, it calls apm_package__delete. There should probably be "on delete cascade" constraint imposed on the package_id attribute in cr_folders for this to happen.

Speaking of this new attribute, package_id, added to cr_folders, are we now assuming that whenever a package is built on top of acs-content-repository, a new package instance is created for every folder in cr_folders? Sorry, I really don't understand why package_id is now in cr_folders. Why can't a package (say ETP) create, if needed, a mapping table that maps cr_folders and apm_packages, with two attributes: package_id and folder_id? It accomplishes the same thing, but it still maintains the cleaner separation between acs-content-repository and any package that builds on top of it. File-storage does not assume that every folder is a package instance, for example.

5.2 An alternative to the triggers defined in the content repository’s content-search.tcl is necessary. ETP will provide triggers on the cr_items table that only submit changes to the search indexer when an item is published.

Aren't you changing the datamodel of acs-content-repository this way (no more package abstraction)? It seems to me that this trigger will benefit more than ETP, so why not make this trigger part of acs-content-repository? ETP (or any package for that matter) should not be creating triggers on CR tables.

Collapse
Posted by Dave Bauer on
Regarding search.

I believe for every custom content_type including etp_page_revision you will need a service-contract for search.

We might want to make a way to automatically create one for etp or require the application developer to provide one when they define a new type.

Collapse
Posted by Dave Bauer on
Is there any provision for user-contributed content? Will I be able to allow user-contributed items just by supplying a customized UI for a new application?
Collapse
Posted by Alex Sokoloff on
Here's one or two suggested enhancements I came up with when looking into using ETP to publish a set of Word documents. The most straightforward approach seemed to be, first save the documents in html format out of Word and then load the resulting files in ETP. To make this more convenient, what was needed was the ability to add page content by file upload rather than in a form (this maybe already in the works, I couldn't tell from the spec). The other thing needed was the option of removing the page header stuff and html tags, that is, abstracting out what appears between the body tags of the page.
Collapse
Posted by Luke Pond on
Thanks Jowell, Dave, and Alex for the good questions.

I'll try to answer Jowell's questions first.

Storage type:  The current design specifies two content types which are intended to be used in different ways, and imply a particular storage type.  etp_page_revision is for entering HTML content which will be combined with a template for presentation.  they'll be stored in the database.  etp_file_revision is for uploading a file which will be presented with its own mime type, like how file storage displays a file.  they'll be stored in the filesystem.  So you end up with text content in the database, binary content in the filesystem.  That seems like the way to get the best performance, so I wasn't planning on making the relevant storage type parameters configurable.

Folder/package mapping:  In my opinion it would be really nice if this problem were solved in the API for content_folder.  Both ETP and file-storage need the exact same functionality; content_folder__delete should delete its children for you, and some folders should be deleted automatically in response to a package being deleted in the site map.  It's cool that the problem is solved for file-storage, but I think anyone building on the CR would appreciate it if the solution was available there.

cr_items triggers for search: that's what I was suggesting; that we revise content-search.sql so that it's possible to index only published content items.  Neophytos and I talked about this a while ago and agreed this would be more useful than the current setup.

Dave:
In order to use the ETP interface you have to have admin permission on the package.  No reason why you couldn't grant admin to Registered Users for a wiki-type site, or to a particular user if you wanted to implement an "everybody gets a homepage" feature.

Alex:
This is not a feature we had planned (uploading the HTML content for an etp_page_revision object), but I can definitely see how it would be useful.  We'll definitely try to figure out a way to provide this option from the default UI, and to distinguish it from the more general file upload feature that is planned.

Collapse
Posted by Jun Yamog on
Hi guys,

Jowell:  Right now modetp already does a "file" storage type of storing the binary files.  Which means it uses the file system and just store the filename on the database.  Previously I used a lob but Luke advised me to use the file system.  It has certain advantages like using copy of modetp will only need copy the filename rather than making a new copy of the binary data.  So believe we will just use the "file" storage type on ETP 2.0 as supporting lob does not seem to give any advantage.

Having ETP mount a new instance for each folder has its benefits like you can have different ETP application type on a single ETP tree.  For example /level1 uses the simple ETP app while /level1/level2 and level2 uses the FAQ ETP app.  It also makes it simple to access ETP data via just simple URLS rather than /get-content?content_id=1234. You can also probably accomplish this via an index.vuh but it does make it more complex.  It has its disadvantages such as traversing the ETP tree to get some data on a different instance.  I have created a simple proc to the get the top most ETP subsection.  This way I can create the main nav links above.  See demo.infiniteinfo.com/modetp.

Alex:  Right now modetp is able to accept file uploads even HTML files.  This will be served up as HTML although it will be be merged with the template.  modetp makes use of a visual editor which basically IE 5's dhtml edit mode on.  It looks cool but I believe its not a long term solution.  If you have a need to get Word like interface then the visual editor that is used in modetp should work.  I think agree with Luke it maybe hard to support uploaded HTML and merged with a template.  But ETP 2.0 should serve HTML pages without merging with templates possible.

All:

I think we should drop the support of ETP coexisting with subsite packages.  This makes more complex stuff.  We should just mount ETP on /foo then just use Host-Node map.  Currently modetp seems to just run fine with this setup.

Anyway got to go I am kinda hungry its way past lunch time.

Collapse
Posted by Jowell Sabino on
Hi Luke,

This should be the model for openacs development -- design decisions are brought in the open for the community to discuss. Thanks for making the process transparent.

I have not delved deeply into the ETP datamodel, because I got hung up on my inability to delete an ETP package instance. I gave up easily, but that is just me. 😉

Anyhow, with regards parametization of storage type, one advantage of filesystem over database storage is performance. But the cost is that it is a bit complicated to backup content because a pg_dump isn't enough. You still need to create a tarball of the actual content on the filesystem. It would be nice if ETP 2.0 would be designed with giving users the choice: performance vs. ease of backup. You could design ETP so that there is only one per-package parameter of storage type. Parameter lookup is done only when new content is added, so you only need to provide a wrapper for content_item__create that includes this added parameter, plus extra logic to determine which storage_type parameter to pass on to content_item__create. Including the parametization of storage type is (hopefully) trivial to add. But then again, it is you who is doing the coding...

My comments regarding folder/package mapping and adding triggers are really about making sure that we maintain the separation of packages that is one of the defining features of Openacs4. I don't want to go back to the monolithic nature of 3.x again.

Oh, did I say that I am interested and willing to alpha/beta test ETP 2.0?

Collapse
Posted by Don Baccus on
It's cool that the problem is solved for file-storage, but I think anyone building on the CR would appreciate it if the solution was available there.
Yes, this is good.
cr_items triggers for search: that's what I was suggesting; that we revise content-search.sql so that it's possible to index only published content items. Neophytos and I talked about this a while ago and agreed this would be more useful than the current setup.
This would solve the problem with bboard forums which, in the Oracle version at least, makes both your post and the dupe massaged for e-mail alerts available to Intermedia.

Of course the new search package does things differently than is done with Intermedia currently (and we have to move Oracle from SWS to the new search for the enxt release) but the general idea, only index published items, makes sense.

Jun - I'm not sure what you mean by dropping support of ETP coexisting with subsite packages. Subsites bring functionality to the table, i.e. membership, master templates, etc.

Regarding lob/filesystem storage I personally would like to see *all* packages in the system give this choice. Storing everything in the DB is inappropriate for busy sites with lots of large binary files that are accessed, but as Jowell points out consistent backups come for free with the RDBMS dump/export commands. Small, non-busy sites are exactly those likely to be run be people who will appreciate a very simple way to make consistent backups, while those running busy, large sites are more likely to be skilled at sysadmin tasks or be in a position to hire someone with those skills.

It is very easy to support both forms if you use the Tcl API I added to the CR Tcl library (check file storage for example).

Collapse
Posted by John Mileham on
Hi Luke,

I like the idea of distributing ETP apps as packages.  The concept of symlinking into the ETP package on install makes me cringe a little bit, though.  A few reasons:

1. A secured ACS installation might not allow nsadmin write access to the tree in a production environment, so the install would be hairy.
2. Symlinks are a very platform-specific thing, though it is looking like OpenACS and UNIX are pretty well joined at the hip anyway
3. Symlinks being created by packages would probably be something of an administration hassle for anybody keeping their ACS tree under version control.

How else can we hook up these pages?  First a little bit of ACS historical perspective.  (note: this is about to go way OT, and is obviously not something that can make it into this release cycle)

ACS/OpenACS <=3.x always had a convention of using web pages as API.    This made things like /general-comments possible.  When developers at aD sought to actually build apps for 4.x, they were shocked and disgusted that there were no guarantees that ANY canonical location existed for a given page.  In fact, a debate raged about whether a canonical location should have been established for every package to clear up this problem.  But that would have been wrong as well (at least as "correct" usage of the toolkit has evolved), as usually, multiple instances of the same application are peers; seperate but equal, and no instance could presume responsibility over the others.

What we need, then, is a system to globally mount utility API.  Maybe this would take the form of a package subdir named "www-util" that gets mounted under "http://server.name/util/package_key/" upon installation of the APM.  Maybe it means adding a flag to ad_proc to actually provide public web procedures referenced like: "http://server.name/web-proc?proc_name=foo&arg1=bar&arg2=baz".  The latter approach could even eventually lead to the integration of ad_page_contract and ad_proc, which as they exist currently are eerily similar, but different.

I know that aDers have had to use some very convoluted techniques to traverse the site map in order to find the location of another package mount point that is an acceptable provider of whatever URL-API they need.  I haven't looked at the code, but I know that Phong had to do some ranglin' to port the general-comments module, as he pretty much had to invent the technique he used on his own.

Some would propose that singleton packages solved this problem.  This would really only be true if 1. singleton packages were always mounted at a set URL (not possible for non-core packages because of potential namespace conflicts with a given user's ACS install) 2. You shouldn't actually have to mount a package in order to use its API, because that's what this is... even if it comes up in somebody's browser window instead of getting called behind the scenes.

Enough of that... too much thinking about reengineering the core will make your brain melt (cough..ACS5).  Back to the topic at hand...

I'm interested to know what kind of perms would be used by the ETP2 system.  I have the feeling that a complex workflow with many actors will require a more elaborate permissions model than a single admin privilege on a given ETP instance.  At the very least, creating, editing and publishing content seem to be different tasks that could be delegated to different actual parties.  Not to mention the possibility that some actors may only have control over an object while the object is in a certain state or set of states.  I'm not saying that this isn't in the works already, It just wasn't clear to me how it was going to work from the document.  For all I know, acs-workflow handles this already, or maybe the whole workflow goes on behind the permission model's back, which may be the Right Thing after all... haven't done enough thinking.

Otherwise, looks great to me.  All of my gripes with ETP1 seem to have been cleared up.  Now I'll just have to port all of the hacking we did to get around its limitations to ETP2 :)

-John

Collapse
Posted by Dave Bauer on
John,

I am thinking it might be possible to setup acs-service-contracts for packages that provide services to other packages. I am pretty sure this is exactly what it is for.

Collapse
Posted by Don Baccus on
Yes ... acs service contract is the way to set up interpackage communication.

Physical symlinks in the OS are something we really want to avoid, does ETP actually do this? I thought the symlinks implemented were just db mappings ...

Template includes are one way to grab a .tcl page that lives in a package that's not mounted, because the template engine works off paths not URLs. acs-integration makes relative paths work by passing the path to the current page as it's path root, but absolute paths look like:

ad_return_template "/packages/your-package-key/www/your-template-name"
This doesn't help for HTML API pages that link to another page (say a post-confirmation page) because the link, of course, contains a URL and therefore must be mounted.

The workflow package isn't integrated with permissions, it uses parties to determine who can execute which user tasks.

But I think it should be integrated with permissions to allow tasks executable by users with a given permission on the object.

I don't know what Lars thinks about that, though. Lars, you here? Read this thread! Tell us what you think!

Collapse
Posted by Jun Yamog on
I would like to remove the support for ETP being able to used as the root together with the Main Site subsite.  Host node mapping is already working so we can drop this feature from ETP.  There are some hacks in ETP to get into to work well together with Main Site subsite.

Now that I see the advantages of supporting both file and lob storage type.  I will implement both for ETP2.  modetp does have both although the lob storage type is not being used.

There is also the caching issue in ETP.  Now that we will support binary files.  Does this mean that will cache the binary data?  Will this make nsd memory footprint bloat?  Currently it uses util_memoize.

I also think that one thing that we may (not sure) support is an export or publish feature.

Collapse
Posted by Don Baccus on
Oh, I see what you mean about subsites then, yes, among other things the  "root hack" seemed to confuse people a bit, i.e. several folks had to ask for help in the forums to get it working.  If you can provide the same functionality using host node mapping and provide clear instructions for doing the mapping then yeah, sounds good.

Caching ... how about a parameter that allows one to choose the largest size file that will be cached and set it by default to a reasonable level?  "0" would imply "no binary files cached".  And maybe that's reasonable default behavior, actually???  You certainly don't want to try to cache something like an Oracle download (1/2 GB) on an average webserver!

Collapse
Posted by Jun Yamog on
Host node map works very well... I think.  This is not just for ETP its for practically for all packages.  So basically you can mount /etp1 for www.virtualsite1.com and
/etp2 for www.virtualsite2.com.  So if a visitor visits www.virtualsite1.com he get
the content as if we has in  www.realsite.com/etp1.  I have tested modetp to work
on this setup.    Not extensively but it works.  Not too sure if the permissions of
subsite will apply since you are in reality on an etp instance.

Check it out its really cool.  So in theory you can house many CMS sites in one OpenACS
instance.... =) Geocities anyone?

Not really sure about the caching issue since I have little knowledge about util_memoize.  Anybody care to explain about util_memoize?

Collapse
Posted by C. R. Oldham on
Two suggestions for the new project:

1. Some way to handle images.

2. Some way to move content from one ETP instance to another.

--cro

Collapse
Posted by Jun Yamog on

Hi Cro,

We will surely do those in ETP2. Those already work with modetp.

Those of who are interested in getting crappy bad code of modetp get it at modetp. Please dont laugh on the code have an open mind look at the concepts that it has proven. I promise to contribute good code in ETP2. Not some slap here and there to meet a marketing deadline.

I will do my best to get as much new stuff in ETP2 particularly the concepts that was done for modetp. I am kinda swapped now so it maybe a little while. Keep the feedback coming.

Collapse
Posted by Stephen . on
How about a parameter for acs-subsite which sets which front page to use instead of the default, similar to the parameter already used to set the master template? Then any package can be the front page of your site (or subsite). I don't think the user registration problem has been solved when mapping a host to a node other than a subsite.
Collapse
Posted by Don Baccus on
Hmmm...that's an interesting idea...
Collapse
Posted by Luke Pond on
Stephen's suggestion would work really well for my purposes, and it would obviate the need for all of the remaining nasty hacks that would otherwise have to be included in edit-this-page (such as adding its own parameters to the acs-subsite package!)

I also believe it would be generally useful: I'm sure people occasionally would just like to deploy the bboard UI as their homepage, for instance.

Are there any volunteers to implement this?

Collapse
Posted by Dave Bauer on
I was thinking about how I am planning on using the ETP package on my web site in regards to categorization.

What I would like to do it enter all of my content in one ETP instance and then choose which category, or categories the page will be listed under.

Currently I have a seperate ETP instance for each section of my site. This makes it more difficult to enter content, because first I have to decide which section, then navigate to where I want an article to appear and add a new page.

I will explore if a categorized application like I imagine can be built with the existing ETP 2.0 design.

It may be this is an ETP 3.0 feature, or an entirely different application.

Collapse
Posted by Clay Gordon on
I use Userland Manila in a hosted environment for my site chocophile.com. I'd love to see an OACS-based version of this, including the ability to enable a managed hosting company to quickly and easily set up virtually-hosted domains (chocophile.com is not on a dedicated server -- not @ $12.50/mo). If you need a tester who is familiar with using Manila, then I'd love to help.

One extension I'd like to see is the ability to be able to create "private" areas for consulting clients, e.g., client1.webwideguides.com, client2.webwideguides.com. This would provide a simple extranet capability.

Manila provides robust templating capability with the ability to define a different template for the home page. There is little workflow, but there are multiple editor roles (managing editor, contributing editor, content editor), which is useful.

One useful aspect of Manila is the plug-in architecture. I can see many OACS modules as plug-ins, and people creating ones specfically to support some types of interaction.

Finally, if ETP for OACS supports the portals module, I can see a lot of very interesting work in the development of "themes" -- packaged designs.

As I said, a great idea. I can't contribute much technically to the discussion of the overall integration w/OACS, but I'd like to help w/features/functions from the user perspective. As an aside: support for SOAP/RSS, take a look at radio.userland.com, and the klogs group at Yahoo! groups for more on how ETP/OACS might evolve.

Collapse
Posted by Jun Yamog on
Hi Guys,

It seems that there are lot of great ideas for ETP 2.  Like using categorization, etc.  I for one would like suggest the idea of linking content items to other content items.  Currently ETP 1 uses cr_folders for its parent_id, we can also use another cr_items as a parent_id.  This way we can have multiple objects related to one another.

But I think the general direction of ETP 2 should be more of a simplification and clean up.  Move more functions to /tcl procs and use more db api's.  I think is in the list of Luke too.  Once we have a good working skeleton CMS system we can then build this cool ETP apps on top of it.  My opinion is move ETP to a platform for CMS like application.  Its like a thin layer in between OpenACS and your CMS app.  So ETP 2 will carry some application (e.g. simple content, w/ workflow) that may serve as a starting point for customization or example how to use ETP core.

So I think we should move ETP 2 to a general direction of having ETP core procs and db api.  Then have very thin applications that make use of this ETP procs and db api.  So its like OpenACS that is a blue print for a Community Based site, ETP 2 will be a blue print for a CMS app.

Collapse
Posted by Sam Snow on
Just in case the ETP2 guys had not seen it, it looks like this site offers a similar service on a site wide basis-- http://www.elexio.com/products/intellisite/ . It is aimed at making a website easy to edit for anyone. As they say-- With IntelliSite, the term "Webmaster" is now redefined to mean, "Anyone who has word processing skills".

Unfortunately they require IE6 for a demo, so I have not yet been able to log in a try the demo site out. Their product might be a good opportunity to swipe some ideas that would be useful for ETP2.

Collapse
Posted by Jade Rubick on
How is ETP 2 coming along? I may have a little time to help
alpha test it, if you're still looking for volunteers.
Collapse
Posted by Dave Bauer on
Jade,

It's just me, the other contributors are too busy right now.

Any help would be appreciated. Right now I am going through the code to see what needs to be added to comply with the original plan. Some features might be left out or changed. I'll put up a list of the latest ideas I have later this week for comment.

Collapse
Posted by Cathy Sarisky on
Dave - thanks for your reply on the other thread.  I'd missed that somehow.  Default values gets me part of the way there.

I'd be delighted to alpha test when you get to that point.  I've got some time this summer while I'm not teaching.  I will need to write some templates/applications this summer which I can share, although it seems I'll be doing them with a hacked etp1.0.  I wish I could contribute more to 2.0, but I'm definitely a 4.5 newbie, and I don't quite "get" the CR yet.

Right now, what I need most is the ability to take user input as text or an uploaded file, process it, and then shove it into ETP as HTML.  (With the ability to turn it BACK into text before the user gets to edit it again... not all users like HTML.)  It would be nice to have an attribute defined for procs to modify the content when it is passed between the user and the content repository.  I will probably hack etp-edit and get that functionality, but it won't be as clean.

Collapse
Posted by Dave Bauer on
Adding another level of permissions to ETP 2.

I am thinking this might be useful enough to add as an ETP 2 feature.

Right now there is an admin template, item-list template, and item-display template.

I am thinking of adding an editor/moderator role in there. So that you can assign some authority to a user to edit and approve content, but not to administer the entire package.

To accomplish this I want to add a editor/ or publish/ directory under edit-this-page/www to hold all the templates for applications that support this level of permissions. It would be optional, but I think it is useful enough to add.

To make this work, I will add a priviliege etp-publish that will be given to users who can edit and approve content.

Does anyone have and ideas about this? Is this the right way to go about it? Or is there a better way?

Collapse
Posted by Jun Yamog on
Hi Dave,

I am ok with adding this custom permission.  I think we will
eventually use it, if not atleast ETP 2 will tell us if its good or
not after its been tested in the real world.  Although I am thinking
about my suggestion moving the files into www/admin may have its
limitations just like what you have discovered.  Since we are going
to check permissions anyway lets just all move it to www/etp.  We
should reduce the number folders under the www folders as to avoid
any conflicts with the cr_folders.

Maybe if people would not mind the lower directories.  We should
move stuff in www/etp/admin and www/etp/templates.

We can also modify index.vuh.  So rather than reserving the etp-* or
in the current direction of ETP2 www/admin to just show admin pages
when a query string is passed like "edit_mode_p=1".

Or another idea which might even solve our problem of having
multiple content items on a single page.  So clicking "Edit this
item" will dispatch edititem?item_id=blah.  Then we get the correct
admin/edit page for that particular content type.  Then we just
separate the adding/deleteing sections admin pages.  I will chat
with you in the IRC channel if the idea is bit fuzzy.

Collapse
Posted by Rafael Calvo on
Hi,

Is there anny progress on ETP 2.0?
How can we help? Does anyone have an estimate on how much effort it would require to finish it?

Collapse
Posted by Sandhya Subash on
Hi,

Creating a new ETP application like news, faq etc only lets you create a new content item. It would be good if we can create subtopic applications in ETP 2.0  and display the extended attributes rather than Title,description,content etc.

will is ETP 2.0 shipping?

Collapse
Posted by David Burch on
The biggest problem with version 1 of ETP, is the lack of support for images.  For example, I want to be able to create an article type that has three paragraphs of text and three image thumbnails with captions (click on the images and a larger image loads).

This seems to me something everyone with a web site wants to do and I cannot find anything in OpenACS, short of static pages, that does it.
Sure, I can build it myself using CMS or CR, but why must everyone reinvent the wheel?

Collapse
Posted by Ben Koot on
Hi folks, I have been playing around with the photodb module and face the same issue. Maybe a simple adjustment would help. The code used to determine the position of images is allready available in Wimpypoint. It seems to me if that was extended to all other modules dealing with images by default we would have solved the problem.
Collapse
Posted by Denis Barut on
Hi,

Is there any news on ETP 2.0?
How can we help? still need beta testers ?

Can we've access to the code ?