Forum OpenACS Development: update of the xotcl packages for oacs

Hi everybody,

here is an update of the various xotcl packages. The following sections describe the changes in the XOTcl pages since the last release.

These packages can be downloaded from:
http://media.wu-wien.ac.at/download/xowiki-doc/xotcl-apm.html

<h3>XOTcl-Core (version 0.27)</h3>

<h4> Object interface to content repository</h4>
This interface provides a simple way to instantiate
and manipulate items from the content repository.
In a very similar way an interface to oacs objects
could be established as well.

The interface is described by examples.

  • create an XOTcl object from an arbitrary CR item (type is taken from OACS) provided that the type has a class definition;
        CrItem instantiate -item_id 6413
    The resulting object has all core attributes of the CR as well as the defined "extra" CR-attributes

  • do something with the object
        6413 append title " - Kilroy was here!"
  • and save it with a new revision
        6413 save
  • ... or make a new item
        6413 save_new
  • ... or delete it in the database (destroy is for XOTcl object only, delete for db)
        6413 delete
  • one can perform as well re-classing, e.g. like
        CrItem instantiate -item_id 6417
        6417 class ::xowiki::Page
        6417 set title "en:[6417 set title]"
        6417 save_new
  • create an answer set containing all XoWiki pages (and its subtypes) contained in a specified CR folder; the answer set can be restricted or ordered, etc. via additional attributes.
        set r [::xowiki::Page instantiate_all -folder_id 3737]
    The result is an ordered composite, suitable for the templating described below. One can iterate over the
    children using the method "children".
        foreach o [$r children] { ......}

<h4>Support for nested Object Structures</h4>

The class OrderedComposite supports object structures similar to XOTcl's aggregations, but where the order of the included elements is preserved by default. The OrderedComposite supports ordering (based on Tcl sort) of the elements as well.

The following example shows how to define complex strucures based on the composite. In this example, we have three different rectangles with different colors nested in each other. The rectangle is a specialization of an OrderedComposite.

   Class Rectangle -superclass OrderedComposite -parameter {color}
   Class Point -parameter {{x 100} {y 300}}
   Rectangle r0 -color pink -contains {
      Rectangle r1 -color red -contains {
         Point x1 -x 1 -y 20
	 Point x2 -x 1 -y 2
      }
      Rectangle r2 -color green -contains {
	 Point x1 -y 123
         Point x2
      }
   }
The sort order of the elements of the OrderedComposite can be optionally determined. When a sort order is defined for an ordered composite the method "children" returns the children nodes in the specified order. Note that when ordered composites are nested, the ordering per composite can be different. This makes it e.g. possible to construct tree tree structures with sort orders. Without "orderby" the order of the insertion is preserved.
   r0 orderby color
   r0::r1 orderby -order decreasing y
In the example above, the content of rectangle r0 is ordered by color, while the content of r1 in r0 is sorted by the attribute y.

Elements can be added as well to the composite incrementally via the add method. Ordered composite is used by the object interface to the CR and by the templating

<h4>Object Oriented Templating</h4>

XoTcl Core contains a simple templating system that is defined in terms of widgets and decorators (in the design pattern sense) for these widgets determining the rendering.

Widgets define rendering-independent structures such as a Table with different kind of columns. A widget might use several ordered composites (e.g. one for the columns, and one for actions, one for the data)

A Decorator is used as a skin for the widget and might be mixed into widget (realized via mixins). Therefore, with rather little effort, alternative skins can be provided and specified at rendering time.

The current implementation supports a TableWidget with Fields and AnchorFields with two skins, one called TABLE (render TableWidget with a HTML table, exactly like ::template::list::create) and TABLE2 using divs and tables. Per default, TABLE2 is currently activated.

The following example is taken from the request monitor. A table widget with two columns is defined, where the columns have handles for sorting. The url query attribute "orderby" stored in $orderby has the syntax as for ::template::list and is passed to the orderby method of the Ordered Composite. The "add" method is used to add incrementally rows to the composite. Finally, asHTML renders the TableWidget with the default renderer.

Example:

   set title "Active Communities"
   set context [list "Active Communities"]

   TableWidget t1 \
      -columns {
        AnchorField community -label Community -orderby community
     	Field count -label Count -orderby count
      }

   foreach {att order} [split $orderby ,] break
   t1 orderby -order [expr {$order eq "asc" ? "increasing" : "decreasing"}] $att

   foreach {community_id users} [throttle users active_communities] {
      if {$community_id eq ""} continue
      t1 add \
      	 -community [dotlrn_community::get_community_name $community_id] \
      	 -community.href [export_vars -base users-in-community {community_id}] \
      	 -count [llength [lsort -unique [eval concat $users]]]
   }
   set t1 [t1 asHTML]
The concept based on mixins can be used as well for individual renderings similar
to the display-template option of ::template::list

Example:
  Class CustomField -volatile \
      -instproc render-data {row} {
        html::div -style {
	  border: 1px solid #a1a5a9; padding: 0px 5px 0px 5px; background: #e2e2e2} {
	    html::t  [$row set [my name]]
	  }
      }

  TableWidget t1 -volatile \
    -columns {
      Field time       -label "Time" -orderby time -mixin ::template::CustomField
      AnchorField user -label "Userid" -orderby user
      Field ms         -label "Ms" -orderby ms
      Field url        -label "URL" -orderby url
    }
Currently, only the request monitor uses these widgets. Only a subset of ::template::list is currently supported (optional sort controls on columns, column labels, HTML attributes of columns, anchor fields, csv output). By using the TABLE skin, the TableWidget is rendered exactly like template::list, the client code is slightly shorter than with template list. For rendering large tables (e.g. 2000 rows) i measured a performance win by a factor of 10 (23 seconds on my notebook with template::list vs. 2.5 seconds for the oo templating). The performance difference is esp. relevant for sites with high loads. template::list seems especially bad with large tables.

<h3> XOTcl-note (version 0.5)</h3>

  • the more or less minimal example, upgraded to the object interface
<h3> XoWiki (version 0.14)</h3>
  • upgraded to the object interface
  • less database interaction
  • various improvements on the forms
  • class names in the ::xowiki namespace
  • upgrade script for using the new class names

<h3>XOTcl Request Monitor (version 0.35)</h3>
  • upgraded to use object oriented templating

  • new function to register community activity, when community information cannot determined from the request url/site-map
             throttle community_access 2410
  • toggle for Java script graphics and poor-men's graphing approach based on divs
Collapse
Posted by Malte Sussdorff on
Hi Gustav,

would you mind putting all your packages into CVS HEAD, otherwise I'd do it :).

Grabbing your releases from the various places and unpacking them to work in my test installations is just too much hasle in comparison to just doing a "cvs update -Pd xo*"

Collapse
Posted by Nima Mazloumi on
This is really great. Why did you decide for an object interface for CR in the first place and not for acs objects right away?

And how does this work together with services like:
- categories
- notification
- general comments
- permissions
- i18n
...

Or would we need an XoTcl replacement for them?

template::list::create has support for
- filters
- pagination
- display_templates
- sorting
- formats (csv-export)

Looking at the source code I see support for:
- sorting
- csv-export

Do you have plans for an object oriented version of ad_form as well?

Maybe we could extend the documentation "Part III: Package Developer's Guide" with this new functionality´, or shall we wait until we have reach a certain stability?

My XoTcl knowledge is still on shaky grounds but I am really looking forward to use that.

Collapse
Posted by Gustaf Neumann on
Why did you decide for an object interface for CR in the first place and not for acs objects right away?
a few reasons: i wanted to do this with some real-world code that i understand (xowiki), therefore i have a better understanding of CR than of the oacs object model in general (acs_attributes do not seem to be used everywhere, one has to deal with the stored procedures with the __).

I see no problems with the services you are listing. why are you asking?

Concerning template::list::create: as i wrote, the oo templating is a subset (e.g. filters, pagination, bulk-actions are missing), some features are already better supported (sorting). there is a replacement for display templates (see above). Concerning formats: i have to get a better understanding how this differs form having different skins and what it used for, before i have an opinion about it. Most probably, there are other features missing.

concerning ad_form: it is not on my urgent to-do-list. nice would be a state based message redirector, maybe in connection with the SMC, but maybe this is an overkill.

This is just the first attempt for this functionalities, the templating will be certainly extended, the object layer is as well in a very early state and should fit into a picture with a layer on oacs objects. i have done some early work in this area (posted here some time ago), i think that neophytos has continued with this, but i have to get in touch with him about this...

Collapse
Posted by Malte Sussdorff on
xowiki 0.14 does not work on a fresh install at all. Sadly I cannot detect what exactly is wrong with it as I have no clue about xotcl. Maybe someone more knowledgable can take a look.

One error right from the beginning preventing the loading of xowiki-procs.tcl

[14/Dec/2005:17:38:05][5796.134557696][-main-] Error: Ns_PgExec: result status: 7 message: ERROR: column "page_id" duplicated
CONTEXT: PL/pgSQL function "content_type__refresh_view" line 55 at execute statement
PL/pgSQL function "content_type__create_type" line 54 at perform

[14/Dec/2005:17:38:05][5796.134557696][-main-] Error: Error sourcing /usr/local/aolserver/dotlrn-demo/packages/xowiki/tcl/xowiki-procs.tcl:
Database operation "0or1row" failed (exception NSDB, "Query was not a statement returning rows.")

ERROR: column "page_id" duplicated
CONTEXT: PL/pgSQL function "content_type__refresh_view" line 55 at execute statement
PL/pgSQL function "content_type__create_type" line 54 at perform

SQL:
select content_type__create_type(
'CrWikiPage','::xowiki::Page','Wiki Page', 'Wiki Pages',
'generic_cr_wiki_page', 'page_id', NULL
)

while executing
"ns_pg_bind 0or1row nsdb0 {
select content_type__create_type(
:object_type,:supertype,:pretty_name, :pretty_plural,
:table_name, :id_..."
("uplevel" body line 1)
invoked from within

Collapse
Posted by Ivan Histand on
Gustaf, I applaud all your hard work on these packages. I've updated all the packages (core, notes, wiki, request-monitor) to the latest you've posted. When testing the wiki package, I come across the following error when clicking on the "Add xowiki page instance" button. Please feel free to test it on my dev server located at: http://coastalportraits.org/xowiki/

=======================================
can't read "instance_attributes": no such variable
::1780 ::xotcl::Object->set
invoked from within
"$data set instance_attributes"
(procedure "edit_request" line 6)
::xowiki::f1 ::xowiki::PageInstanceEditForm->edit_request
invoked from within
"::xowiki::f1 edit_request $item_id"
("uplevel" body line 1)
invoked from within
"uplevel #4 {::xowiki::f1 edit_request $item_id}"
("uplevel" body line 1)
invoked from within
"uplevel 1 $args"
(procedure "ad_page_contract_eval" line 2)
invoked from within
"ad_page_contract_eval uplevel #$level $edit_request"
(procedure "ad_form" line 411)
invoked from within
"ad_form -extend -name [my name] -validate [my validate] -new_data $new_data -edit_data $edit_data -on_submit $on_submit -new_request $new_request -..."
(procedure "generate" line 43)
::xowiki::f1 ::Generic::Form->generate
invoked from within
"::xowiki::f1 generate"
("uplevel" body line 52)
invoked from within
"uplevel {
ad_page_contract {
This is the edit page for notes.

Collapse
Posted by Gustaf Neumann on
This week i was/am very busy, so no time for fun stuff and reading forums. excuse the slow response.

Malte, concerning CVS: a couple of reasons:

  • the cvs rules say "only GPL code". Practically all of the code i write uses the liberal BSD style license (like Tcl) not GPL. It is used in many products, where GPL would be a problem. I must say that some of the most valueable feedbacks were coming from such installations, so i see no reason for changing this. Is it safe for me to ignore the statement from the CVS guidelines?

  • the xotcl request monitor depends on libthread. i have submitted a TIP for it, but so far there was descision about this AFAIK. Without libthread the package will spit out a bunch of error messages.

Malte, how do you manage your installations? you check out in part from say oacs-5-2 and part from HEAD? don't you get confused about this? what about the patches for bootstrap and api-browser. isn't running a mixed version asking for troubles?

malte: concerning the fresh install. i could reproduce the problem with the fresh install. It comes from the code that tries to convert old xowiki data into the new types and structures. As i think that most installations don't have big collections of xowiki pages yet, i have comment the section out and put xowiki 0.15 on http://media.wu-wien.ac.at/download/xowiki-doc/index.html

Ivan, there seems to be a bug in the content repository in oacs 5.1 in content_type__create_attribute(). Under oacs 5.2 the view contains the extra attributes (such as "instance_attributes", under 5.1 it does not. check: \d xowiki_page_instancex from pgsql. Is it a problem for you to upgrade to 5.2?

Collapse
10: GPL vs. BSD licenses (response to 7)
Posted by Andrew Piskorski on
A BSD license doesn't effect OpenACS in the slightest, as BSD is fully GPL compatible. However, (my non-lawyer's understanding is that) as soon as someone starts contributing GPL code to a BSD codebase, that project effectively becomes GPL immediately. This is the viral nature of the GPL at work. (You could always rip out all suspected-GPL code, or just plain revert to the original BSD code, but as more contributions and improvements come in, that gets pretty unpleasant pretty fast...)

So, allowing code under non-GPL Open Source licenses into the OpenACS CVS repository is not a problem at all for OpenACS, but it might well be considered a problem by Gustaf or others who have written effectively independent packages or applications and published them under a BSD license. If a package really is independent (rather than a "derivative work" of the GPL'd OpenACS), and its author wants it to remain under a BSD license, then minimally, if I were the author, I'd include some license text in the code making that very very clear. Note that the AOLserver codebase has prominent notices top about the AOLserver license at the top of every major source file, etc.

Collapse
Posted by Gustaf Neumann on
found another issue with xowiki, that showed up due to the new version. please get xotcl-core 0.28 from the link above as well.
Collapse
Posted by Malte Sussdorff on
Gustaf, I usually run a mix of oacs-5-2 and HEAD. The steps involved are:

cvs co -r oacs-5-2 openacs-4
mv openacs-4 mysite
cvs co openacs-4
cd mysite/packages
rm -rf *packages_to_come_from_head*
cp -pr ../../openacs-4/packages/*packages_to_come_from_head* .

After that an update is done:

cd mysite/packages
cvs update -Pd * >update.log

Very easy to handle, works perfectly nice.

As for the license. I'm not entirely sure how the BSD license affects our code base, but as I did not detect a license agreement in xotcl-core and probably did not read the download page carefully I did not detect an incompatability. It is now in CVS anyway, will ask the OCT about it.

Thanks for the fixes, will look into them momentarily

Collapse
Posted by Nima Mazloumi on
Where can I find out if type has a class definition?
Collapse
Posted by Nima Mazloumi on
On a 5.1.3 installation I tried this:

CrItem instantiate -item_id 2302162

knowing that the item is a file storage object. I got this error:

ERROR:
invalid command name "::file_storage_object"
while executing
"$object_type create ::[expr {$revision_id ? $revision_id : $item_id}]"
(procedure "instantiate" line 12)
::Generic::CrItem->instantiate
invoked from within
"CrItem instantiate -item_id 2302162"
("uplevel" body line 1)
invoked from within
"uplevel 1 $script"

Oh. You mean I need to have class definition for file storage objects?

Collapse
Posted by Dave Bauer on
Nima,

File storage object is a poor example because it is not a properly defined type and doesn't have a proper type specific table.

Collapse
Posted by carlos lucero on
Hi,

i was tried the xotcl-note package and occur the following 
trouble just when i tried add sample note or subnote:

can't read "page_title": no such variable
    while executing
"lang::util::localize ${page_title}"
    invoked from within
"set __adp_properties(title) "[lang::util::localize ${page_title}]""
    ("uplevel" body line 5)
    invoked from within
"uplevel {
          set __adp_output ""

    set __adp_master [template::util::url_to_file "[ad_parameter -package_id [ad_conn subsite_id] DefaultMaster ..."
    (procedure "template::code::adp::/var/lib/aolserver/test5/packages/xotcl..." line 2)
    invoked from within
"template::code::${template_extension}::$__adp_stub"
    (procedure "template::adp_parse" line 68)
    invoked from within
"template::adp_parse [file root [ad_conn file]] {}"
    (procedure "adp_parse_ad_conn_file" line 5)
    invoked from within
"$handler"
    ("uplevel" body line 2)
    invoked from within
"uplevel $code"
    invoked from within
"ad_try {
                $handler
            } ad_script_abort val {
                # do nothing
            }"
    invoked from within
"rp_serve_concrete_file [ad_conn file]"
    (procedure "rp_serve_abstract_file" line 60)
    invoked from within
"rp_serve_abstract_file "$root/$path""
    ("uplevel" body line 2)
    invoked from within
"uplevel $code"
    invoked from within
"ad_try {
            rp_serve_abstract_file "$root/$path"
            set tcl_url2file([ad_conn url]) [ad_conn file]
            set tcl_url2path_info..."

had somebody the same problem? 

i tried the xotcl-note-05 package in oacs 5-2, with xotcl-core-0.36, xotcl-1.4, i applied the corresponding patchs to xotcl-core, according 
http://www.openacs.org/xowiki/pages/en/xotcl-core

thanks in advance.
Collapse
Posted by gustaf neumann on
Carlos,
here is an updated version in sync with the current xotcl-core versions.

http://media.wu-wien.ac.at/download/xotcl-note-0.6.apm

best regards

Collapse
Posted by Mark Aufflick on
Boy have I been missing out on all the action by not keeping up to date on the openacs massage boards. Gustaf you're a genius. I can't wait for a project to try xotcl-core out on.

Imagine - an OO interface to the content repositary. Amazing!

Collapse
Posted by Stan Kaufman on
I agree that xotcl is amazing (brilliant work, Gustaf!), but it also precipitates situations that appear to be "a riddle inside a mystery wrapped in an enigma" (as Churchill once said).

To wit: with the latest xotcl-note package (0.6) with an oacs-5-2 install (xotcl-core 0.38), when adding a first ever note, this server error results:

Database operation "0or1row" failed (exception NSDB, "Query was not a statement returning rows.")

ERROR:  null value in column "name" violates not-null constraint
CONTEXT:  SQL statement "insert into cr_items ( item_id, name, content_type, parent_id, storage_type ) values (  $1 ,  $2 ,  $3 ,  $4 ,  $5  )"
PL/pgSQL function "content_item__new" line 89 at SQL statement
PL/pgSQL function "content_item__new" line 20 at assignment

SQL: 
	select content_item__new(NULL,-100,null,null,null,'506',null,null,
				 'content_item','::CrNote','Test Title',
				 'Test description',NULL,
				 NULL,'{Test content } text/html','text')
    while executing
"ns_pg_bind 0or1row nsdb0 {
	select content_item__new(:name,-100,null,null,null,:creation_user,null,null,
				 'content_item',:object_type,:title,
				..."
    ("uplevel" body line 1)
    invoked from within
"uplevel $ulevel [list ns_pg_bind $type $db $sql]"
    ("postgresql" arm line 2)
    invoked from within
"switch $driverkey {
                oracle {
                    return [uplevel $ulevel [list ns_ora $type $db $sql] $args]
                }
       ..."
    invoked from within
"db_exec 0or1row $db $full_statement_name $sql"
    invoked from within
"if {[regexp -nocase -- {^\s*select} $test_sql match]} {
....etc etc etc

Obviously the attribute "name" isn't getting defined so the CR barfs, but where is the problem? Alternatives:

  • Maybe name needs to be included in the "fields" set in the ::Generic::Form constructor in /packages/xotcl-note/www/edit.tcl?
  • Does this constructor need some param that feeds into the new_data block of ad_form (which ::Generic::Form wraps, I think)? How do you setup on_submit, new_data, and other ad_form-type blocks with ::Generic::Form?
  • Maybe there's a bug in ::Generic::Form itself such that "name" isn't getting initialized from "title" or something else?
  • Maybe there's a bug in the CR itself? That's virtually impossible since the CR seems to work fine elsewhere, but how can you tell from the available error info?
  • And is this a problem in the xotcl code, the tcl code, the plpgsql code, or the sql? Again it's almost certainly just a bug in the xotcl, but in a more complex case it would be very hard to tell, eh?
  • Or is it something else?

Anyway, in the Old Days where add.tcl set up the form, add-2.tcl processed the form for errors etc, and add-3.tcl stuffed the database -- and all SQL was inline -- it was easy to tell exactly and immediately where problems occurred. Now we wrap things in the CR, ad_form, ::Generic::Form, plus a few callbacks to keep you looking in the rear view mirror while you drive, and it's, well, confusing.

When things work well, the kind of code streamlining and simplification that xotcl etc can produce are astounding. But in the time I've spent so far trying to figure out why this simple illustration package won't create a new note, I could have coded from a blank page an entire old-style package. That subtracts a bit from the efficiencies. I'll catch on eventually, though.

Collapse
Posted by Gustaf Neumann on
sorry for this. i was not paying notice xotcl-note. there was a confusion between "cr name" and "cr title" in the code of xotcl-core, xotcl-note and xowiki. xowiki 0.27 fixed that http://www.openacs.org/forums/message-view?message_id=434040
but i did not upgrade xotcl-note. your first impression was correct, "title" had to be replaced by "name" in the form fields. Please get the new xotcl-note 0.7 from http://media.wu-wien.ac.at/download/.

This update triggered one more bug related to this issue in xotcl-core. please update xotcl-core from cvs oacs-5-2 as well.

Collapse
Posted by Stan Kaufman on
Thanks, Gustaf. That indeed fixes things, and shows that it really wasn't that complicated a problem. Still, the ascent up Mount XoTcl is rather steep. 😉
Fixing this problem without knowing the framework and the history is no xotcl-beginners task. in my (biased) opinion i don't think that xotcl is complex, but it turns out that people try with xotcl more complex and abstract things than with pure tcl.