Forum OpenACS Development: OpenACS Developer tools

Posted by Kapil Thangavelu on
the extractor tool, i was talking up earlier this week is out and
ready for developers.

it can be downloaded from

it conforms to the latest additions to the query dispatcher xml
spec, as referenced on the bboard and the files in cvs.

i rewrote the extractor's parser so its a lot cleaner, faster (cut
running time in 1/3:), and does better assoc of queries->procs.

running it on the acs-packages cvs takes about 4.4s on my laptop,
with no abnormal errors. the errors that were generated were all
from library files that either use namespaces, do db calls outside
of procs, or db calls in ad_proc_register_filter. if people feel
that these types of things should be added in let me know.

theres a good amount of documentation included (including some auto
generated documentation of the apis).

i'm going to work on transforming the package files to html tomorrow
and then i'll start helping out with the port.

if you find any bugs, or have some comments post them here or email

Posted by Don Baccus on
I'll be looking at this over the weekend.  It has the potential to be a big help as it automates identifying and extracting (into separate files) the current defaul queries.

Thanks a lot, Kapil!

3: Nice Html of Package Info (response to 1)
Posted by Kapil Thangavelu on
i've uploaded a tarball to

which contains a nice set of html pages for browsing package
information. it includes a set of indexe pages that include packages
sorted by release date, number of dependencies, version number,
number of files, and also a detailed page for each packaged.

just untar and point your browser to the index.html

it would be nice if this could be setup on the web here on the, until a fully dynamic one based on either the apm or
sdm comes out.

please note that i'm moving the openacs-tools (the extractor) to

the script i used to write the html will be included in the next
release of the openacs-tools along with html generation for the
query files.

Posted by Kapil Thangavelu on
new release of openacs tools, from now on all files related these
tools can be gotten from

(my last post has several typo errors, doh).

included in this release are AcsPackageInfoWriter which generates a
set of html pages of package info for browsing and options to the
AcsSqlExtractor to write html output of the sql statements.

also new in this release are dependencies for the new features. the
core sql extractor that writes to xml does not need these but to use
other features you will need to install some python packages.

AcsPackageInfoWriter requires PyXML, DocumentTemplate, and DateTime
AcsSqlExtractor with html output requires DocumentTemplate

because these packages are incompatible with the GPL (which i'm
releasing the tool under), i can't bundle them with it.

i've made a tarball of the DocumentTemplate and DateTime packages
which are at the download location listed above.

PyXML can be gotten from

i'm done adding new features to the tools for a little while, only
bug fixes and customization of the current features.

so now i'm going to start working on the port, starting with
templating, and then porting over the datamodel for ecommerce4 and
workflow. i'm primarily focused on the sql models for package
porting so if anyone wants to work with me on the tcl, let me know
(esp. the ecommerce4 package which is incomplete).

btw. i'm being paid to work on the openacs4 port for now, so if
anyone needs some help with something let me know.

Posted by Don Baccus on
Dan's already working on workflow - let's not duplicate efforts.

Can you package up the query extractor by itself so we don't need to download things like pyXML in order to run the basic extractor?  Or am  I reading too much into your post?

I'm going to e-mail you some comments on the query extractor, overall it looks like a great start...

Posted by Kapil Thangavelu on

its already packaged so you don't need the support libs unless you
specifically use features that require it. which are namely the
package info html generator or the extractor WITH html generation. the
default extractor with xml generation does NOT require the support libs.

i accidently left enabled the html generation in the default
configuration for the extractor which requires the document templates.
turning this off in the config (by commenting it out) will remove the
need for the document templates.

as far as what to work on, its quickly becoming apparent that we need
a package by package break down of who is doing what, and we need it
soon. exchange by private email and geographically restricted meetings
is not cutting it, we need this info publicly available on the web. so
two of the three packages i stated above seem to have prior
commitments, thats fine by me. but this info should be known so work
isn't duplicated and effort wasted.

i'm installing the openacs4 at the moment, and i'm planning on writing
up an install guide based on my experiences.

Posted by Don Baccus on
Kapil ... I just started talking to folks about package assignments via e-mail about five or six hours ago.  I don't think it's terribly fruitful to post every question I put to an individual here on the bboard.  Group discussions - yes.  Individual posts - that's way too much detail.  Dan informed me sometime yesterday or early this AM that  he was going to start in on acs-workflow, because the content repository has a dependency on it.

Jon Griffin is going to look into acs-notification.  He and I exchanged e-mails just a few hours ago (see above).

So ... give folks a chance to get back to me before you demand that I summarize everyone's answers in public, OK?  It's hard to summarize responses that have yet to hit my e-mail inbox.

For the record,
you didn't answer my question as to whether or not you wanted to work on templating for FIVE DAYS (I posted it on April 3rd).  The fact that  I turned elsewhere FIVE DAYS after asking you in public and getting no response either in public or via private e-mail isn't really an indication that I'm plotting in secret to assign packages to  others behind your back or whatever.

As soon as we figure out exactly how we're going to structure the tree and how we'll decide who can commit where I will work on getting a list of who is doing exactly what and on which package.

I can't move forward on this sub-issue until I hear from Ben, and he's  incommunicado this weekend.  I'm sure he'll get back to me on Monday and we'll get things figured out and published by Wednesday.  The machine belongs to his company so questions about access are by nature  going to be his to make, not the group.

Cut us a little slack, Kapil.  I wasn't going to mention that it took you FIVE DAYS to get back to me on my question about templating, but if you're going to slap me around in public I'm going to slap back!

As I mentioned in my e-mail to you regarding the query extractor, it looks like a great starting point but there are some additional things that could be done that would make it even better.  The extractor's a great idea, I really appreciate your doing it, and when improved it is  going to remove a tremendous amount of tedious work from our plate.

This is where you can really help us the most at the moment.  At the moment Ben (or I if he doesn't have time tomorrow) need to modify the query dispatcher to allow the rdbms to be spec'd in the queryset as a default, and the extractor needs to build them that way.  As I mentioned, you don't need to build the list of bindvars unless -bind is used, and querytype is no longer needed.

Posted by Don Baccus on
so now i'm going to start working on the port, starting with templating, and then porting over the data model for ecommerce4 and workflow. i'm primarily focused on the SQL models for package porting so if anyone wants to work with me on the Tcl, let me know
At the risk of sounding more irked than I am...

Look, folks, anarchy isn't going to cut it.

Kapil, complaining that I'm not properly telling folks who is working on what while suggesting folks e-mail YOU rather than ME if they want to work on ecommerce isn't very productive.

If you want me to do a better job organizing this project, don't make my job harder by suggesting folks circumvent me. In seems rather unfair that you slap me because you don't feel the process is open enough while simultaneously working to make that job harder for me.

Likewise, don't just simply announce "I'm going to work on workflow" without checking with me first. If you'd checked with me I would've told you that Dan was working on it. For the record, Dan checked with me first to make sure that no one else had picked it up, just as he did for the acs-content-repository.

No one else is picking up ecommerce4 at this point, so you're more than welcome to it, and I'll put you down for it.

Posted by Kapil Thangavelu on

another release of the extractor tools is out. same place as before


mainly changes to become in line with the spec and comments made by don baccus.

  • removed query_type tag in xml
  • removed query text bind vars
  • added in experimental support for -bind vars (i'm still trying to figure out whats supposed to go here.)
  • made the default config file not require any extra libs
  • cosmetic changes to output
  • changed file output from .dbname.xql to -dbname.xql
  • added in docs of new features
  • checked xml output with pyxml

as for don's comments on this thread, i'm going to respond directly via email since IMO discussion of these comments in a public forum is of dentriment to the community.

Posted by Tom Jackson on

I saw the links at, but zope found nothing. Can Kapil or someone email me the tools (, I'd like to take a look.

Posted by Don Baccus on

If it was needed by the query dispatcher, it would've been included.

(I've e-mailed more details to Kapil, but the spec didn't suggest we needed support for them in the query dispatcher, and the spec was reviewed by Dan Wickstrom and I, who both worked on the bindvar parts of the driver, i.e. we would know if support was needed so our silence wasn't entirely out of ignorance...)

Posted by Kapil Thangavelu on
whoops... links fixed.
Posted by Domingo Alvarez duarte on
I wrote a small perl script that swap the parameter value with
the parameter name in function calls.

obj_type    => 'acs_object',
description => 'area'


'acs_object',  -- obj_type =>
'area'         -- description =>

save the code bellow to a file.

while (<>) {
    if (/(s+)(.*=>) (.*)/) {
            print "$1$3		-- $2
    else {
	print $_;

execute it:

perl file_where_script_saved file_to_swap_paramteres.sql
Posted by Don Baccus on
That's cute ...

You have to be *really* careful, though, because parameters used out-of-order in the Oracle version which don't cause the type signature to change will cause subtle and very hard to debug errors to creep into the code.

So ... tricks of this code should be combined with a *very* careful read of each call to make sure the parameter order is correct.