Forum OpenACS Development: Query Extractor and Query Dispatcher

Collapse
Posted by Gilbert Wong on
This message is long. I have a few questions surrounding the Query Extractor and the Query Dispatcher.

1. Query Extractor: picking up wrong procs?

Is this the correct behavior for the Query Extractor?

from ecommerce-widgets-procs.tcl

proc ec_creditcard_widget { {default ""} } {

(no db query statements)

}


proc_doc ec_date_widget {column {date ""}} {Generates a date widget.} 
{
  switch $date {
    now {
      set date [db_string date_widget_select "select to_char
(sysdate, 'YYYY-MM-DD') from 

dual"]
    }
  }

...

}

proc_doc ec_time_widget {column {time ""}} {Generates a time widget.} 
{
  switch $time {
    now {
      set time [db_string time_widget_select "select to_char
(sysdate, 'HH24:MI:SS') from 

dual"]
    }
  }

...

}

extracted to ecommerce-widgets-procs-postgresql.xql

...

<fullquery name="ec_creditcard_widget.date_widget_select">
      <querytext>
      select to_char(current_timestamp, 'YYYY-MM-DD')
      </querytext>
</fullquery>


<fullquery name="ec_creditcard_widget.time_widget_select">
      <querytext>
      select to_char(current_timestamp, 'HH24:MI:SS')
      </querytext>
</fullquery>

...

If I understand how the QD works, shouldn't the fullquery names be:

ec_date_widget.date_widget_select 
ec_time_widget.time_widget_select

For some reason, the QE latched on to ec_creditcard_widget and used that.

2. Query Extractor: query passed in as variables

I have several instances where the Query Extractor missed queries of this format:

set sql_query "some sql query ..."
db_foreach query_name $sql_query { ... } 

It didn't extract the query and write it to the appropriate .xql file. Is this a normal behavior for the QE?

3. Query Dispatcher: "documented proc"

I got the following error message (in debug mode) when running a query:

[17/Jul/2001:18:13:27][24011.5][-conn0-] Debug: QD_LOGGER = proc_name 
is 

-ec_category_widget-
[17/Jul/2001:18:13:27][24011.5][-conn0-] Debug: QD_LOGGER = there is 
no documented proc with 

name ec_category_widget -- we used default SQL
[17/Jul/2001:18:13:27][24011.5][-conn0-] Debug: QD_LOGGER = PRE-QD: 
the SQL is
    select c.category_id, c.category_name,
           s.subcategory_id, s.subcategory_name,
           ss.subsubcategory_id, ss.subsubcategory_name
      from ec_categories c, ec_subcategories s, ec_subsubcategories ss
     where c.category_id = s.category_id (+)
       and s.subcategory_id = ss.subcategory_id (+)
  order by c.sort_key, s.sort_key, ss.sort_key for dbqd..NULL

...

The query was in the form of the previous problem (#2 above). So I had to manually add the code in the .xql file.

from ecommerce-widgets-procs.tcl

proc ec_category_widget { {multiple_p "f"} {default ""} 
{allow_null_categorization "f"}} {
...
    set sql "
    select c.category_id, c.category_name,
           s.subcategory_id, s.subcategory_name,
           ss.subsubcategory_id, ss.subsubcategory_name
      from ec_categories c, ec_subcategories s, ec_subsubcategories ss
     where c.category_id = s.category_id (+)
       and s.subcategory_id = ss.subcategory_id (+)
  order by c.sort_key, s.sort_key, ss.sort_key"

    set category_counter 0
    set old_category_id ""
    set old_subcategory_id ""
    db_foreach get_category_info_joined_w_children $sql {
        incr category_counter
...

    }

...

from ecommerce-widgets-procs-postgresql.xql

...

<fullquery 
name="ec_category_widget.get_category_info_joined_w_children">
      <querytext>

    select c.category_id, c.category_name,
           s.subcategory_id, s.subcategory_name,
           ss.subsubcategory_id, ss.subsubcategory_name
      from ec_categories c
           LEFT JOIN ec_subcategories s using (category_id)
           LEFT JOIN ec_subsubcategories ss on (s.subcategory_id = 
ss.subcategory_id)
  order by c.sort_key, s.sort_key, ss.sort_key

      </querytext>
</fullquery>

...

Looks like the QD isn't picking up this query. I hope I didn't make any spelling error :) Any ideas on the above problems? Thanks!

Collapse
Posted by Gilbert Wong on
Okay.  I think I figured out #3.  I changed proc to proc_doc and the correct query was selected.  Was this the intended behavior?  Thanks.
Collapse
Posted by Dan Wickstrom on
The QE won't pick up sql that is passed in as variable.  It's a regular expression based parser, and it has no way of evaluating the code, so if all you have for a query is a variable name like $sql, it won't be able to figure out the query.  To get around this, I've always done prep work to move the queries inline if possible so that the QE will be able to automatically extract them.  The QE is not 100% reliable, so you have to expect to have to handle some of the queries manually.

As far as the other problem, the QD relies on information provided by proc_doc, which is an alias for ad_proc, so you need to declare all of the procs with either proc_doc (which is deprecated) or ad_proc.  CMS, CR, and templating did not use either proc_doc, or ad_proc, so I had to go change all of the proc delcarations before running the QE.  I imagine that there might be quite a few other modules that need this type of prep work before running the QE.

Collapse
Posted by Jonathan Marsden on
Dan, the QE need for ad_proc is probably something that should probably be added to the porting FAQ and/or the wimpypoint page about "Using the Query Extractor".
Collapse
Posted by Gilbert Wong on
Dan -  Is the proc_doc/ad_proc requirement documented in the porting guidelines?  I did a quick sweep of the documentation but didn't see any reference to it.  It might be helpful for other porters.  Thanks.
Collapse
Posted by Don Baccus on
The dependency's in the Query Dispatcher ... I just added a new page to the wimpy point presentation that mentions this and some other stuff.
Collapse
Posted by Kapil Thangavelu on
I haven't done any work on the query extractor in several months
after being told that the openacs team had what it needed, but as
the original author of the extractor i think i can answer these
questions. In addition, i'm planning on doing some additional work
on the extractor in the near future so if you have any feature
requests/bug reports (with logs) please email them to me or submit
them directly to http://sindev.dyndns.org/Projects/OpenACS/tracker
(i'll be moving this weekend so the site will be down for a day).

first, if you're using the query extractor please take a look at the
README and use the log files as their there to help identify
situations which require manual intervention.

Gilbert the case you point out where the extractor is identifying
the wrong proc query association, is clearly an extractor bug. if
you could send me the log files from which generated this i'll work
on it.

The case with dynamic queries was a design decision that was made
before information regarding how the QD was to handle partial and
fully dynamic queries was known. In the case of a fully dynamic
query the QE will write a message to the logfile and ignore the
query. In the case of partially dynamic queries the QE will write
out the query as a fullquery with a message in the logs. I'll change
the behavior of the partially dynamic query into the now standard
partial query syntax. I'm not sure what the proper standard syntax
is for fully dynamic queries in xql files... ?

Some of the modifications by the openacs team, involved disabling
some of the logging, so I'm not 100% sure that all the relevant info
is logged.

the QE works with both proc and ad_proc as noted in the README. i
believe Dan did some work to make it also work with the template
dbapi.

As Dan noted the QE is not 100% reliable, it is meant as tool to
automate alot of the drudgery of porting. use it, save time, and
have fun porting, but do check over its output and the logfiles.

Incidentally if anyone is interested in porting to another rdbms,
please contact me, as i've done some preliminary work on tools to
automate the ddl (including function prototypes) porting to other
databases.

Collapse
Posted by Don Baccus on
I'm not sure the QE is going to be able to help with dynamic queries, which are being mapped with db_map calls in the "set" command that builds a query snippet (or 100% dynamic queries).

Flagging fully dynamic queries would be sufficient.

If I have:

set foo "select foo from bar"

db_... dynamic_foo $foo ...

The translation is

set foo [db_map foo "select foo from bar"]

db_... dynamc_foo $foo

There's really nothing in the "set" statement that the QE can recognize.

Also queries buried in query files can now have Tcl variables in them,  so you can have something like:

<fullquery ...> ...

select foo from bar where $foo_where_clause

</fullquery>

And the Tcl variable will be substituted.  These can be put into the .xql file (I think they are already?).  The "set" command that builds "foo_where_clause" that needs to be modified to use db_map.  A flag in the .xql file indicating a dynamic query would help folks out, though grepping for dollar signs helps too!

Collapse
Posted by Gilbert Wong on
Kapil,  I'll send you the logs tonight.

I have one more question.  For the tcl procs, I noticed that some fullquery names were extracted as  proc_name.query_name and some were extracted as query_name only.  Just to confirm, they correct notation should be proc_name.query_name, right?

Collapse
Posted by Kapil Thangavelu on
Dan, looking over your example i'm not sure how much the QE can help
here. As you suggested i think flagging is probably the best way to
go. Currently their will be messages to the logs, but i can alter
the QE to generate a flag in the xql file as well. Queries with tcl
vars are written out with the fullquery syntax and a corresponding
message in the log (warning) about a Partial Dynamic Query. i'm not
sure what the difference is between what you just posted and a
partial query syntax though, i need to look at the source a little
more.

Gilbert, the QE was developed against the acs-packages repository
from the aD cvs, which contained most of the packages that currently
make up the openacs. the ecommerce package though was not from aD,
and due to different coding conventions and because its a
translation from a 3x module are likely to cause the QE some
problems, due to the fact that its mainly a regex parser which has
been tuned for source conventions in the aD package cvs. i'm going
to run the extractor against the openacs-4 ecommerce cvs, and try to
fix things, so they'll work better for you.

for tcl library files, the extracted query name should consist of
proc_name.query_name.

i'm very busy at the moment, so while i'm going to try and do this
asap, it very well might not get done till i get back from the
oreilly conference next friday.

Collapse
Posted by Kapil Thangavelu on
s/Dan/Don sorry
Collapse
Posted by Gilbert Wong on
Kapil - I've done a first pass port of most of the queries already so there isn't a great urgency to fix the QE for any of the bugs I mentioned (for me at least).  I've been comparing the QE results to the original procs, and for the most part, they are coming out okay.  I'm going to do a detailed second pass through the ecommerce procs to verify the QE results and the manual edits I did.  I think the ecommerce package is important enough for me to sit there and do this comparison 😊
Collapse
Posted by Don Baccus on
There's really no semantic difference between fullqueries and partial queries at the moment, they're just meant to be informative for human readers.  There *could* be semantics hooked onto these, though, sometime in the future.

The intent is that an incomplete query set to a tcl variable via db_map should be labelled as a "partial query", while any query passed  to the "normal" db_* routines (db_string etc) should be labelled a "full query".