Forum OpenACS Development: Query dispatcher

Collapse
1: Query dispatcher
Posted by Michael Feldstein on 01/22/01 02:53 PM
Don mentioned that OpenACS 4.1 Core will include a "query dispatcher
(to provide a systematic method for replacing incompatible Oracle
queries with their SQL92 or PG equivalents without removing the Oracle
queries from the source)." Is this purely a way to keep a common code
base for ACS Classic and OpenACS, or are there time-saving benefits
(now or down the road) as well?
Collapse
2: Response to Query dispatcher (response to 1)
Posted by Don Baccus on 01/22/01 04:33 PM
It will actually cost us time in the short run (because it needs to be
written).  The benefit is really down the road.

We want to keep code compatibility with Oracle because this increases
the odds that OpenACS/ACS Classic can supply single modules that
"know" how to run with more than a single database.

This also will help simplify the support of other databases in the
future, if we choose to do so, because Postgres supports the standard
SQL92 for some features Oracle has private syntax for, most notably
outer joins.  Interbase and a couple of the other database systems of
interest also support the standard SQL92 syntax for this important
feature...

Collapse
3: Response to Query dispatcher (response to 1)
Posted by Michael Feldstein on 01/22/01 04:39 PM
I guess what I'm really trying to ask is whether the query dispatcher, once written, will make module porting faster. Will you still have to go in and change a lot of stuff by hand in order to port each module, or will the query dispatcher automagically translate some of the SQL on the fly, thus reducing the amount of code that needs to be changed?
Collapse
4: Response to Query dispatcher (response to 1)
Posted by Don Baccus on 01/22/01 09:55 PM
We're probably going to translate those that need massaging by hand,
as the automatic stuff we can do with regexp is just very simple stuff
anyway.

And Tcl8x leaks memory on regexps so we want to avoid doing them on
each and every query like we've done with OpenACS 3x/Tcl7x.  ACS 4.1
uses Tcl8x language features and we've got to use it ...

Collapse
5: Response to Query dispatcher (response to 1)
Posted by Dan Wickstrom on 01/22/01 10:29 PM
My impression was that the memory leaks in tcl8x had been fixed.  I ran some extended tests on aolserver3_2+ad10 with tcl8x a while back and there was not any evidence of memory leaks.  Does anyone have an example regexp that triggers the memory leak?
Collapse
6: Response to Query dispatcher (response to 1)
Posted by Don Baccus on 01/23/01 02:54 AM
Well, my information comes straight from Rob Mayoff, who first
reported it.  He seems to think they're still there.  Perhaps some of
them have been fixed, but not all???

Feel free to e-mail Rob if you want more information, he's very quick
to respond.

Collapse
7: My experience with the actual query dispatcher (response to 1)
Posted by Domingo Alvarez duarte on 04/22/01 09:51 PM
<p>
My description of whats is need to perform the pg port and do it in a generic way that can be used to port to other databases as well. </p>
<p>
The basic mechanism of the last query dispatcher is based on tag all queries and at runtime find a substitute query to tha actual db driver and use it instead of the original one. If there isn't any substitute try use the original one.
</p>
<p>
Ok it sounds fine but anyway we need go through the code line by line because some queries are dynamic generated and some are assigned to a string and passed as argument to a database procedure and <b>finally some of then are evaluated in another context breaking the actual query tag mechanism</b>, the first example I've got in <b>acs-developer-support</b> when I tried to use the actual xml scheme to write the substitution queries.
</p>
<p>
So it get so complicated to match every possibilitie. In my opnion we can use the actual idea but instead of use the query tag to find and substitute the query text we should use a local procedure that wil do it locally.
</p>
<p>
If someone knows where is the most apropriate space to discuss this please tell me.
I'll make an exmple with a small pacakge to demonstrate it.
</p>
Collapse
8: Response to Query dispatcher (response to 1)
Posted by Don Baccus on 04/22/01 11:16 PM
Please see my post regarding the query extractor, which will create first-cut queryfiles for you.  Download it and read my "README_PORTING" file.

I just uploaded this piece of code earlier today, however I did mention the existence of the query extractor in the thread where you mentioned that you'd started doing some porting.  I believe I mentioned that we were waiting for it before getting started on query porting in a big way.

We've been waiting for it to avoid the exact tedium you describe.

As far as dynamic queries you can just keep the Tcl code in the query.  The query dispatcher returns the query completed with the embedded Tcl, and the db API executor does a subst on the query in the caller's context.  In other words it does exactly what it did when inline.

We don't have a clean solution where the dynamic query is db-dependent.  In these (rare) cases we probably will need to  check the db_type in the .tcl file, as gross as that is (unless anyone has a better idea).

Collapse
9: Response to Query dispatcher (response to 1)
Posted by Don Baccus on 04/22/01 11:18 PM
As far as some being executed in another context than the caller's, we'll need an example.  This is surely very rare and should be discouraged whenever possible.
Collapse
10: Response to Query dispatcher (response to 1)
Posted by Don Baccus on 04/22/01 11:30 PM
Are you talking about those evil util_memoize calls that don't utilize  a proc wrapper but rather pass in the database query directly, where it then gets executed in util_memoize's context?

Those should be fixed by wrapping the query in a proc rather than passing "[list db_string etc etc]" into util_memoize.  The query dispatcher should then be able to properly associate the query with the wrapper proc.

This is a PITA but at least they can be found by grep'ing for "util_memoize".

We recognize that this first porting effort's going to take a fair amount of work, some of it tedious.  We can't automate the entire process of separating queries from Tcl but we can automate some of it.

The good news is that once we're one with this first effort porting to  other databases will be much simpler.

And, of course, unlike anything seen from aD we will be supporting two databases right off the bat.  Simply porting this to PG and abandoning  Oracle would certainly simplify our job but we want to support multiple databases.

Collapse
11: Response to Query dispatcher (response to 1)
Posted by Don Baccus on 04/22/01 11:38 PM
(sorry for the series of responses!)

It also may be possible to name the query passed into util_memoize differently, causing the query dispatcher to find it.  Or the QD could  search up the proc stack looking for a match rather than just in the current context.

In short, no need to write off the QD because of these calls.