Forum OpenACS Development: duplicate queries in .tcl, .xql, -oracle.xql, -postgresql.xql

I really like how the query processor lets us separate out our SQL in
an xml file. However it seems this work has not yet been completed and
I am not quite sure how the query processor chooses queries. For
example, the file apm-procs.tcl contains sql queries, as does
apm-procs.xql, apm-procs-postgresql.xql and apm-procs-oracle.xql.
Should the queries in apm-procs.tcl not be elliminated, are they ever
used? Are the queries in apm-procs.xql the ones that are common to all
databases that we support?

Thanks for helping me catch up on query processing!

Just from experience (not having looked at the query parser code), the precedence seems to be:

-postgres.xql/-oracle.xql

.xql

.tcl

The queries in the .xql file are common to all databases, but can be overridden in a db-specific xql file.  For instance, the .xql file might have a query with a SQL92 outer join.  This would work fine with postgres but not with Oracle 8x, so it will appear in the oracle xql file, but not the postgres xql file.  As for queries in the tcl files, yes, they could be eliminated.  They're only used when no corresponding query exists in a .xql file.  I personally like having them inline, as it's easy to see what the query is supposed to do.  It also speeds up development greatly to keep everything in the tcl file, then split it out later with automated scripts.

On a somewhat related note, it would be really nice if there was an extension to Emacs somewhere that would let you look up the query in the .xql file when your cursor is over the query reference in a .tcl file.  Anyone built such a beast?

I have lost time in debugging because I thought the query in the tcl file was getting run, when it was really being pulled from an xql somewhere.

I can only offer this kludge below, which binds C-c a to opening the corresponding xql file, C-c a the adp file and so on. Additions highly welcome.
;; define shortcuts to open the corresponding xql/tcl/whatever file to
;; the current open buffer.
(defun acs-find-file (fileext) "open acs buffer" (interactive "s")
  (find-file (concat (substring (buffer-file-name) 0 (string-match "(.tcl|.adp|-postgresql.xql|-oracle.xql|.xql)$" (buffer-file-name))) fileext)))


;; don't know how to bind functions with parameters to keystrokes, so
;; i have to define one function for each possible parameter value
(defun acs-find-file-adp () "" (interactive) (acs-find-file ".adp"))
(defun acs-find-file-tcl () "" (interactive) (acs-find-file ".tcl"))
(defun acs-find-file-postgresql () "" (interactive) (acs-find-file "-postgresql.xql"))
(defun acs-find-file-oracle () "" (interactive) (acs-find-file "-oracle.xql"))
(defun acs-find-file-xql () "" (interactive) (acs-find-file ".xql"))

(global-set-key "C-ca" 'acs-find-file-adp)
(global-set-key "C-ct" 'acs-find-file-tcl)
(global-set-key "C-cp" 'acs-find-file-postgresql)
(global-set-key "C-co" 'acs-find-file-oracle)
(global-set-key "C-cx" 'acs-find-file-xql)
i use ctags to find the queries in the xql files. but as mentioned above, sometimes you don't realize that there is a query in the xql file and that is the actual query running. i think we should remove queries from the tcl files. it makes things confusing and i have found that they are not kept in sync with the queries in the xql files. it becomes very confusing and not easy to manage.
I agree with both... It's much easier to write code using SQL right in the Tcl.  The second you extract those queries, though, the queries should be removed from the Tcl file.  I find that the general niceness of having a little bit of reminder code in the Tcl file to give me the gist of what's going on is totally outweighed by my propensity to start editing that query and then scratch my head for a few minutes (or more) when it does nothing to the app.  Luckily now conditioning has kicked in and I figure out the problem pretty quickly or manage to avoid it altogether, but it was a painful pavlovian experience that I wouldn't wish on anybody else.
i think it is fine to have the queries in the tcl file during development of an app, but once you decide to commit it to the openacs tree i think everything should be moved to xql files for everyone else's sanity. many developers that are new (and even some that are not so new) to openacs development get confused by this.
Something like the following works reasonably well in xemacs (probably it also works in regular emacs):

(defun dcw-find-query ()
  "run grep on selection 
   Uses x-get-selection to get the grep string"
  (interactive "")
  (let* ((file-name 
          (concat (substring (buffer-file-name) 0 
                             (string-match ".tcl$" 
                                           (buffer-file-name))) "*.xql")))
    (grep (concat "grep -n "" (x-get-selection) "" " file-name))))

Then bind this function to a function key. To use just highlight the query name in the .tcl file and hit the function key.

All of the above is correct. Let me summarize to make sure everything's clear.

Precedence

  • foo-[oracle/postgresql].xql
  • foo.xql
  • foo.tcl

As noted, there are no errors if a query is in specific as well as generic .xql files. The outer join example from a previous point illustrates why. PG isn't the only RDBMS to support standard syntax. We started out with the notion that we might support several RDBMSs (and we may, yet, who knows). All that use the standard SQL 92 outer join syntax can share the generic form, only Oracle needs the oracle-only form. In other words the SQL 92 version isn't in the PG .xql file because it's not PG-specific but standard. All standard queries should go in the generic query file.

Does this make sense?

It would be good to strip out the queries from the .tcl files, yes. In fact if any one wants to play with the Query Extractor to investigate automatically doing this it might be an interesting exercise (I've considered this many times but just never seem to have the time to dig in).

My longer-term notion is to change the syntax of the db_* API:

db_foo query_name
grabs query from query file (as db_map does today) while
db_foo -query $sql
would execute the inline query. The idea being that some queries will always be db-specific (check out the schema browser if you don't believe me!) and might as well be inline (the schema browser has entirely different proc lib files for the two dbs) and we still want the ability to conditionally build up one or another query using db_map, perhaps passing the result to a proc that does the actual call.

This would make clear whether or not a QD entry exists, would allow the QD to give an error if a query's not found (it now just passes the query from the Tcl file), etc.

I use ctags/etags to index the .xql and -postgresql.xql files so that in Emacs M-. can find a named query in the XQL files. This approach is not 100% fool proof as M-. finds the FIRST query by that name. This could be in a file that doesn't correspond to the .tcl file containing the named query.

Nevertheless, I find it useful in most cases. Here's the command line to add queries to the Emacs TAGS file:

find . -type f -name "*.xql" -and ! -name "*-oracle.xql" -exec etags --append -o TAGS --lang=none
      --regex='/[     ]*<(partial)?(full)?query name="([^     ]+)">//' {} ; -print

Please note that this command does NOT index the oracle files. This is because I work predominantly with PostgreSQL.

In my last post I use x-get-selection to get the query name, but this won't work when doing remote work using a tty. To handle that case, you cold use something like the following:

(defun dcw-find-tag-default ()
  "return a string near the point"
  (interactive "")
  (save-excursion
    (while (looking-at "sw|s_")
      (forward-char 1))
    (if (or (re-search-backward "sw|s_"
				(save-excursion (beginning-of-line) (point))
				t)
	    (re-search-forward "(sw|s_)+"
			       (save-excursion (end-of-line) (point))
			       t))
	(progn (goto-char (match-end 0))
	       (buffer-substring (point)
				 (progn (forward-sexp -1)
					(while (looking-at "s'")
					  (forward-char 1))
					(point))))
      nil)))


(defun dcw-find-query ()
  "run grep on selection 
   Uses dcw-find-tag-default to get the grep string 
   that encloses the point"
  (interactive "")
  (let* ((file-name 
          (concat (substring (buffer-file-name) 0 
                             (string-match ".tcl$" 
                                           (buffer-file-name))) "*.xql")))
    (grep (concat "grep -n "query  *name=.*" (dcw-find-tag-default) "" " file-name))))


To use this, you only need to place the cursor on the query name (highlighting the query name is not necessary) string and then hit the function key which is bound to this function. I've also modified the grep string to be more selective, so it should now only return the relevant queries.

Since, the bboard ate some backslashes, I'm reposting:

(defun dcw-find-tag-default ()
  "return a string near the point"
  (interactive "")
  (save-excursion
    (while (looking-at "\sw\|\s_")
      (forward-char 1))
    (if (or (re-search-backward "\sw\|\s_"
				(save-excursion (beginning-of-line) (point))
				t)
	    (re-search-forward "\(\sw\|\s_\)+"
			       (save-excursion (end-of-line) (point))
			       t))
	(progn (goto-char (match-end 0))
	       (buffer-substring (point)
				 (progn (forward-sexp -1)
					(while (looking-at "\s'")
					  (forward-char 1))
					(point)))) nil)))



(defun dcw-find-query ()
  "run grep on selection 
   Uses x-get-selection to get the grep string"
  (interactive "")
  (let* ((file-name 
          (concat (substring 
                   (buffer-file-name) 0 
                   (string-match ".tcl$" (buffer-file-name))) "*.xql")))
    (grep (concat "grep -n "query  *name=.*" 
                  (dcw-find-tag-default) "" " file-name))))

Hope this works.

Blah, I missed one.  There should also be a backslash before .tcl$, but it will probably work anyway.