Forum OpenACS Development: Why XML for .xql query dispatcher files?

I was just watching my Dev server take a very long time to start up as usual, do in large part to parsing all those query files with ns_xml, I think. (The coming switch to tdom should probably speed that up a lot, of course. And it is running on a slow Sun box currently.)

And I got to wondering, way back when you folks first took ACS 4.2 and started hacking it into what became OpenACS 4.5, why did you choose to use XML for the query dispatcher files, rather than, say, just using Tcl, perhaps in the style of the AOLserver config.tcl file?

Here are a couple old threads from March and June 2001 which convey some of the thinking at the time, but I'm more interested in how it looks to the OpenACS designers in hindsight. Knowing what you know now, if you were to go back in time to do it all over again, would you have taken the same path? Or done what differently?

Posted by Don Baccus on
Ben wrote it, an xml parser was available, he decided to use it.  That's most of it.  Dan and I were busy on other basic tasks, my first work on it was to teach the boostrap controller to automatically determine which db is being used, Dan jumped into porting the kernel datamodel.

We each worked independently.  That was Ben's piece of the foundation work, using XML seemed as good as any other idea, and after some public discussion he implemented it.

Also we had the notion of scoped searches for queries matching a name and that scoping wouldn't fit with a "define a Tcl proc for each query" approach.  Of course while scoped query searching is implemented we haven't taken advantage of it - queries get duplicated across various scripts and procs and that's probably OK.

We didn't know parsing the query files would be as expensive as it is.  With OpenACS 5.0/.LRN 2.0 we'll also be parsing catalog files, and while Peter's written that as a scheduled proc that doesn't block startup, it takes a lot longer to parse than the query files (it looks like we at least partially support a *lot* of languages.)  We'll want to go to lazy loading of the message catalog for this reason.  But lazy loading of queries isn't practical with the current implementation of the query dispatcher ...

Posted by Don Baccus on
Oh, yeah ... other languages.  For awhile there were some folks seriously interested in rewriting the toolkit in Python.  At that point in time ACS Java was being labelled a translation of ACS 4 to Java (rather than a full redesign.)

Putting the queries into a language-independent format allowed us to say "we're not interested in friggin' Python but we'll separate out the queries to make porting interested if you want to do it" etc etc.

Of course no one has ...

Posted by John Sequeira on
For portable.nsd, I convert all the XQL files to a TCL file to speed up the code/restart Apache/test cycle. You can download the code from the pnsd site if you're interested - it's pretty simple.

#Use this command to cache the xql into a .dat file for quick loading.
#This will indescriminantly load each and every query for all supported databases - it can be made more efficient

proc load_xql {} {

    if [ ns_config -bool pnsd/parameters LoadXqlFromCache ] {
	ns_log error "Your config file has disabled xql-file parsing.  Please change option pnsd/parametere LoadXqlFromCache"

    if [empty_string_p [info commands ::db_qd_load_query_file]] {
	ns_log error "db_qd_load_query_file is not defined.  Did you forget to pnsd::source_openacs?"

    set arrName OACS_FULLQUERIES
    set filename [file join $::pnsd::home xql.dat]
    uplevel 1 [list trace var $arrName wu [list ::pnsd::persist'save $filename]]
    set xqlfiles  [ ::fileutil::findByPattern $::pnsd::root -glob  *.xql    ] 

    foreach  file  $xqlfiles {
#	puts $file; 
	if {! [regexp postgresql $file]} {
	    db_qd_load_query_file $file



See this tcl wiki page for a look at array serialization procs I'm using.