Forum OpenACS Development: Invalid XML in XQL files

FWIW here's a test script

[... set root to root folder of openacs]
package require tdom
package require fileutil

set files [::fileutil::findByPattern  $root *.xql]
foreach file $files {
  set doc [dom parse [fileutil::cat $file]]
}

If this doesn't croak,  then the xql files are fine.

Someone should probably ping Ben on this since he wrote the QD, but I think Jeff's summary of my position's more or less correct.  Another way to put it: the Query dispatcher transforms .xql files into valid XML before parsing them and my presumption has been that Ben did this by design, to make the query files less obnoxious to write.

But I've never asked him ...

Caching for quicker startup would be nice, of course, but that's another issue.

Yet another issue that we need to tackle soon is to drop libxml and switch to Zoran's tDom AOLserver module.  It was supposed to be "released any day now" late last summer so one would hope it's actually available now?  Does anyone know?  ns_xml only exposes a trivial subset of libxml and since everyone seems to believe that Zoran's stuff's better than libxml and since supposedly he's exposed everything in his AOLserver module, it seems much more reasonable to switch rather than extend ns_xml into something more useful.

Sorry if this has already been discussed.  Is tDOM a drop-in replacement for nsxml?

As much as I hate having queries in a seperate file, I am not sure if pre-processing the tcl is such a good idea. If the first step in installing openacs is "compile the actual tcl scripts" I would have serious concerns that the we would see two things

• Mysteriously broken code which would result from errors in the parser or quirks in the particular source tcl.
• even more difficulty in merging improvements from particular sites back into the code base.

If it were a cleaner language or if we had a full parser I would not worry quite so much but regardless of how good the parser is, the later problem will remain.

I don't see a lot of evidence that the QD overhead is anything to be that concerned about relative to going to the DB in the first place.

I think there is probably a better argument for precompiling adp files rather than doing them on the fly, although there it is much clearer that what you are doing is compiling and no one would really be tempted to discard the adp's and work directly with the resulting code.

I would like to propose an alternative to the megafile.  I know there's a python script somewhere that takes SQL out of the tcl files and into the XQL files.  I've prototyped a script that does the reverse - takes queries out of the XQL files and inlines them into the TCL script.  It then deletes/renames the XQL files so that they're not found and parsed on start-up - the query-dispatcher commands effectively become nop's.

The advantage of using this is that you don't take the XQL parsing hit on server start-up (which is substantial),  and you don't have to run through the query dispatcher code on each query or chew up more memory caching queries that are stored in bytecode-compiled tcl.

I initially did this so that I could get the openacs source code to play nice with nstcl,  but it seems like it could be useful to others.  I can definitely see why you'd want to have XQL files - they came in handy when I worked on porting the OpenACS kernel to MSSQL,  for instance.  However,  for people developing apps in OpenACS (as opposed to people developing Openacs) it seems as though the XQL abstraction just slows things down.

Using the script wouldn't require re-architecting OpenACS or even any kind of direct support,  but I just wanted to broach the idea before time was spent on upgrading the qd.

Roberto,

It doesn't currently handle db_map.  I think there are only about 10 uses of it in the kernel,  so I hadn't gotten there yet.

It seems as though replacing db_map calls with { db_map_sans_qd  "string-from-xql-file" } wouldn't be very difficult...

I'll email you the script.

Everyone's made some good points as to why openacs developers wouldn't want to inline queries.  And Jeff's point about the parser being difficult to implement is well-taken.

For my current effort,  OTOH,  minimizing startup time is of huge benefit.  I spent 15 minutes and implemented OACS_FULLQUERIES as a persistent array (see http://mini.net/tcl/3469 ),  and just made db_qd_load_query_file a no-op.  This benefited startup time enormously,  as you'd expect,  and my work now proceeds faster.  OpenACS' implementation of OACS_FULLQUERIES is different than my/nstcl's own,  but using a trace in this way would probably achieve the desired results.

Anyway,  although I can see why it's not appropriate for the toolkit,  I have only worked on single-db projects to date and I still like the idea of inlining.  I might revisit it someday by porting my ugly perl script to a tcl one that uses ParseTools ( http://www.oche.de/~akupries/soft/ptools/ ).  If it worked as advertised,  it might actually do the trick.

People who are developing single-DB applications in OpenACS don't need to use query files at all for their custom code.

People who are developing multi-DB applications will want use query files.

The QD overhead, as Jeff suggests, appears to be very low when compared to the expense of executing a query.  There's really nothing to be gained here performance-wise.

It does require some memory ... on OpenACS.org about 3MB IIRC (we looked at this when trying to figure out why our site was eating up so much memory).  On a modern machine that's not a large number.  And if we follow through and pull queries out of the Tcl files belonging to standard OpenACS project supported packages of course the compiled Tcl bytecode will shrink by a roughly comparable amount so the overall overhead is very low.

The single issue which I think all would agree is important is the start-up issue, as start-up time can be fairly slow on sites that use a large number of packages.  It's not just query files, though - we're compiling every */tcl/*-[init/procs]*.tcl file on startup, too.  It's not clear how much can be done about this.

I really don't see much value in a tool that folds .xql files back into .tcl files, to be honest.  It's not something I'd ever use.