Forum OpenACS Development: Templating Tool for Tcl

Collapse
Posted by Tom Jackson on
OpenACS has one of the best templating engines around I like it and have always found it easy to use.

The only drawback is that you have to have OpenACS to use it, and not all the Tcl code I write is in OpenACS.

Some years ago I wrote a template compiler with these goals:
1. safe from execution of arbitrary code.
2. syntax validation.
3. independent compilation and usage.
4. easy to track down syntax errors.
5. looks like Tcl, read manpages to get documentation.
6. minimum quoting rules.
7. whitespace control: easy to use with plain text or to write program code files.
8. interactive testing
9. very fast

I have just updated this tool, fixing a few issues I noticed over the years and added a bunch of additional commands. I have posted some examples and the compiler code here:

http://junom.com/document/twt/view/www/

http://junom.com/gitweb/gitweb.perl?p=twt.git;a=tree;f=packages/view

One example in the above is an operational Tic-Tac-Toe game. Unless I get a better idea, I'm calling this tool <tic-tac-toe>.

The README has a summary, which I'll repeat here:

Variables and arguments have a limited form. Neither can
contain [, not even array variables, this is for safety.

Variable references must begin with a letter.

In a template file, there is a minimum need to use escapes.
1. Dollar sign '$' only needs to be escaped if the following
text matches a variable expression. $1.24 or $ don't
need to be escaped.
2. Square bracket '[' only needs to be escaped if the following
text matches a command (see below). [file ...] would not need
escaping since the [file] command is not a template command.

Command arguments are delimited as words (see Tcl) with the
exception that no opening square bracket can be in the plain
text of an argument. Variable values can contain square brackets,
for example [regexp $pattern $string a b] where pattern could be:
{[a-zA-Z]}. Since $pattern contains a square bracket, it can't
be set in the .tmpl file, but must be created in the controller
script.

Commands all have the form [command arg arg ... *], where * is either
empty or /. The / slash indicates that any following whitespace
be removed. Using just ], preserves whitespace.

The [ws] and [ws/] commands either terminate, start or continue
whitespace removal.

The [comment arg ] or [comment arg /] commands allow a one arg,
potentially multi-line comment:

[comment {this is a comment}]

The [ws arg *] command is an additional method to comment.

The compiler is a syntax validating parser. When a syntax
error is detected, the parser exits, leaving behind a
partially compiled result. The result is easy to examine
to determine the location of the syntax error.

The compiler can also be used at a command line. Type in
the text of a template and see the result.

The result of compiling a template is a Tcl script which can
be sourced by a Tcl script which as set any required variable
references. The result of sourcing a compiled template is
stored in a local variable __string.

Compiler commands are in two broad categories:
1. Those which return a value
2. Those which don't return a value

Since some of the commands below have a dual mode in Tcl, only one mode
is supported. Viewing a compiled template or typing in commands
at a command line will quickly reveal which mode is used.

In general, commands which transform data but leave variable values
unchanged print out using:
append __string ...

Those commands which alter the value of a variable or are block commands
such as [if]...[/if] and [foreach]...[/foreach] are printed as script
level commands and don't append to __string.

Only commands which have distinctly different argument signatures have
the potential to support both modes. One example is [set]
[set a] ==> prints 'append __string [set a]'
[set a b] ==> prints 'set a b'

Currently [regexp] follows the second form. Although the signatures could
be different enough to support both, [regexp] returning a 0/1 indication
is impossible to use, as it would need to be used as an argument, which
cannot contain square brackets.

The compiler currently has these commands:

[if]..[elseif]..[else]..[/if]

[while]..[/while]

[foreach]..[/foreach]

[foreacharray]..[/foreacharray]

[switch]..[case] [/case]..[/switch]

[continue]
[break]
[expr]
[set]
[incr]
[string]
[format]
[scan]
[join]
[regexp]
[lindex]
[lsearch]
[llength]
[lset]
[lrange]
[lreplace]
[assign] (foreach varList dataList {break})
[mc] (::msgcat::mc)
[ws] (whitespace tool)

Collapse
Posted by Andrew Piskorski on
Tom, if I needed a web templating system that worked outside of OpenACS, the first thing I'd do is re-factor the OpenACS Templating code into standard Tcl packages, set up to work in plain AOLserver, and where feasible, plain tclsh as well. (Plus probably the same for the db_* database API, lots of the Tcl utility code, etc.) AFAIK, although real work, that should all be doable.

How many people here have anything more than an academic interest in another incompatible templating system? Whatever interesting features your stand alone templating system has, well, I'd rather see you consider integrating them into OpenACS!

Collapse
Posted by Tom Jackson on
Tom, if I needed a web templating system that worked outside of OpenACS, the first thing I'd do is re-factor the OpenACS Templating code into standard Tcl packages, set up to work in plain AOLserver, and where feasible, plain tclsh as well.

And? My templating system works even without plain tclsh for the compile part (can you run 'make'?). But any tclsh can read the compiled output.

But your comment "(Plus probably the same for the db_* database API, lots of the Tcl utility code, etc.)" is very instructive. This tool is the View component of the MVC design pattern. Excluded from the View component is the Model component...and the Controller component. This is the entire point from beginning to end.

If you look over all of the code I have written you will find that I don't produce highly specialized components. They are intentionally designed for reuse. This tiny part (templating) of a larger system is a good example. If you need the database component, I have improvements to the db_* API which work with just AOLserver (not yet just tclsh). Yes it works better than the db_* API, and works with abstract datasources like files, ldap, etc. This is not about the Model component.

I hold OpenACS in very high regard. If you disagree with this please find a comment where I have not shown this regard. I usually get in trouble here by supporting what I believe are the basic foundations of OpenACS. My usual, and probably tiresome complaint is that too many features get pushed down into the core functions.

So if you want to complain that I have not provided a large enough package, I will agree with that. This software solves exactly one problem. That is it.

But your comment seems very strange to me. First consider the fact that nobody has to use this code. And I agree that most, maybe all, here will not be using it. (If they do, I might have to do some work.) Why should this software conform to the needs of OpenACS? I'm not writing software for inclusion in OpenACS core, so who cares? Maybe this has been my point for year after year. I don't force my idiotic code on anyone else. I am happy to write my code for my own reasons. Yet somehow it is more acceptable to complain about code which has not even been proposed to become part of OpenACS than to complain about actual committed code. WTF?

Enough of that. I am very happy to see this comment:
"How many people here have anything more than an academic interest in another incompatible templating system?"

Yes incompatible. The ATS is so compatible with OpenACS. So when I introduce a templating system which is compatible with OpenACS and tclsh somehow that is incompatible. My templating system is about as incompatible as any other unix command. Assuming you can compile a simple program, if not, you probably cannot compile AOLserver or Tcl.

Finally, your comment: "Whatever interesting features your stand alone templating system has, well, I'd rather see you consider integrating them into OpenACS!"

They are integrated! They are as integrated as any other unix command. Does everything have to lick the OpenACS ass to satisfy you? Maybe we should reexamine the use of all this javascript bs. Somehow I don't think these developers even know that their software is being used here. And...it is actually being used! Any discussion on this? Complaints? Maybe we should contact the authors and complain about how they didn't integrate their stuff with OpenACS.

Here is an idea: do what I do. Release all of your code, talk about it, offer it up for criticism and analysis, make an ass out of yourself and make mistakes, be willing to admit you are a total idiot, be willing and very interested in being convinced that you are wrong.

Why would I waste my time writing this junk code?

I think it started pretty simply. The ATS tried to reinvent the expression syntax. There was one operator 'in' which didn't work; it was like a regexp. But certain things required using <% syntax which was part of the AOLserver templating system. So ATS was not complete. In addition, the use of the escapes into pure Tcl code is a huge security risk, worse than that, you can't accept a user template because they could be unsafe.

My templating system is safe. You can accept a user supplied template and know that it can't do serious damage. You can also rest easy that your template writers cannot exploit the system or make an unfortunate mistake. Templates are safe! Is that important to you? Maybe not. It is to me. Does it require Tcl packages or OpenACS db_* commands? NO!!!!! I'm not sitting around reinventing the wheel. But I am inventing stuff, which is available for anyone with a browser. More important, nobody is subjected to using it!

I will continue to write software which has a minimum of requirements. If I think this might be useful to the OpenACS community, I will announce it. But I'm not going to specialize my software beyond what is required.

I never got around to pointing out that my template compiler actually does syntax checking. If the template isn't valid, it won't compile. Also, the partial result is placed into a temporary file, which you can examine and pretty easily figure out the error. Important to you? Maybe not. It is important to me. ATS templates are difficult to debug.