Forum OpenACS Development: new templating model: ideas, questions

1: new templating model: ideas, questions

Posted by Tom Jackson on 02/29/04 08:20 PM

I'm working on a new templating model, and before I get too far along, I wanted to see if anyone else has advice on how it looks and works.

The basic goals are:

Safety: template writers should be unable to affect much more than the appearance of the page.
Portability: the template compiler should be a separate system which can be used by any component of a larger system.
Independence: the compile stage should not depend on the system using the compiled template.
Modularity: each template page is a complete script. The compiler can verify syntax, although variable references will be unchecked.
MVC model: A controller will be responsible for setting up external resources and providing them to the template code. The template code will not be able to access resources outside the scope of what is provided. For instance, a number of templating systems allow includes. This allows a template view to become the controller by directing the resources it uses. The controller will establish resources which have a type, a name and optional parameters.

Those are the goals, so here is what I have working so far: a compiler written in C using flex and bison. Here is an example controller:

set title "test foreach"
set list [list "one" "\[two]" "three" four "\$jane"]

set joe(name) "Joe Smith"
set joe(age) "38"
set joe(city) "Bellevue"

set jane(name) "Jane Doe"
set jane(age) "22"
set jane(city) "Boston"

set array [list joe jane]

set template "foreach.out"

source $template

puts $string

The template:

<html>
<head>
 <title>$title</title>
</head>
<body>

<p>Got a list $list
<p>Looping over list:
<ul>
[foreach l $list]
 <li>$l</li>
[/foreach]
</ul>

<p>Got an array $array
<p>Looping over Array:
<table>
<tr>
 <th>Name</th>
 <th>Age</th>
 <th>City</th>
</tr>
[foreach a $array]
<tr>
 <td>$a(name)</td>
 <td>$a(age)</td>
 <td>$a(city)</td>
</tr>
[/foreach]
</table>

</body>
</html>

Another example:

<form action=go>
<select name=test>
[foreach {value description selected} $select]
  <option value="$value" $selected>$description</option>
[/foreach]
</select>
</form>

And an if statement:

[if {$a == $b}]b[elseif {$a == $c}]c[else]d[/if]

The foreach can also handle multiple arrays or a combination of arrays and regular variables.

The questions I have are both broad and specific. First, is this a total waste of time? The compiler is working and can separate the tcl code from the html or other text. But a complete system would require a lot more work. It is easy to add new tags, but it only works on files, not buffers. It would probably be easy to add code so a multirow datasource could be used with the foreach.

More specific. The look of the tags are not xml-like. I haven't figured out how editors like Dreamweaver can identify the tags, so this is one disadvantage. One advantage, is that the tags will be ignored by Dreamweaver, etc, and you can include them inside other xml-like tags. For instance <a href="[if $a]a.html[else]b.html[/if]">go here</a> would probably validate. On the other hand, using the set command you could just set the variable based on the value of $a ahead of time, outside of the a tag. So one alternative could be command which look similar to the current ATS tags:

<tcl:if arg>...</tcl:if>
<tcl:set arg arg \>..

2: Re: new templating model: ideas, questions (response to 1)

Posted by Don Baccus on 03/01/04 12:45 AM

I think the separation of script logic and presentation, as is done by the existing acs templating system, is a *good* thing.

In particular at Greenpeace, for instance, I extended ad_return_template to be "smart" about locale settings and to optionally return a custom .adp file for a particular locale if one is provided. This means designers for various Greenpeace sites can simply modify the .adp files to change details of look and feel (though not structure to a very great degree) while all share a single Tcl script file. The latter's kept safely in the hands of the programmers.

I don't understand the motivation here. acs templating works very well and if I were to list 100 projects to work on to improve OpenACS, replacing it wouldn't be on the list. Or the list of the next 100 projects. And so on.

The ADP compiler can be extended with custom tags and you can dump Tcl code inline ("foreach") for instance. What advantages would your templating system provide over that approach which we've rejected? (though this is how AOL writes their pages).

3: Re: new templating model: ideas, questions (response to 2)

Posted by Tom Jackson on 03/01/04 06:56 AM

Separation of script and display code is important for the reasons you stated. This idea would never replace ATS because the functions in the ATS cover a very broad area. I like ATS, it works very well in most cases, and at the moment it is all that is available for OpenACS.

My motivation for working on this are stated above, where I listed the goals.

In addition, ATS only works with files, making it difficult to apply the separation concept outside of page display. It would be nice to have a merge feature where data can be setup inside a procedure and and a template could be sourced to produce a result. Any place that tcl code is used to create html or text or similar output, this idea would work well to simplify the code.

Also, ATS uses the adpparse code from AOLserver. The tags are actually registered just like regular ADP tags. Because of the design, you cannot rely on the system to protect you from malicious or mistaken programming errors by the template designer. The entire server api is available for their use. This precludes a number of possibilities that I am interested in.

4: Re: new templating model: ideas, questions (response to 1)

Posted by Barry Books on 03/01/04 12:05 PM

My only real complaint about the current templating system is the adp files are not valid XML documents. This makes moving to xhtml a bit painful.

Now that aolserver supports tDom I would look at using XSL for the kind of templating you are talking about. The advantages would be no code (or very little), the designer would have access only to the XML document not the entire TCL namespace and it's fairly well documented. Unfortunately XSL is about as Dreamweaver (and perhaps designer) unfriendly as it gets.

5: Re: new templating model: ideas, questions (response to 1)

Posted by Andrei Popov on 03/02/04 03:06 PM

As mentioned elsewhere, if one was to look at XSL (that *is* valide XML), there are loops and conditions there as well.

In this case syntax outlined in the last 2 lines is a possibility (namespace can be left to a separate discussion).

We could get things like:

<tcl:if test="condition">
...
</tcl:if>

<tcl:choose>
<tcl:when test="condition">
...
</tcl:when>
<tcl:otherwise>
...
</tcl:otherwise>
</tcl:choose>

<tcl:for-each select="list">
...
</tcl:for-each>

"condition" above could be something like "$admin_p" or "$foo='bar' and $ham='spam'", etc.

6: Re: new templating model: ideas, questions (response to 1)

Posted by Tom Jackson on 03/02/04 08:36 PM

The change from [set arg arg] to <tcl:set arg arg /> is easy, it can even be a compile time option (for the compiler not the scripts). However, representing the args as attributes might be a problem. The issue is the quoting rules for XML attributes conflicting with the quoting and other rules for Tcl args.

The model I am using is to allow the lexer find valid arg tokens and pass them to the parser to validate the grammer. Neither of these parts can verify the validity of the arguments (neither can the Tcl parser, arguments are opaque). I do examine the arguments a little. I verify they do not contain embeded commands, but combining two quoting schemes might confuse more than help the situation.

I looked into XSL, as others have pointed out, it is complex and unfriendly. It is also a full programming language, and could access resources based on what is contained in the XSLT. But the main issue is that it isn't apparent how the XML document would be created from the Tcl scripts, since the model with Tcl is to define variables and execute commands. You would still be stuck with the task of creating an XML document, or DOM from Tcl structures. My goal was to use the Tcl language as much as possible, and as safely as possible so that display code can be completely separated from other types of code. I also want to be able to use it in as many places as possible, like within a database or in procedures where templates might be stored in the database.

The webserver tclhttpd uses Safe-Tcl and a simple templating language. Files ending in .tml are run through subst. Unfortunately this eliminates all the important selection and iteration commands that make a templating language useful.

7: Re: new templating model: ideas, questions (response to 6)

Posted by Andrei Popov on 03/03/04 02:38 PM

I am not suggesting using XSL as such -- only following the approach, as this creates a valid XML file. I guess that a sample template could be something like this (using original example):

<?xml version="1.0" encoding="utf-8"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
 <title>@title@</title> <!-- keep @var@'s -->
</head>

<body>
  <p>Got a list @list@</p>
  <p>Looping over list:</p>
  <ul>
    <tcl:foreach l="@list@">
      <li>@l@</li>
    </tcl:foreach>
  </ul>

  <p>Got an array @array@</p>
  <p>Looping over Array:</p>
  <table>
    <tr>
      <th>Name</th>
      <th>Age</th>
      <th>City</th>
    </tr>
    <tcl:foreach a="@array@">
      <tr>
        <td>@a(name)@</td>
        <td>@a(age)@</td>
        <td>@a(city)@</td>
      </tr>
    </tcl:foreach>
  </table>

</body>
</html>

When <tcl:...> tag is parsed in this case, XML quoting (single or double quotes) should simply be stripped before evaluating quoted expression. Then an expression should be evaluated using Tcl evaluation rules. Using namespace prefix (tcl:) should actually help parser a lot, since it would no that only these tags need to be expanded, others can remain as they are.

I am sure that this is *very* simplistic, but it may just work... Just don't think of it as "implement templating system in XSL", but rahter "borrow XSL syntax for template definition language".

8: Re: new templating model: ideas, questions (response to 1)

Posted by Barry Books on 03/03/04 03:21 PM

If you change the syntax slightly to

<tcl:foreach name="a" select="array">
      <tr>
        <td><tcl:value-of select="a(name)"></td>
        <td><tcl:value-of select="a(age)"></td>
        <td><tcl:value-of select="a(city)"></td>
      </tr>
    </tcl:foreach>

then it works pretty much exactly like xsl. The the parser only has to find tags in the tcl namespace and process them.

10: Re: new templating model: ideas, questions (response to 8)

Posted by Andrei Popov on 03/03/04 05:41 PM

<tcl:foreach name="a" select="array">
      <tr>
        <td><tcl:value-of select="a(name)" /></td>
                                          ^^^

...to be exact

9: Re: new templating model: ideas, questions (response to 8)

Posted by Tom Jackson on 03/03/04 06:50 PM

Thanks for both of your input so far. I think there is some value in making this look XML/XSL like. There are some issues.

Attributes don't have to be in any order, in tcl order is required. Would that confuse users? The template is essentially a shorthand for a tcl script.
Quoting for tcl, then for XML will likely be very confusing in some cases.
In Tcl you have variables which look like '$a', '$a(b)', '$a($b), and then braced ${a}, etc. For instance, ATS doesn't support the '$a($b)' form, which is one reason I don't want to copy the @a.b@ syntax. Lexing would become much more involved at that point.
How do you include variable values inside other tags? In my original post I built up a select list, which inside the option tags were included the value and the selected variables. What does that look like in XSL?

I played around with Dreamweaver, and it does work with unnamed attributes, but it is confused a little by them. You can't edit them in the side panels. The main problem with attributes will be untangling the double quote, single quote confusion.

12: Re: new templating model: ideas, questions (response to 9)

Posted by Andrei Popov on 03/04/04 10:25 AM

How do you include variable values inside other tags? In my original post I built up a select list, which inside the option tags were included the value and the selected variables. What does that look like in XSL?

This would be a bit tricky, I agree. Our current

  <body<multiple name=attribute> @attribute.key@="@attribute.value@"</multiple>>

is very unXML-like :)

If you were doing actual XSLT, you'd probably do something like:

  <xsl:text>&lt;a</xsl:text>
  <xsl:attribute name="href">
    <xsl:value-of select="$foo" />
  </xsl:attribute>
  <xsl:text>&gt;</xsl:text>
    foo
  <xsl:text>&lt;/a&gt;</xsl:text>

which is ugly :)

There's gotta be an easier/nicer way...

Probably what we need is:

  <tcl:with-tag name="a">
    <tcl:attribute name="href" select="$foo" />
    <tcl:attribute name="title" select="$bar" />
    <tcl:text>$baz</tcl:text>
  </tcl:with-tag>

Which sort of says: "here's an HTML tag 'a'; please create it with attributes href='$foo', title='$bar' and put text $baz within it

13: Re: new templating model: ideas, questions (response to 12)

Posted by Andrei Popov on 03/04/04 10:13 PM

And as far as your example is concerned, maybe something along these lines:

<form action="go" method="post">
<select name="test">
<tcl:foreach bindlist="value description selected" array="$select">
<tcl:with-tag name="option"
<tcl:attribute name="value" select="$value"/>
<tcl:attribute name="selected" select="$selected"/>
<tcl:text>$description</tcl:text>
</tcl:with-tag>
</tcl:foreach>
</select>
</form>

Note that this is not real XSL, it only looks sort of like that.

14: Re: new templating model: ideas, questions (response to 13)

Posted by Tom Jackson on 03/05/04 07:09 AM

It seems quite bizarre that the XSL can have valid '$var's in their tags and html cannot. One thing I like the XMLish tag for, WRT the set proc is the possible usage of something like <tcl:set var />, which would be like [set var], instead of setting a variable, it would return the value of, in this case, 'var'.

I'm pretty sure I'm not going to rewrite HTML tags as part of a solution to this. That seems not helpful for designers or developers.

In the example given, I wonder how the selected attribute works, since it is just the text 'selected' inside the option tag?

I've verified this scanner/parser will work with UTF-8, for now except for '\0', which I am working on.

15: Re: new templating model: ideas, questions (response to 14)

Posted by Andrei Popov on 03/05/04 11:53 AM

Tom,

As I said before -- whatever you see above is not real XSL. For one thing, when transforming an XML doc one does not need to handle situations of foreach {foo bar baz} $select nature, since you always look only at a foo or at a bar or at baz (as elements) and never at all of them together — you are following through the branch trees, pretty much one at a time.

My point was not in saying "Let's just use XSL/XSLT as templating basis" but rather "Look, templating language *can* be made valid XML and that without (much) sacrifice of readability"

Whether a '$var' or a 'var' should be used in the examples (and I think I was not very consistent in that usage) — this needs to be looked at and decided. You can pretty much have whatever you want in quotes, just as well as within a CDATA in XML. It would be Templating System's parser task to actualy decide upon what needs to be done with a particular '$var'. This could be made more Tcl-like (as in [set foo "bar"] as opposed to Perl's $foo="bar") or less.

Oh, and if we wanted to really use XSL, I think we could just use tDOM to do the transformation.

16: Re: new templating model: ideas, questions (response to 15)

Posted by Tom Jackson on 03/06/04 09:41 AM

Well I've looked more into Dreamweaver and found that it is very easy to add new tag types. I think I can come up with a compromise that will allow easy usage in Dreamweaver and allow my compiler to easily identify valid code.

It turns out that in Dreamweaver, if you want to include code escapes inside attributes, you really need to use what they call an empty tag. For example, jsp/asp tags: <% %>, there is no ending tag.

So at the moment, I'm considering the following examples as equivalent:

[if $x]           [%if $x %]
[elseif {a==$b}]  [%elseif {a==$b} %]
[else]            [%else%]
[/if]             [%/if%]
[set a b/]        [%set a b /%] 
[set a/]          [%set a%] or [%= a /%] (ret value of a)
[foreach a $b]    [%foreach a $b %]
[/foreach]        [%/foreach%]
[resource text "header" ${title} /] ...

The templating language parser is helped by different ending tokens for tags like set and resource, thus the ending /] or /%] for those tags.

17: Re: new templating model: ideas, questions (response to 16)

Posted by Andrei Popov on 03/06/04 10:44 PM

Wouldn't it make code a bit unreadable? Also, why not then just using ADP as it was designed -- embeding code into your pages? You'd get the <% %> pair for free then :))

18: Re: new templating model: ideas, questions (response to 17)

Posted by Tom Jackson on 03/07/04 01:28 AM

Umm, it isn't adp! Only a few processing commands will be available, and command arguments and variables cannot contain embeded commands, and adp, requires AOLserver...First I have to satisfy the goals outlined above.

11: Re: new templating model: ideas, questions (response to 1)

Posted by Barry Books on 03/03/04 08:17 PM

I think to put xsl inside an html tag you do this

<a><xsl:attribute name="href">
<xsl:value-of select="foo" /></xsl:attribute>
foo</a>

19: Re: new templating model: ideas, questions (response to 1)

Posted by Andrew Piskorski on 04/04/04 06:23 PM

Tom, could you explain please just what you plan to use this new templating engine of yours for? That's by far the part I understand least in this thread. I mean, are you just doing this for fun and to learn from, or do you actually plan to use it on a real live site somewhere? And if so, why, precisely? Where's the big win over existing systems?

20: Re: new templating model: ideas, questions (response to 19)

Posted by Tom Jackson on 04/04/04 09:32 PM

Andrew, please refer to the "basic goals" in the initial message. I'll elaborate here:

Safe: I need a template language, tcl based, which is safe. By safe, I mean than a designer cannot create a template which does something bad to the system, like delete files, open sockets, access resources beyond what are needed. In practice, what can be done is determined by the controller script. This allows you to safely split design/display/view, from the model and controller. Designers or even users could provide templates without great risk to the system.
Tcl has a few thing which make it impossible to make safe in a simple way. One problem is embeded commands, things like $a([exec rm -rf /]) look just fine to Tcl and the exec is executed first before Tcl realizes things are not good. The 'for' command is also unsafe. You can do 'for {script} {test} {script} {...}', and whatever is in the first (and third?) argument is executed. There is a way to make a safe interp using Safe Tcl, but this is unworkable in AOLserver, and cumbersome to setup. Also, Safe Tcl isn't about templating.
This system allows a restricted set of what constitutes an argument. There cannot be unescaped '[' in an argument. Variable names must start with a letter and can only contain a limited set of chars, specifically excluding '['.
Portable: It would be nice if you could use the template system everwhere Tcl can be used: in OpenACS, AOLserver, Tcl packages, inside a database which supports Tcl, etc. This also implies that the compiler should work on buffers as well as files. I haven't made this change yet, but it should be easy.
Independent: the template compiler should not have to look outside of itself to figure out what to produce (or what is safe), templates compiled on one system will produce exactly the same result (tcl script) as on another system.
Modular: each template is a complete script. If-elseif-else and foreach statements have to be complete. Commands have to at least look like valid code. The compiler will check the grammer of the template and will produce an error if the template isn't valid. This is still similar to tcl: you can't verify the runtime validity, but you will not end up with strange errors at runtime due to bad syntax.
MVC Model: It is obvious that template systems are most useful for separating code (model/controller) from display (viewer). However, most systems allow arbitrary includes: this turns the template into the controller. In addition, if a system only works on files and cannot work in memory, you end up writing tcl code which handles the formatting/display functions. If templates can be handled as strings, and are known to be safe, you can use the MVC model everywhere, not just in the final stages of presentation. You can even use a template as a formatting tool: again safely.

21: Re: new templating model: ideas, questions (response to 19)

Posted by Tom Jackson on 04/04/04 09:36 PM

I should add that another goal is to provide a system which takes advantage of the strengths of Tcl, in an attempt to avoid creating another 'language'. This compiler isn't a language, it simply enforces certain restrictions and transforms the input template into 'possibly valid' Tcl code. It also allows infinite extensibility without changes to the compiler by the use of a 'resource' tag. It is a simple hook that allows the developer to give access to additional resources to the designer. One resource 'type' is an include. It allows the designer to include the results of an external script or compiled template and allows the designer to pass variables along. The script/template is evaluated outside the context of the caller and the results are returned as text. A similar resource (probably called source) will allow an external script to be evaluted in the caller page so you can break templates into chunks for modularity and readability, and avoid passing variables. What resource types are used will not impact the compiler, the calling environment only needs to impliment:

resource::init
resource::add type name,
resource::exists type name and
resource::eval type name args

(I'm having trouble posting this in one chunk or forums doesn't like my text, I have a further example, if it doesn't show up, it is because forums didn't like it).

22: Re: new templating model: ideas, questions (response to 19)

Posted by Tom Jackson on 04/04/04 09:42 PM

As an example of usefulness (although not beauty) take a look at this photo album. This wasn't an OpenACS site but since I had a simple photos package, written for OpenACS, I decided to adapt it to an AOLserver module. The original used ATS, and didn't use the 'grid' tag, so five images would just take up five columns and extend off the page. I didn't have a grid tag, but I did have an 'if' tag which takes standard tcl expressions, so it was easy to create a grid using a template fragment as simple as:

 [if {($photos(rownum) % $cols) == 0} ]
 </tr>
 <tr>
 [/if]

One nice feature of the ATS grid tag is that you can flip the orientation. In my system you would need to create a resource to do this, but it turns out to be very efficient. I created a proc which takes a list and creates another list, with a modified order, depending on the number of columns you want in the output. Since the foreach tag takes a list of arrays, the new list just accesses the arrays in a modified order, allowing you to create the flipped orientation.

I also found that the group tag is easy to replace. However, it is slightly ugly:

[set v ""]
[foreach row $rlist]
 <tr>
[if {"$v" ne $row(a)}]
  <th colspan="6" align="left">a = $row(a)</th>
 </tr>
 <tr>
  <td>&nbsp;</td>
[else]
  <td> </td>
[/if]
[set v $row(a)]
[/foreach]

You would have one if block for each group by.

23: Re: new templating model: ideas, questions (response to 1)

Posted by Dave Manginelli on 05/30/05 12:06 AM

Tom: I have a need for just such a “templating system for untrusted users” as you seem to have described in this year-old thread.

Has the system worked well in production and, if so, are you releasing the code?

Thanks.

24: new templating model: ideas, questions (response to 1)

Posted by kousalya S on 11/19/10 01:42 PM

I am able to add no of passengers data with grid for one ticket nr and show it in index page .I am unable to get the edit page of the grid view with all the passengers data in the grid .

Clicking on a link - should route to the edit page of the grid view with all the values.

Trying to have the edit page of the grid view.