Forum OpenACS Development: TWiST version 0.9.9

Collapse
Posted by Tom Jackson on
I just tagged a new version of TWiST, twist-0.9.9, available as a tar.gz file at:

http://code.google.com/p/twsdl/downloads/list

Also available via svn:

svn checkout http://twsdl.googlecode.com/svn/tags/twist-0.9.9 twist-0.9.9

What is new in this version of TWiST is a greatly enhanced ability to derive new simpleTypes via restriction. The two main branches of derived types in XML-Schema are 'string' and 'decimal'. Each has a set of facets which can be applied to a base type which restricts the 'value space' of the base type.

For example string has these main facets:

* length
* minLength
* maxLength
* pattern
* enumeration
* whitespace

The enumeration facet cannot be used with any other facet in a single restriction, so the original method <ws>type enumeration is used when necessary. In TWiST, whitespace is always preserved, a requirement of XML 1.0, therefore this facet is ignored.

The facets for decimal restriction are:
* totalDigits
* fractionDigits
* pattern
* whiteSpace
* enumeration
* maxInclusive
* maxExclusive
* minInclusive
* minExclusive

Enumeration is handled separately as with string restriction. Whitespace is always collapse (string trim), so it is ignored. The complexity of decimal restriction will be addressed in a separate post. Note that to reduce confusion, there are many checks for the validity of the inputs, with appropriate feedback, prior to creating the type. This will help developers define valid restrictions. At runtime, the validation procedure can return additional error codes to identify exactly why the input value failed. The two major complications with writing this type of code are the need to create a canonical value from the input lexical representation, and the fact that Tcl screws up with certain values. Example: 0x23 and 023 are considered integers, but one is hex and the second octal. 029 is considered an error in Tcl (invalid octal). And obviously Tcl doesn't mind comparing abc > 123.

The test server for these new types is

http://junom.com/ws/mywebservice/

Collapse
2: Re: TWiST version 0.9.9 (response to 1)
Posted by Tom Jackson on
I have also created a set of test vectors for decimal. Each value is tested to see if it passes the base decimal test and also if it passes the Tcl [string is integer -strict] test. Tcl does not have a decimal type. For every value a formatted line shows the different parts of the decimal value, the canonical value and the total digits and fractional digits found in the value. There are a few entries with leading and trailing whitespace which dont pass the decimal test, but in TWiST, all values derived from decimal will be trimmed prior to testing.

The test vector chart is at:

http://junom.com/ws/decimal.tcl

Although the test vectors give a general idea of the expected behavior, there are a few important distinctions to keep in mind:

* If a pattern is given, the input value is tested against the pattern directly without collapsing whiteSpace or removing a leading +, or leading or trailing zeros, or any canonization. However, the value still has to pass the base decimal test which does all of these prior to testing.

* The chart assumes that the canonical decimal point representation is "." if the input value fails the Tcl integer test, otherwise if it looks like a Tcl integer, the decimal point representation is the empty string.

* The code behind this chart tests several useful helper procedures. One is called ifDecimalCanonize. It returns either true or false. It also upvars a list which can be joined [join $canonList ""] to produce the canonical value of the input value. Optionally the procedure upvars a list of totalDigits and fractionDigits found in the canonical value. The second procedure validateWithInfoArray also returns either true or false. If the input contains at least one digit, the procedure upvars an array with component parts of the decimal. This procedure requires a collapsed (string trim) value.

Aside from the necessity of this code for TWiST, these procedures and validation code can be used in Tcl or OpenACS to test values prior to passing them to a database or an external (or internal) procedure which requires more rigorous limits on acceptable values. Currently Tcl does not provide any built in validation for the wide variety of lexical forms which are used to represent decimal types. In addition there is no way to override the default behavior for operators which compare values. In most cases this is okay, but the lack of validation in Tcl means that developers have to work harder to return meaningful error messages when a database query doesn't work out. Although it is never possible to guarantee that any amount of data type validation will avoid errors in database queries or other external applications, most errors can be caught in Tcl, where they are much more easily handled.

Collapse
3: Re: TWiST version 0.9.9 (response to 2)
Posted by Malte Sussdorff on
Hi Tom, how about getting your procedures and validation code into TCLLIB? This is (in my opinion) a very good place for them, especially as TCLLIB is installed with every new OpenACS release whereas people might not want to rely on TWiST if they do not need it. I don't think it makes sense to put these into OpenACS as TWiST relies on them but not on OpenACS.
Collapse
4: Re: TWiST version 0.9.9 (response to 3)
Posted by Tom Jackson on
It's a nice idea, but tcllib is harder to install than TWiST and contains a lot of code that probably doesn't need to be in tcllib, it would not be easy to remove just the typing code either.

The model I used was as an AOLserver Tcl module. Installation only requires that you place tWSDL as a child directory of the private Tcl library for the virtual server, and then add a line to the config file telling AOLserver about the module.

The API was written so that it depended very little (one proc) on AOLserver, so you could replace AOLserver. However, I haven't found an easy replacement for AOLserver. If I do find such a replacement, there is a tiny abstraction package I need to write for a few ns_* commands, similar to the ones already there for ns_log. Also, TWiST can run from a tclsh, I use nstclsh which is part of AOLserver. Obviously you can't do an <ws>return, but you can create types, and set a few variables and then generate a WSDL, along with the commands to create and validate the SOAP messages.

Hmm, that suggests that what I really need here is a <ws>server command to go along with an abstracted <ws>return command, then the <ws>server command would be like calling nsd.

But my idea for OpenACS is the TWiST OpenACS package available either at google code:

http://code.google.com/p/twsdl/downloads/list

or on my test server:

http://junom.com/openacs/

The OpenACS TWiST package is only 3.2K and makes TWiST even easier to install on OpenACS than it is to install into AOLserver (without editing any files on disk). But it does depend on either 1) subversion, or 2) also downloading the latest TWiST tagged and tar.gz'd file at googlecode downloads.

And best of all, if you don't need it, it isn't there taking up any space.