Forum OpenACS Development: Re: Tcl Web Services Toolkit: TWiST

Posted by Malte Sussdorff on
I got the feedback from my partners that I need to encode HTML within the return values, so for them it is impossible to have something like this:

<?xml version="1.0" encoding="utf-8"?>
<soap-env:Envelope xmlns:xsi=""
<GetProjectResponse xmlns="urn:tcl:projectm">
<Title>Electric Fry</Title>
<Description>The smaller kitchen of the "Vier Jahreszeiten" restaurant
needs an electric fry<br /></Description>

See the "<br />" tag.

Using a different tool for testing I got "exception: org.xml.sax.SAXException: SimpleDeserializer encountered a child element, which is NOT expected, in something it was trying to deserialize.", while WSDLPULL complains "An Exception occurred ...@7:7 Expected a closing tag for Description Error validating schema instance".

I solved it for my application by returning [ad_quotehtml $description] instead of just $description as earlier. If this is universally true, then it would be an enhancement if TWiST would do that work for me. If it is not, then obviously you should not do it.

Posted by Tom Jackson on

This question is like asking why your application doesn't quote the description before returning it to TWiST, this is a programmer's decision when crossing the boundary of an application. TWiST takes the data you give it, assumes that you have given it the correct data, in the format you have chosen, and then handles it.

BTW, this issue provides a perfect example of why I setup TWiST as a 'configuration', not as an on/off switch for a proc like you get with .NET. This provides for maximum code reuse.

Imagine if you had to go back and define another internal API just to quote some output so TWiST could use it. The TWiST API <ws>proc is designed to allow you to call your internal API, do what you need to the normal inputs or outputs, and return the result in a format of your choice, completely independently of the exact internal API signature.

This idea should become very useful for procedures defined via ad_proc, where there are optional switches and the order of attributes isn't well defined. You can use <ws>proc to handle common defaults, and to prevent complete exposure of the internal API. For instance, you would not want to allow an application to choose the current user_id without validation. (OOB validation isn't handled by TWiST since it is OOB and application dependent.)

Also, you asked earlier if you could validate an output prior to sending. You do this inside of a <ws>proc.

There is no built in way to do this because the server should know that it is constructing a valid document. But it can be done inside the <ws>proc.

Once you have the data ready, feed the same thing you would return from <ws>proc (a list), to create a return message, then feed this reference to the validation routine. If you get an invalid result, you should return a server error (500).

Offline, you can do the same thing. Call your newly defined <ws>proc (in your new namespace) to get the return value list, create your XML message document, and then validate. This setup skips all the SOAP stuff, network interaction, etc. This is also a good place to do your unit tests.

If you have examples of what you are trying to do, it would benefit everyone if you would post the TWiST configuration, or parts of it. Real world examples are the best way to explain how things should work, and, as I said, they benefit everyone.

Posted by Malte Sussdorff on
First of all, here is what I do:

I can post the whole examples, but this is going to grow, so a link is probably better.

With regards to your answers, I understand your idea, my logic was simply: If the standard (WSDL 1.1) does not allow it, then it would be great if TWiST "catches" it, fix it (in the case of HTML within the returned value) or give a server error, without the developer having to do manual catching and checking within the <ws> proc.

With regards to the on/off switch of a procedure, I understand why you would like TWiST to be more flexible and I like it that way. Yet my projectm::GetTask is a perfect example where I as the user would prefer a simple way to have TWiST take the procedure, plug it in it's pieces and return them accordingly. But that might as well be an extension written purely for OpenACS, as we do have more information about the procedures (or could retrieve it from the (proper) documentation.

Maybe this clarifies my point that from a user perspective I would like to have a couple more things taken care for me. But I completely understand if you prefer that not to be part of TWiST as it would limit the configurability.

Posted by Malte Sussdorff on
Just got more feedback:

"for a type "float" it's not possible to have an empty value, set to 0 instead


Not sure if this is according to standard WSDL or application specific on the other's side. I can put all of that on a separate page for other developers what I had to keep in mind when dealing with WSDL in general and TWiST in particular, as I am sure I will miss a couple of more things.

Posted by Tom Jackson on
"for a type "float" it's not possible to have an empty value, set to 0 instead"

Or you can define a new type which allows the empty string. All built in numeric types in TWiST and tWSDL do not allow an empty string (like string is -strict), for obvious reasons.

Posted by Tom Jackson on
I guess there isn't a cure for your approach to these issues. Your reasoning is pretty amazing. I provide software which allows you to expose an internal API as a web service. That is what the software does. The software allows you to do this as easily as possible, and it doesn't make any assumptions about how you should do it. Now, you want it to impose assumptions upon all users so that you are happy. Next week, somebody says: "Hey I already quote my html, now it is double quoted! What's up with that?"

So to be clear: there is no way to "catch" this. Do you even remember the 'noquote' discussions which have taken place over the years right here on OpenACS? This is exactly the same issue.

Another mistake you are making is your belief that ad_proc somehow has more information about types, especially _return types_, than TWiST, especially with regard to the requirements of WSDL/SOAP. Data typing, construction and validation are way beyond what is available in OpenACS, it has exact information on every type. It can validate every type it handles. It can construct documents of a given type. But Tcl itself does not carry type information. There is no way to tell if incoming data has been quoted once, twice or never. Maybe this is available in some language, but not Tcl.

TWiST and WSDL are about 'interoperability', not 'bells and whistles', especially cracked bells and broken whistles.

It is somewhat amazing that you fixed this for yourself in exactly the correct way, by adding the typical method for handling this, and yet that isn't enough for you. You shouldn't be bothered to think at all. You wish for TWiST to use magic to guess what you want it to do.

Maybe another example: XML or XHTML. The more you restrict the functionality of a component, the more generally useful it becomes. The software becomes very simple when it doesn't have to guess at what the humans wanted to do, but couldn't figure out how to do. TWiST isn't smart, and that is simple. You are the one which has to be smart.

Here's an idea: if OpenACS truly has all the information you need to do this, then there is no issue at all, simply write your procedure to return what you want! Or, add a quote flag to your own internal API if it doesn't have this information, then use the flag inside the TWiST <ws>proc to return the correctly quoted text, or optionally a CDATA wrapped, but unquoted text.

I apologize for my tone, but you keep bring up examples of your own failings and blaming them on TWiST. If you want my tone to change, look a little deeper into the issue prior to revealing your thoughts. That's my feedback to you. TWiST is not a cure for not thinking.

I've already come up with the title of the thread you need to start about your improvements to TWiST (Why TWiST Sucks, and What I'm Going to do to Fix It). I promise not to complain in that thread about your ideas about your software. But can we stay on topic here: I'm not soliciting feedback on improvements, I have a whole list of them myself that are first in line. I'm very interested in bugs and if you give enough details I can fix them, or offer a temporary work-around.