Forum OpenACS Development: Re: Tcl Web Services Toolkit: TWiST

Collapse
Posted by Malte Sussdorff on
Some detailed questions:

1) Are the following two equivalent (one found in /stockQuoter/tcl/stockQuoter-procs.tcl):
::wsdl::types::simpleType::new $sqns name xsd::string

and

<ws>type simple stock::name (found in /wsapi/www/stockQuoter/index.tcl)

2) If I use the <ws> syntax, do I still need to have the definition of the type in the -procs.tcl ?

3) Is my assumption correct that each <ws>proc definition will be turned in an Operation which you can choose from, adaptly named ${proc_name}Operation ?

4) Which type types do we have. What is ENUM supposed to be, and do you need the "<ws>type pattern stock::Code {[0-9]{4}} xsd::integer" in the example provided? (and if yes, will you be able to define this way that ::Code is a variable of type integer that consists of 4 characters? Is pattern always a regexp (as I could deduct from looking at datetime/index.tcl).

5) It seems that the datetime/index.tcl does not seem to work, at least no option 1 and 3. When I use 3 (get day name from number), I always get an incorrect input value (which I realize is due to the fact that you where using enumeration with xsd:integer, which does not seem to work. I managed to get it to work with "wsdl" pull by just using enum with the optional values {0 1 2...6}, though that lacks the type definition.

6) How can I reload a service definition. At the moment, whenever I change e.g. /wsapi/www/datetime/index.tcl I have to restart the server. Is this due to the fact that I have a symbolic link from an OpenACS packages directory or is this for real. if it is for real, how can it be prevented ?

7) Would it be possible to wrap the service description at http://e4.cognovis.biz/twsdl/wsapi/datetime/?WSDL into some <code> tags, as Safari is happy in displaying ... nothing (I found out that it actually works by opening the page with FireFox).

8) Is there  a "cleaner" documentation than the pure service description, where we could story a little bit more information what the variables are supposed to do. Somewhat like ad_proc, if you like. There does exists a <wsdl:documentation ...> tag, maybe use that one ?

The more I look at the examples (and fix some issues e.g. with datetime) the more I like it. Sadly I fear the ease of <ws> procedure will be lost on me as the other side expects a certain naming convention, but I will see.

Collapse
Posted by Tom Jackson on
Malte,

Maybe your interest is in fixing stuff instead learning of how TWiST works? There are no known bugs, at least with the things you are talking about. Please stop making requests for code changes until you actually get this installed. TWiST is a simplified API on top of tWSDL. tWSDL is an attempt to satisfy the requirements of the WS-I Basic Profile for web services, it is not an attempt to satisfy you or the OpenACS community. Therefor the way things work, has a lot to do with the WS-I Basic Profile. Criticisms directed at the functional requirements are appreciated, those directed to a matter of preference are not.

Personally I think it is cool that an unrelated tool (wsdlpull, etc.) can grab a wsdl file generated by TWiST and construct C++ code to create a SOAP request and parse the result document. It is cool that for simple types, and enumerations it can validate inputs prior to sending. But this is not software that does everything, or should do everything. Type definition is a great example. Types are constructed according to the way they are derived in XML-Schema. The reason is obvious: in order to build a validation script inside a client, or server, you have to know how the types are derived from the basic types. Patterns are regular expressions, because that is what they are in XML-Schema. If you are uninterested in type definition, type validation, etc, then don't use WSDL, that is the whole point.

But it is not cool to have an application which satisfies some short term interest at the expense of interoperability. Anyone wishing to hack on tWSDL or TWiST should read all of the documentation first, and the references, especially those listed in:

http://twsdl.googlecode.com/svn/trunk/packages/doc/www/current-standards.html

Otherwise very many misunderstandings will result.

Collapse
Posted by Malte Sussdorff on
Thanks for pointing out that the code works on your website. I got this and as mentioned before, it works on mine as well :-). Sadly, the difference is this:

noname:~ malte$ wsdl http://junom.com/ws/datetime/?WSDL
1.AddDurationToDateTimeOperation
2.CheckDateTimeOperation
3.DayNameFromNumberOperation
Choose one of the above operations [1-3] :3
DayNumber: 2
Incorrect input value 2
noname:~ malte$

On my system, after appying a change to /datetime/index.tcl as mentioned above it looks like this:

noname:~ malte$ wsdl http://e4.cognovis.biz/twsdl/wsapi/datetime/?WSDL
1.AddDurationToDateTimeOperation
2.CheckDateTimeOperation
3.DayNameFromNumberOperation
Choose one of the above operations [1-3] :3
DayNumber: 2
DayName:Wednesday

noname:~ malte$

Now, if you say that "wsdl" from "wsdlpull" is not a program to use for your code, then this is fine, but it was mentioned by you, so I do assume that I can use it for testing purposes.

Collapse
Posted by Malte Sussdorff on
I am not sure how you manage to come to the conclusions what I am interested in or what I suspect TWiST to do. Obviously it is not written to satisfy my needs, how could it, after all I have not voiced them before actually using the software.

What I am interested though is understanding how I can export pm::task::new as a webservice, so I could use e.g. wsdl to create a new task. That is my ultimate goal, and this is why I am asking all those questions which I hope you will answer eventually, unless you say it is all obvious from the documentation, then I will admit defeat and humbly ask you where it is answered in the documentation (as I read through a lot and might have missed it).

Last but not least, I have not intention to hack on tWSDL or TWiST. Unless you define exchanging "proc" with "ad_proc" for your proc definitions as hacking. And you made your wishes *very* clear and I respect that. Still I feel the installation and documentation could be improved, what you make of that is entirely up to you, this is *my* personal opinion and not meant as an insult.

Collapse
Posted by Malte Sussdorff on
More things I noticed while doing the < ws > approach:

1) The Service Description does not get updated. I guess this is because of "

# eventually must add check to delete prior namespace and wsdb code
# also might put in variable to freeze additional changes (better word?)
# this 'freeze' would operate in <ws>proc as well to silently abort and log event" in <ws>namespace init ?

Either way, rediting, I found out that I need to unfreeze my namespace :-). So I assume that, when I want to change your demo code, I need to make sure to either not freeze the namespace or at least unfreeze it before working on it.

2) I cannot use xsd:: types directly but have to make a type declaration first. Though I am not sure why in your example for stockQuoter you have been generating identical types (::quote, ::dailyMove, ::lastMove). What am I missing here? Wouldn't it be enough to just define stock::float and then use that for the sequence definition? I mean, it worked when I tested it, but maybe there is a deeper meaning behind it which I am unaware of yet.

3) I found the documentation (more or less) for the < ws> functions. Not linked from anywhere, but it seems autogenerated using the man command. I linked it up at http://e4.cognovis.biz/twsdl/doc/api-index.html.

4) I might be wrong, but does < ws>proc check if the data type returned matches the value? I am not sure if it should, but I would probably prefer a server error on my machine than on the querying machine in case the returned type does not match the definition of the to be returned type (as an example, take the dateTime example. If you set the dateTime manually to "TWiST" then it is still returned and I could not find any error /Notice that what I was returning was not of the datetype specified.

5) How can I give a default value that does not have to be filled in. Or, lets be precise, how can I test it? I tried with the stockQuoter example of yours not to pass in the "verbose" value and failed, both with wsdl as well as the web interface. At the moment I have to fill it in, regardless what it says in the definition and this makes me wonder where the trick is.

Collapse
Posted by Malte Sussdorff on
Sorry for posting again, but I realized I asked questions which might be misleading and I wanted to give quick answers to them. Also note that you need to click on the link to the "in response to", otherwise these answers do not make any sense.

After working with the toolkit, here are the answers to my questions.

1) Yes
2) No
3) Yes
4) Given by Tom
5) This was a problem due to my misunderstanding of duration and dateTime. Once I looked both datatypes up in the WSDL specifications and provided correct values it works. Read the specification on Duration though, it has a special format
6) Make sure not to freeze your webservice. This is the default in all examples given, probably something that should be noted.
7) Okay, this question was incorrectly asked. I wanted to have a second place where I could get the generated service definition wrapped in <code> tags so I could read the output (full) for debugging purposes. It obviously does not make any sense to wrap the ?WSDL in those tags. For the time being, just open the link in your browser and click "Show Source" to see the outcome.
8) No, and Tom has no intention to provide it. Just in case you wonder, I am talking about extend <ws>proc by a documentation block which will result in <wsdl:documentation ..> tag being inserted.

Collapse
Posted by Tom Jackson on
Okay, I'll lighten up, at least a little bit. First I want to say that Malte did bring up some valid points for future development, but I thought it more important to respond to certain misunderstandings, and what I thought was an attempt to provide an altered version of the TWiST API to the OpenACS community.

This last issue I take very seriously, and I don't think it is appropriate or helpful for the TWiST/tWSDL project to be forked _in any way_. But let me explain how I had planned for the re-use of tWSDL so that anyone who wishes to provide a different layer on top of tWSDL is can do so.

There is a package in tWSDL called wsapi. The wsapi-procs.tcl file has this comment:

# Linked files may not carry the same copyright as this file.
# Files linked here should present higher level APIs which:
# 1. must use the twsdl api
# 2. act as a layer above the twsdl api
# 3. provide useful defaults ...

No this isn't in any external documentation, but then again, nobody will be able to provide a higher layer without becoming very familiar with all of tWSDL.

So TWiST is _but one_ higher level API above tWSDL which uses the tWSDL API, and provides useful defaults.

If any person or organization wishes to extend tWSDL, they are free to do it, and will hopefully contribute and maintain their code so that it is available either on Google Code (in twsdl or otherwise), or somewhere else.

But please, please, do not modify the TWiST API, or extend it and offer an extended version calling it TWiST, etc.

In a number of places I also clearly identify TWiST as an _example_ use of tWSDL. Anyone wishing to develop an separate layer on top of tWSDL should follow this example. The tWSDL API must be invoked in a more or less fixed sequence, or at least a number of things must be done prior to some important step.

Another important point about TWiST, and why it must remain free from enhancements is that I use it to test and debug tWSDL. For instance, TWiST allowed me to quickly rewrite some of the tWSDL code so that it was immune from certain user errors or practices. One of these is the ability to reinvoke the same commands without messing up the web service.

Another: although it would be nice to imagine that all code will remain bug free into eternity, this doesn't seem to ever occur. Page level debugging is very difficult to perform, and usually requires insertion and eventual removal of ns_log statements. Instead of this, tWSDL and TWiST can be debugged using a Tcl shell. The AOLserver layer is so thin that it only takes a few set statements to setup the TWiST service. If an error occurs, you can immediately investigate the current state of the process (because there is only one state and you are right in the middle of it).
Also note that if available, the tWSDL will load libnsd.so so that normal ns_log statements are printed, if not, the text of the log statements are printed, minus the timestamp.

Documentation!

Malte is of course correct that the documentation could be improved, or easier to find. But there are different forms of documentation aimed at different kinds of users. The TWiST API was documented as an AOLserver module, which uses nroff files which are compiled into man and html files. I'm not sure why Malte thinks that tWSDL/TWiST is not a proper AOLserver module, but it seems to be. I don't think it is correct to provide tWSDL/TWiST as a Tcl package, since in general it requires the AOLserver API for normal usage. So this is why the primary documentation for TWiST is in provided in this form. I'm not particularly fond of nroff, but that is the current state-of-the-art with AOLserver.

Also, as Malte points out the <ws>namespace freeze command is poorly documented. He correctly discovered that you should only use it when you complete your development of the TWiST configuration file. In addition, if you don't use the freeze subcommand, you need to either wait for your connection thread to exit, or restart your AOLserver _if you are changing an existing simpleType or complexType_, either directly or via a changed <ws>proc call.

WSDL documentation. The tWSDL database (WSDB) contains internal representations of the WSDL XML file for each web service. These are child namespaces of ::wsdb::definitions. You can browse this using the inspect package. Using the ::xml::* commands, you can add to this internal representation, and it would be pretty easy to write a higher level API to insert documentation. The question is: what would the tWSDL API look like, and what about the TWiST API?

Personally, I like the documentation method used in SQL: separate commands which identify the thing to be documented, followed by the documentation. I do not like the concept of adding an additional block to the TWiST API. Here's why: you should be able to document types. Types can be reused, in fact, they can't be repeated in the WSDL file. Therefore, if you document the types, documentation shows up in one place. Operations are a higher level concept. Documentation for Operations identify why you would invoke. This is kind of like an advertisement. Until you are convinced to invoke, the exact input and output signatures are unimportant. If the documentation mixes type info with 'why to use' info, it will not be as easy to read.

Additionally, it might be useful to separate the documentation API from the decision to include the documentation in the WSDL, or elsewhere. Providing a separate switch-like API for use in the TWiST configuration file would provide a place for documentation even if this is not published.

Internal API documentation: as Malte has pointed out, it appears that I think I'm the only one working on the internal API. At the moment this is true. But there is external documentation of what the expected API for tWSDL would be. Guess what? They don't match up. The best way to explain this, or make an excuse for it, if you wish, is that there is a vast difference between what you think is going to work, and what actually ends up being a better way. I do not subscribe to the view that it is best to document things first and then stick with this regardless of new discoveries. This doesn't mean that documentation should not follow up when the code matures, but if there is no code to document, or if the code needs cleanup and reorganization, which should come first? Code which does not work does not need documentation, it need work. Code which works but is difficult to understand or of limited re-usefulness, does not need documentation, it needs work. At some point, code becomes clean and useful, which means that others will use it. This is when documentation should be produced. So at the moment, the TWiST API should be the focus of documentation, and it does have some. I will update this with the next release, which will add significant functionality to tWSDL and TWiST.

Another idea comes to mind: is there no way to provide documentation for external software and integrate it with OpenACS? That seems like a good idea, especially for those packages which provide Tcl API.

Collapse
Posted by Malte Sussdorff on
Another idea comes to mind: is there no way to provide documentation for external software and integrate it with OpenACS? That seems like a good idea, especially for those packages which provide Tcl API.

The way to so this is top provide a wrapper package in OpenACS which provides e.g. twsdl::ws::proc, being a wrapper for calls to < ws >proc. This would be defined using ad_proc, allowing for all the documentation needed and wanted.

with regards to the wsdb documentation I would love to see both documentation for types and operations. But to my understanding the standard WSDL 1.1 only supports the documentation tag in the operation, not in the type defininition. If I am wrong, then please find a way to describe the types, this does make much more sense (as you noted).

How could the WSDB documentation be represented: In general I would have the WSDB be enhanced by documentation fields, so the inspect functions could display the documentation as well. Here having additional commands for the documentation might be a good thing.
When displaying the webservice itself (using ?WSDL or your internal display method) the generated < documentation > tags can then be viewed when you call your page (w.g. at e4.cognovis.biz/twsdl/projectmanager), which list all the operations, along with the corresponding documentation. Once you click on one operation, you will see all the entry fields along with their documentation. Probably you would have to display what is expected to come back with documentation as well. Not sure how this works.

If using < ws >, I would really not have a separate procedure for the documentation, both for type as well as for proc (and namespace, if that makes sense). Let the < ws > proc handle the dual calls to insert the webservice along with the documenation into the WSDB. Not so sure that having a different command to desribe the documentation afterwards is the way to go, but probably I am just too much an OpenACS guy.

Collapse
Posted by Tom Jackson on
The way to so this is top provide a wrapper package in OpenACS which provides e.g. twsdl::ws::proc, being a wrapper for calls to <ws>proc. This would be defined using ad_proc, allowing for all the documentation needed and wanted.

I'm somewhat confused by this. Are the wrapper procs merely to provide documentation for the <ws>* APIs? Looking at the internals of the <ws>* APIs, indicates that the wrapper have to have a similar internal structure, because there are subcommands. Therefore the evolution of the wrapper would have to concern itself with the internals of the wrapped APIs. This same handholding would be required for every other external package. Additionally, this duplicates the documentation effort, even if it is copy/paste, the wrapper API provides nothing new in functionality. But it does provide another layer which must be maintained and debugged and traced through in case of errors.

If using <ws>, I would really not have a separate procedure for the documentation, both for type as well as for proc (and namespace, if that makes sense). Let the <ws>proc handle the dual calls to insert the webservice along with the documenation into the WSDB. Not so sure that having a different command to desribe the documentation afterwards is the way to go, but probably I am just too much an OpenACS guy.

I'm trying to maintain some civility here, but this is somewhat interesting way of looking at things. First a minor issue. The API is written <ws>*, there are no spaces. It is like constantly mispronouncing someone's name, on purpose due to laziness. Second, I'm not interested in how you think it should be done or where the information should be stored (I am interested in _what_ should be documented). Anything like this must first have a corresponding API in tWSDL. If you read my next-to-last post, I included a comment I put in a file when I first wrote TWiST: higher level APIs must use the tWSDL API. The meaning of this is that higher level APIs cannot mess with the internal storage of the WSDB. This forced use of the API helps identify deficiencies in tWSDL as well as hacks which need code cleanup. I have done several rounds of this already in tWSDL and TWiST.

Having a separate documentation API keeps the two functions separate. I can't imagine how it would be possible to not have massive duplication of code if every <ws>* API had to have its own documentation handler. I have already outlined the deficiencies with the ad_proc method. First is that I would have to duplicate what ad_proc does. But which ad_proc do I follow? There are several versions, and I doubt that we have seen the last version. Changes in version require redoing previous documentation blocks, or maintaining backwards compatibility. If similar to, but not quite the same as ad_proc, a documentation block will lead to inevitable confusion for those who use ad_proc and expect it to be identical. For those unfamiliar with ad_proc, they must learn how the documentation block works. Finally, <ws>proc does more than define a procedure. It defines simpleTypes, complexTypes, input and output messages and an operation.

But most important, and this is really important: I cannot think of a single use of documentation blocks in API calls. Does anyone call db_multirow with a documentation block? I guess you could call ad_page_contract an example. It is a great example of one procedure doing too many things. I'm sure the code for it is difficult to maintain. But just a clue to my method of API development: I write monolithic procs which contain too much code at first. I get things to work, and then I chop up the internals and produce single function APIs to support the original one. I don't keep adding functions to existing API unless it is 'functionally necessary'. Of course I don't always succeed, but it is a valid criticism for code which needs cleanup. Okay one more thing: documentation is data. I don't know if you can include documentation in the schema of a WSDL, but since the namespace of the schema is the xml schema, the rules are the same as that namespace. But even if the WSDL standard doesn't allow the inclusion of documentation in a particular place, this has no importance with respect to the decision to document, at least internally, particular components. The documentation inside the operation might contain a url to an html document, compiled from more complete internal documentation.

My original comment was that it would be nice if there was an independent OpenACS package which allowed you to access the documentation of an external package as if the APIs were part of OpenACS. For instance: Tcl, AOLserver, tDOM. Does anyone wrap these procs? If you could create a package for AOLserver, then the documentation for TWiST would also show up in the same place. Of course my inspect package shows the current state of the code and variables in that thread, but this is not documentation, it is for debugging and analysis.

Collapse
Posted by Malte Sussdorff on
Tom, provide a page to which OpenACS can give a procedure name of your toolkit and it redirects us to the appropriate procedure documentation, then this can be easily implemented in the OpenACS api-doc if there exists an OpenACS package for your work. It would show up on the right hand sight of the API Doc below the search for TCL and AOLserver procedures.

In an ideal world I would be able to register all procedures with their names within the API Browser, so that searches for "string" would turn up the commands for the TCL string function, but this is not how the API Browser works *yet*.

Writing a package for OpenACS only makes sense once the API has stabilized though, which holds true for any wrapper functionalities.

I have to correct myself, if someone is interested in a tighter integration between tWSDL / TWiST and OpenACS then  write procedures similar to the <ws>* procs in OpenACS "style" which make use of functionalities that only make sense if you have installed OpenACS. We have ample of examples where OpenACS wraps existing external API into it's own functionality blocs, like ad_return* or ad_proc .... So do not write a simple wrapper which does not add any new functionality except documentation,  but I guess from my posting it was not obvious that this advantage comes in hand as well :-).

Collapse
Posted by Tom Jackson on
My suggestion has been that a 'value added' OpenACS package should start with tWSDL. With the interactive features of a web application, you could provide even more assistance to the developer than what is provided with TWiST. For instance, you could guide the developer through the process of proc selection, type development, testing, documentation, etc. These are the logical processes required to create and document the web service, but the developer doesn't necessarily need to know the details of the API, if they have a wizard type package. BTW, both tWSDL and TWiST are already wizard like since the main function calls simply fill a database and write code, setting up the server, TWiST handles the boring details needed to please tWSDL.

However, if I were to write a quick 'integration' package for OpenACS, it would import the correct version (for the OpenACS package) from the google code repository. After that import, it can then be packaged up into an .apm file for distribution, or maybe the package would simply do the import upon installation. The package would then provide the TWiST API. The next step would be a wizard type interface which would produce a TWiST configuration file.