Forum OpenACS Development: Re: Tcl Web Services Toolkit: TWiST

1) Are the following two equivalent (one found in /stockQuoter/tcl/stockQuoter-procs.tcl):
::wsdl::types::simpleType::new $sqns name xsd::string

and

<ws>type simple stock::name (found in /wsapi/www/stockQuoter/index.tcl)

2) If I use the <ws> syntax, do I still need to have the definition of the type in the -procs.tcl ?

3) Is my assumption correct that each <ws>proc definition will be turned in an Operation which you can choose from, adaptly named ${proc_name}Operation ?

4) Which type types do we have. What is ENUM supposed to be, and do you need the "<ws>type pattern stock::Code {[0-9]{4}} xsd::integer" in the example provided? (and if yes, will you be able to define this way that ::Code is a variable of type integer that consists of 4 characters? Is pattern always a regexp (as I could deduct from looking at datetime/index.tcl).

5) It seems that the datetime/index.tcl does not seem to work, at least no option 1 and 3. When I use 3 (get day name from number), I always get an incorrect input value (which I realize is due to the fact that you where using enumeration with xsd:integer, which does not seem to work. I managed to get it to work with "wsdl" pull by just using enum with the optional values {0 1 2...6}, though that lacks the type definition.

6) How can I reload a service definition. At the moment, whenever I change e.g. /wsapi/www/datetime/index.tcl I have to restart the server. Is this due to the fact that I have a symbolic link from an OpenACS packages directory or is this for real. if it is for real, how can it be prevented ?

7) Would it be possible to wrap the service description at http://e4.cognovis.biz/twsdl/wsapi/datetime/?WSDL into some <code> tags, as Safari is happy in displaying ... nothing (I found out that it actually works by opening the page with FireFox).

8) Is there  a "cleaner" documentation than the pure service description, where we could story a little bit more information what the variables are supposed to do. Somewhat like ad_proc, if you like. There does exists a <wsdl:documentation ...> tag, maybe use that one ?

The more I look at the examples (and fix some issues e.g. with datetime) the more I like it. Sadly I fear the ease of <ws> procedure will be lost on me as the other side expects a certain naming convention, but I will see.

noname:~ malte$ wsdl http://junom.com/ws/datetime/?WSDL
1.AddDurationToDateTimeOperation
2.CheckDateTimeOperation
3.DayNameFromNumberOperation
Choose one of the above operations [1-3] :3
DayNumber: 2
Incorrect input value 2
noname:~ malte$

On my system, after appying a change to /datetime/index.tcl as mentioned above it looks like this:

noname:~ malte$ wsdl http://e4.cognovis.biz/twsdl/wsapi/datetime/?WSDL
1.AddDurationToDateTimeOperation
2.CheckDateTimeOperation
3.DayNameFromNumberOperation
Choose one of the above operations [1-3] :3
DayNumber: 2
DayName:Wednesday

noname:~ malte$

Now, if you say that "wsdl" from "wsdlpull" is not a program to use for your code, then this is fine, but it was mentioned by you, so I do assume that I can use it for testing purposes.

1) The Service Description does not get updated. I guess this is because of "

# eventually must add check to delete prior namespace and wsdb code
# also might put in variable to freeze additional changes (better word?)
# this 'freeze' would operate in <ws>proc as well to silently abort and log event" in <ws>namespace init ?

Either way, rediting, I found out that I need to unfreeze my namespace 😊. So I assume that, when I want to change your demo code, I need to make sure to either not freeze the namespace or at least unfreeze it before working on it.

2) I cannot use xsd:: types directly but have to make a type declaration first. Though I am not sure why in your example for stockQuoter you have been generating identical types (::quote, ::dailyMove, ::lastMove). What am I missing here? Wouldn't it be enough to just define stock::float and then use that for the sequence definition? I mean, it worked when I tested it, but maybe there is a deeper meaning behind it which I am unaware of yet.

3) I found the documentation (more or less) for the < ws> functions. Not linked from anywhere, but it seems autogenerated using the man command. I linked it up at http://e4.cognovis.biz/twsdl/doc/api-index.html.

4) I might be wrong, but does < ws>proc check if the data type returned matches the value? I am not sure if it should, but I would probably prefer a server error on my machine than on the querying machine in case the returned type does not match the definition of the to be returned type (as an example, take the dateTime example. If you set the dateTime manually to "TWiST" then it is still returned and I could not find any error /Notice that what I was returning was not of the datetype specified.

5) How can I give a default value that does not have to be filled in. Or, lets be precise, how can I test it? I tried with the stockQuoter example of yours not to pass in the "verbose" value and failed, both with wsdl as well as the web interface. At the moment I have to fill it in, regardless what it says in the definition and this makes me wonder where the trick is.

This last issue I take very seriously, and I don't think it is appropriate or helpful for the TWiST/tWSDL project to be forked _in any way_. But let me explain how I had planned for the re-use of tWSDL so that anyone who wishes to provide a different layer on top of tWSDL is can do so.

There is a package in tWSDL called wsapi. The wsapi-procs.tcl file has this comment:

# Linked files may not carry the same copyright as this file.
# Files linked here should present higher level APIs which:
# 1. must use the twsdl api
# 2. act as a layer above the twsdl api
# 3. provide useful defaults ...

No this isn't in any external documentation, but then again, nobody will be able to provide a higher layer without becoming very familiar with all of tWSDL.

So TWiST is _but one_ higher level API above tWSDL which uses the tWSDL API, and provides useful defaults.

If any person or organization wishes to extend tWSDL, they are free to do it, and will hopefully contribute and maintain their code so that it is available either on Google Code (in twsdl or otherwise), or somewhere else.

But please, please, do not modify the TWiST API, or extend it and offer an extended version calling it TWiST, etc.

In a number of places I also clearly identify TWiST as an _example_ use of tWSDL. Anyone wishing to develop an separate layer on top of tWSDL should follow this example. The tWSDL API must be invoked in a more or less fixed sequence, or at least a number of things must be done prior to some important step.

Another important point about TWiST, and why it must remain free from enhancements is that I use it to test and debug tWSDL. For instance, TWiST allowed me to quickly rewrite some of the tWSDL code so that it was immune from certain user errors or practices. One of these is the ability to reinvoke the same commands without messing up the web service.

Another: although it would be nice to imagine that all code will remain bug free into eternity, this doesn't seem to ever occur. Page level debugging is very difficult to perform, and usually requires insertion and eventual removal of ns_log statements. Instead of this, tWSDL and TWiST can be debugged using a Tcl shell. The AOLserver layer is so thin that it only takes a few set statements to setup the TWiST service. If an error occurs, you can immediately investigate the current state of the process (because there is only one state and you are right in the middle of it).
Also note that if available, the tWSDL will load libnsd.so so that normal ns_log statements are printed, if not, the text of the log statements are printed, minus the timestamp.

Documentation!

Malte is of course correct that the documentation could be improved, or easier to find. But there are different forms of documentation aimed at different kinds of users. The TWiST API was documented as an AOLserver module, which uses nroff files which are compiled into man and html files. I'm not sure why Malte thinks that tWSDL/TWiST is not a proper AOLserver module, but it seems to be. I don't think it is correct to provide tWSDL/TWiST as a Tcl package, since in general it requires the AOLserver API for normal usage. So this is why the primary documentation for TWiST is in provided in this form. I'm not particularly fond of nroff, but that is the current state-of-the-art with AOLserver.

Also, as Malte points out the &lt;ws&gt;namespace freeze command is poorly documented. He correctly discovered that you should only use it when you complete your development of the TWiST configuration file. In addition, if you don't use the freeze subcommand, you need to either wait for your connection thread to exit, or restart your AOLserver _if you are changing an existing simpleType or complexType_, either directly or via a changed &lt;ws&gt;proc call.

WSDL documentation. The tWSDL database (WSDB) contains internal representations of the WSDL XML file for each web service. These are child namespaces of ::wsdb::definitions. You can browse this using the inspect package. Using the ::xml::* commands, you can add to this internal representation, and it would be pretty easy to write a higher level API to insert documentation. The question is: what would the tWSDL API look like, and what about the TWiST API?

Personally, I like the documentation method used in SQL: separate commands which identify the thing to be documented, followed by the documentation. I do not like the concept of adding an additional block to the TWiST API. Here's why: you should be able to document types. Types can be reused, in fact, they can't be repeated in the WSDL file. Therefore, if you document the types, documentation shows up in one place. Operations are a higher level concept. Documentation for Operations identify why you would invoke. This is kind of like an advertisement. Until you are convinced to invoke, the exact input and output signatures are unimportant. If the documentation mixes type info with 'why to use' info, it will not be as easy to read.

Additionally, it might be useful to separate the documentation API from the decision to include the documentation in the WSDL, or elsewhere. Providing a separate switch-like API for use in the TWiST configuration file would provide a place for documentation even if this is not published.

Internal API documentation: as Malte has pointed out, it appears that I think I'm the only one working on the internal API. At the moment this is true. But there is external documentation of what the expected API for tWSDL would be. Guess what? They don't match up. The best way to explain this, or make an excuse for it, if you wish, is that there is a vast difference between what you think is going to work, and what actually ends up being a better way. I do not subscribe to the view that it is best to document things first and then stick with this regardless of new discoveries. This doesn't mean that documentation should not follow up when the code matures, but if there is no code to document, or if the code needs cleanup and reorganization, which should come first? Code which does not work does not need documentation, it need work. Code which works but is difficult to understand or of limited re-usefulness, does not need documentation, it needs work. At some point, code becomes clean and useful, which means that others will use it. This is when documentation should be produced. So at the moment, the TWiST API should be the focus of documentation, and it does have some. I will update this with the next release, which will add significant functionality to tWSDL and TWiST.

Another idea comes to mind: is there no way to provide documentation for external software and integrate it with OpenACS? That seems like a good idea, especially for those packages which provide Tcl API.

I'm somewhat confused by this. Are the wrapper procs merely to provide documentation for the &lt;ws&gt;* APIs? Looking at the internals of the &lt;ws&gt;* APIs, indicates that the wrapper have to have a similar internal structure, because there are subcommands. Therefore the evolution of the wrapper would have to concern itself with the internals of the wrapped APIs. This same handholding would be required for every other external package. Additionally, this duplicates the documentation effort, even if it is copy/paste, the wrapper API provides nothing new in functionality. But it does provide another layer which must be maintained and debugged and traced through in case of errors.

If using &lt;ws&gt;, I would really not have a separate procedure for the documentation, both for type as well as for proc (and namespace, if that makes sense). Let the &lt;ws&gt;proc handle the dual calls to insert the webservice along with the documenation into the WSDB. Not so sure that having a different command to desribe the documentation afterwards is the way to go, but probably I am just too much an OpenACS guy.

I'm trying to maintain some civility here, but this is somewhat interesting way of looking at things. First a minor issue. The API is written &lt;ws&gt;*, there are no spaces. It is like constantly mispronouncing someone's name, on purpose due to laziness. Second, I'm not interested in how you think it should be done or where the information should be stored (I am interested in _what_ should be documented). Anything like this must first have a corresponding API in tWSDL. If you read my next-to-last post, I included a comment I put in a file when I first wrote TWiST: higher level APIs must use the tWSDL API. The meaning of this is that higher level APIs cannot mess with the internal storage of the WSDB. This forced use of the API helps identify deficiencies in tWSDL as well as hacks which need code cleanup. I have done several rounds of this already in tWSDL and TWiST.

Having a separate documentation API keeps the two functions separate. I can't imagine how it would be possible to not have massive duplication of code if every &lt;ws&gt;* API had to have its own documentation handler. I have already outlined the deficiencies with the ad_proc method. First is that I would have to duplicate what ad_proc does. But which ad_proc do I follow? There are several versions, and I doubt that we have seen the last version. Changes in version require redoing previous documentation blocks, or maintaining backwards compatibility. If similar to, but not quite the same as ad_proc, a documentation block will lead to inevitable confusion for those who use ad_proc and expect it to be identical. For those unfamiliar with ad_proc, they must learn how the documentation block works. Finally, &lt;ws&gt;proc does more than define a procedure. It defines simpleTypes, complexTypes, input and output messages and an operation.

But most important, and this is really important: I cannot think of a single use of documentation blocks in API calls. Does anyone call db_multirow with a documentation block? I guess you could call ad_page_contract an example. It is a great example of one procedure doing too many things. I'm sure the code for it is difficult to maintain. But just a clue to my method of API development: I write monolithic procs which contain too much code at first. I get things to work, and then I chop up the internals and produce single function APIs to support the original one. I don't keep adding functions to existing API unless it is 'functionally necessary'. Of course I don't always succeed, but it is a valid criticism for code which needs cleanup. Okay one more thing: documentation is data. I don't know if you can include documentation in the schema of a WSDL, but since the namespace of the schema is the xml schema, the rules are the same as that namespace. But even if the WSDL standard doesn't allow the inclusion of documentation in a particular place, this has no importance with respect to the decision to document, at least internally, particular components. The documentation inside the operation might contain a url to an html document, compiled from more complete internal documentation.

My original comment was that it would be nice if there was an independent OpenACS package which allowed you to access the documentation of an external package as if the APIs were part of OpenACS. For instance: Tcl, AOLserver, tDOM. Does anyone wrap these procs? If you could create a package for AOLserver, then the documentation for TWiST would also show up in the same place. Of course my inspect package shows the current state of the code and variables in that thread, but this is not documentation, it is for debugging and analysis.

However, if I were to write a quick 'integration' package for OpenACS, it would import the correct version (for the OpenACS package) from the google code repository. After that import, it can then be packaged up into an .apm file for distribution, or maybe the package would simply do the import upon installation. The package would then provide the TWiST API. The next step would be a wizard type interface which would produce a TWiST configuration file.