Forum OpenACS Development: Tcl Web Services Toolkit: TWiST
A TWiST configuration script placed in the pageroot of an AOLserver is all that is required to enable a web service.
TWiST is available from Google Projects:
http://code.google.com/p/twsdl/
An example service and man2html pages are at:
From my testing, TWiST seems to work fine with the request processor. The TWiST script is treated just like a regular tcl page. It also should not matter exactly when tWSDL and TWiST are loaded, but it is not yet available as an OpenACS package. It can be loaded as an AOLserver Tcl module.
Using TWiST requires zero knowledge of WSDL, SOAP, XML or XML Schema.
This level of simplicity is a big deal, and is comparable to .NET or Java platforms.
This work is under the MIT license, which is GPL compatible. It was funded by United eWay.
I'd encourage the OCT to take a look at this and consider it for inclusion in core.
A couple of questions, though:
- What is the status of the client side of tWSDL?
- Looking at TWiST I see that the examples <ws>proc always return simple types. How do I deal with complex types? Should I define them using the tWSDL types API?
TWiST handles a simple general situation: interfacing with a single Tcl API (proc). A Tcl proc, in general, takes a list of values and returns a single value (which might be a list, who knows). The problem is that in Tcl there is no way to tell what types should be passed in or even less what is returned. An additional problem is that the passed in or returned value might be a list or the name of an array. The TWiST API provides a simple wrapper to handle these different situations.
Another important point is that helpful tools like ad_proc make simple introspection of an existing Tcl proc impossible. Handling switches is relatively complicated, and there are several version of ad_proc available. Same problem arises with the use of args, or for API which return complex data such as db_multirow.
Still another point is that an external web service should probably not expose every possible use of an internal API. Simple introspection in any language can never overcome this problem, and the developer will very quickly find that they are stuck hand coding a wrapper to handle the situation.
So the development cycle with TWiST is aimed at handling the typical situation: decide what to expose. Figure out what inputs are required. Write a script to create the inputs for the internal API. At some point these input will be composed of simpleTypes, and to use your internal API you must know how to transform these into the correct form. The next step is to call your internal API. Then these are transformed into a return value which at some level are represented as simpleTypes. The above script is the body of <ws>proc. This development cycle is identical to a tcl page in OpenACS prior to invoking the template code. The main difference is that instead of ad_page_contract, tWSDL handles, validates and transforms the input. Instead of ad_return_template, tWSDL transforms the result into a SOAP Response. TWiST is a configuration of the entire process.
Now: what about complexTypes? If you look at the testItOperation, notice that it returns a complexType {A B C}. A return type configuration is similar to the input type configuration, with the exception that the return types cannot specify default values. However this is only one level of complexType. I'm working on adding deeper structure to types, it isn't difficult, but the convenience proc which creates a sequence only handles a sequence of elements which contain simpleType content. I have verified that the same validation procedures work with complexContent, but I've just finished with a hand written example and it isn't integrated into tWSDL yet. Eventually there will probably be a TWiST API: <ws>type to expose this, or I might change the internal storage so that the API can look up the type and figure out what to do.
Using tWSDL allows more complete control over defining types, but I haven't specifically tested the idea of creating a type in tWSDL just to use it in TWiST. However, note that TWiST is already doing this. Types are specified by similar syntax as used in ad_proc: myVar:typeName. The difference is that TWiST and tWSDL used namespaces for types, the default being XML-Schema types, which has a prefix of xsd. The prefix is used in the schema database as a Tcl namespace name. Each prefix has only one namespace uri. Externally, you can use a different prefix. Internally they are mapped to specific prefixes. So, when you create a new web service with TWiST, the tail of the Tcl namespace is used as the prefix internally. The namespace uri becomes 'urn:tcl:yournamespace' by default. This can be modified. So if you use tWSDL to create a type, you would create it in some xml namespace with associated prefix. To use it, you use the prefix: {a:mytypes::specialType b:somexsdType}. It is important to always derive your types from xsd types since tWSDL always imports all types into the WSDL. That way, clients will be able to validate your documents.
Most of the functionality is already in tWSDL, just not arranged into a client. My hope was to have an API similar to the tWSDL server API. One difference is that the client should be able to build itself from reading the WSDL file. I have begun work on an XSLT script which reads the WSDL generated by the server. So far this approach seems to work pretty well.
I installed and tried the the sample 'mywebservice' script and it works as expected, even if during aolserver startup the log file reports this error:
[17/Feb/2007:11:18:41][4470.3083290288][-main-] Notice: TWS:---->[ns_info config] = '/usr/local/aolserver_45_r0/base.tcl' [17/Feb/2007:11:18:41][4470.3083290288][-main-] Error: TWS:no such set: while executing "ns_set size $modules" (in namespace eval "::wsdl::server" script line 14) invoked from within "namespace eval ::wsdl::server { variable hostHeaderNames foreach option {address hostname name server servers config} { log Notice "---->\[ns..." (file "/usr/local/aolserver_45_r0/servers/server1/modules/tcl/twsdl/packages/wsdl/tcl/wsdl-server-init.tcl" line 7) invoked from within "::source /usr/local/aolserver_45_r0/servers/server1/modules/tcl/twsdl/packages/wsdl/tcl/wsdl-server-init.tcl" ("uplevel" body line 1) invoked from within "uplevel ::source "$file"" (procedure "::tws::sourceFile" line 4) invoked from within "::tws::sourceFile [file normalize [file join [file dirname [info script]] "wsdl-server-init.tcl"]]" (file "/usr/local/aolserver_45_r0/servers/server1/modules/tcl/twsdl/packages/wsdl/tcl/wsdl-init.tcl" line 11) invoked from within "::source /usr/local/aolserver_45_r0/servers/server1/modules/tcl/twsdl/packages/wsdl/tcl/wsdl-init.tcl" ("uplevel" body line 1) invoked from within "uplevel ::source "$file"" (procedure "::tws::sourceFile" line 4) invoked from within "::tws::sourceFile [::tws::util::package::directory [file join $packageName tcl ${packageName}-init.tcl]]" (procedure "init" line 4) invoked from within "init $package" PACKAGE wsdl [17/Feb/2007:11:18:41][4470.3083290288][-main-] Notice: TWS:Package xml: Initializing...
I followed your instructions regarding a pure aolserver configuration and now I wish to configure OpenACS: is it enough to symlink the twsdl packages under the www directory of the OpenACS service?
TIA,
Claudio
Yes, it works as expected even with that error on startup. I developed this on 4.5, but when I tested it on an older AOLserver, I got the same problem. (but it looks like yours is also 4.5?) This will affect the startup of tWSDL services (since no calls to ns_register_proc), so I'll need to address it at some point to figure out why the error shows up.
TWiST works because it uses the default process to return a page in pageroot.
tWSDL has a series of packages in a structure similar to but independent from OpenACS. Installation into OpenACS should be exactly the same as regular AOLserver. Some packages have a www directory, and to use the files, just symlink them where you want (from parent directory):
ln -s /path/to/and/including/package/www mydir
But note that only the www files go under pageroot, all the others are library files.
One thing I haven't discussed is ideas on where to put the TWiST configuration files, but short and simple, you can keep them with the API that you are exposing in a separate file, using whatever name you thing is best, but making sure that it doesn't get sourced at startup. Then create a directory structure under pageroot to reflect the overall hierarchy of your web services. Each directory then gets a symlink from the TWiST config file to an index.tcl file in that directory. That way, you get everything in one place. If you have directory browsing setup, then it will work like
Subdirectories will have separate services and no annoying tcl file in the service URL.
Also, for those interested, the same TWiST script will probably work inside an adp file.
Another note for older AOLservers: tDOM is loaded using ns_ictl in tWSDL. This will need to be changed to the method recommended by tDOM. Also, you might already have tDOM loaded for AOLserver. In that case you should not load the tWSDL tdom package. You can do this by editing the config file:
twsdl/packages/tws/tcl/tws-local-conf.tcl
Remove 'tdom' from the ::tws::packages list.
Another hint: the procedures and data structures for tWSDL and TWiST are generated on startup or after visiting a service page. To inspect the code, you should use the inspect package. Link in the www directory so you can browse the exact procs and vars, but also note that this is per thread configuration. For TWiST, visit the service then immediately visit the inspect browser. Otherwise AOLserver will eventually exit the thread and the data will not there (for TWiST).
BTW, most of the tWSDL and TWiST API works in a command line mode using nstclsh (even tclsh!) and sourcing the twsdl/init.tcl file. If you then run the <ws>* api, you will see what variables are not set due to not being in a conn. You can set these one by one and re-run the failed API (usually <ws>namespace) and eventually get the service configured. This highlights the fact that AOLserver is a thin, but important, layer over tWSDL and TWiST. If you replace this thin layer, you can run either of these inside any other server.
One idea I am considering is to do something like this:
proc ws { args } { return "<ws>"}
[ws]namespace ...
[ws]proc ...
[ws]return ...
Then the developer can replace the ws proc above to return different things, such as ::myCustomMods::ws_, or whatever text you want. The above code runs the standard API.
The above decoration is somewhat like what is used for C#, and with args it might offer some additional setup. For instance, the namespace of the commands might hold additional hints as to what defaults to apply to the service (like the targetNamespace, etc.)
This is also an interesting way to do a 'soft import' of a command, since in Tcl command import is almost useless except as a convenience to specify a command name (variable remain in the original namespace).
Thank you very much for the other tips!
Creating a Web Service has never been so easy: this is really SOAP for the masses!
I've created a tag for the initial release so that I can continue to work. You can check out tWSDL/TWiST using svn:
svn checkout http://twsdl.googlecode.com/svn/tags/release-0.5 twsdl
I've completed the overhaul of the XML package, so I have another release/tag. The overhaul didn't change the TWiST API, although there were small adjustments to the tWSDL API. The reason for this work was that as tWSDL was developed, the XML procedures were put in toward the end, once I knew that the internal representation was working okay. This delay resulted in three ways to create XML nodes. Now there is only one method. This also allowed me to handle more valid documents from clients.
Another change is that XML node structure is checked/validated prior to the datatype of the children. This should speed up rejection of invalid documents.
The new svn release is here:
svn checkout http://twsdl.googlecode.com/svn/tags/release-0.6 twsdl
I just copied this over my install at junom.com and it seems to work as expected.
One very simple client is wsdlpull. It is a C++ command line tool and API. It is available here (with links to the source): http://dag.wieers.com/rpm/packages/wsdlpull/.
I've tested it on the example web service. It produces the expected results and it also rejects invalid inputs prior to sending them to the server. TWiST has a built in tool for testing which does allow invalid inputs so that you can test the response documents.
after downloading the release-0.6 tag and playing a little with TWiST and the basic tWSDL API I have some doubts as how to proceed further.
First of all checking the environment sourcing init.tcl with nstclsh I get the following error:
[24/Feb/2007:11:59:47][4792.3083896496][-main-] Error: TWS:no server while executing "ns_info $option" ("foreach" body line 2) invoked from within "foreach option {address hostname name server servers config} { log Notice "---->\[ns_info $option\] = '[ns_info $option]'" }" (in namespace eval "::wsdl::server" script line 3) invoked from within "namespace eval ::wsdl::server { variable hostHeaderNames foreach option {address hostname name server servers config} { log Notice "---->\[ns..." (file "/usr/local/aolserver_45_r0/servers/server1/modules/tcl/twsdl/packages/wsdl/tcl/wsdl-server-init.tcl" line 7) invoked from within "::source /usr/local/aolserver_45_r0/servers/server1/modules/tcl/twsdl/packages/wsdl/tcl/wsdl-server-init.tcl" ("uplevel" body line 1) invoked from within "uplevel ::source "$file"" (procedure "::tws::sourceFile" line 4) invoked from within "::tws::sourceFile [file normalize [file join [file dirname [info script]] "wsdl-server-init.tcl"]]" (file "./packages/wsdl/tcl/wsdl-init.tcl" line 11) invoked from within "::source ./packages/wsdl/tcl/wsdl-init.tcl" ("uplevel" body line 1) invoked from within "uplevel ::source "$file"" (procedure "::tws::sourceFile" line 4) invoked from within "::tws::sourceFile [::tws::util::package::directory [file join $packageName tcl ${packageName}-init.tcl]]" (procedure "init" line 4) invoked from within "init $package" PACKAGE wsdl [24/Feb/2007:11:59:47][4792.3083896496][-main-] Notice: TWS:Package xml: Initializing...Note that this error don't show up during aolserver startup, which however continues to exhibit the previosly posted error.
Regarding the basic tWSDL API I finally built a working sample service which accepts a user_id and returns the name and the email of the user.
So doing I learned how to build complex types, how to define an operation and associate to it a tcl proc and now I'd like to comlicate the matters a little, but I don't know how to populate my complex types within my tcl proc.
Given the following definitions:
eval [::wsdl::elements::modelGroup::sequence::new $usersns oneUser { {name userdecoder::name} {email userdecoder::email}}]how do I put the values pulled from the db into the Users structure?eval [::wsdl::elements::modelGroup::sequence::new $usersns Users { {oneUser userdecoder::oneUser}}]
A few other questions.
Is it possible to view the generated WSDL, like in TWiST?
When you define a ::wsdl::operations::new each parameter of the tcl proc is followed by the string 'Value': what is its meaning? Is it mandatory?
TIA,
Claudio
- I found how to show the WSDL definitions using xml::document::print ::wsdb::definitions::${serverName}]
- I ingenuously assumed that once defined a complex type I could use it as a component of a higher level complex type, but it is not so: when I try
eval [::wsdl::elements::modelGroup::sequence::new $usersns Users { {oneUser userdecoder::oneUser}}]
I get the errorcan't read "::wsdb::types::twist2ns::oneUser::validate": no such variable
and so my question is: how do I define higher level complex types?
I know the API call seems simple enough, but it hides a lot of code generation, including the type validation routines and the invoke routine. These routines would look different than the simple case. This seems to suck pretty bad until you consider the number of Tcl API which take or return multi-dimensional data.
It turns out that type validation is the easiest to extend to multi-dimensional data, but overall, the problem is not too difficult and was planned for in advance.
This is the meaning of the {ElementA Value ElementB Value} configuration (called a conversionList). The modelGroup::sequence API assumes actual values for each child element. This configuration is used in the invoke API (which is auto-generated) by the API ::xml::childElementsAsListWithConversions
Currently this simple API only handles the Value type, although if you look at it, other types are commented about.
Let me just say that developing the type system and the API is very delicate, it has to reflect the method of type development used by XML-Schema, otherwise we cannot publish the interface in a way that a generic client can use. Beyond the simple interface, the developer must become more aware of how things work throughout the entire system and in XML-Schema, WSDL, etc.
I'll post another note about the invoke procedure interface, which is a key abstraction. Any developer using tWSDL for complex services must understand this interface.
Here is an example auto-generated invoke procedure:
proc ::wsdb::operations::openacs::CheckEmailOperation::Invoke { inputXMLNS outputXMLNS } { variable conversionList ::xml::childElementsAsListWithConversions $inputXMLNS $conversionList return [::wsdb::elements::openacs::CheckEmailResponse::new $outputXMLNS [::openacs::CheckEmail $Email]] }
This tiny proc is the interface between the XML world and the Tcl world. This procedure could be completely hand written as long as it takes the same input (where to find the Tcl rep of the XML document and where to put the output document) and returns the same output (reference to the return document, a child of outputXMLNS).
In addition, the operation's namespace variable 'invoke' must be set to the fully qualified proc name.
From the point of view of the server, the size and complexity of the document isn't important. But obviously the underlying Tcl API which can handle complex data is beyond the scope of the interface: it is hidden inside the invoke procedure. This is the purpose of this interface, otherwise it would take a few seconds of thought to discover something you couldn't handle, and hacking would begin. The invoke procedure is independent of other parts of the tWSDL API. You can completely test this procedure without the need to run the SOAP server, therefore it is also independent of the binding or protocol. If someone adds a new binding, the invoke procedures remain valid.
One use case which I am working on is a complexType which contains a sequence of children all of the same type, such as a sequence of database rows. It works as expected, but I don't yet have a higher level API to write the code.
If you run the modelGroup::sequence::new API with 'puts' or 'log Notice' instead of 'eval', you can see what code is actually generated.
Both errors you have reported are due to the fact that a server isn't defined. When you use nstclsh, as a command line tool, you will not have certain functions and data structures: ns_conn, etc. This is normal, it is not a bug. To use tWSDL to deliver a service, you must currently use a virtual server.
Virtual servers are enabled by moving the nssock* drivers to the global modules ns_section:
ns_section ns/servers
ns_param jnm {Junom Server}
ns_param twsdl {tWSDL}
ns_section ns/modules
ns_param nssock1 $bindir/nssock.so
ns_section ns/module/nssock1
ns_param port 8001
ns_param hostname example.com
ns_param address 192.168.1.102
# Important:
ns_param defaultserver jnm
# Virtual Server Host headers:
ns_section ns/module/nssock1/servers
ns_param jnm maria:8001
ns_param jnm junom:8001
ns_param jnm 192.168.1.102:8001
There are probably other changes to enable virtual servers, but maybe someone has done this for OpenACS?
The other option is to replace the script in wsdl-server-init.tcl.
I installed the wsdlpull client, but I'm getting this error, no matter what service I'm calling:
Error processing http://www.xmethods.net/sd/2001/CurrencyExchangeService.wsdl An Exception occurred at 0:0 Wsdl Parser Exception : http://www.xmethods.net/sd/2001/CurrencyExchangeService.wsdl could not be openedApparently the compilation went good: could you kindly post the command you are using to invoke your stockQuoter service?
http://junom.com/ws/openacs/?WSDL
From my experience the wsdlpull client more correctly constructs SOAP messages. It doesn't know how to verify restrictions based upon patterns (regexp), but it does verify enumerations.
On the command to invoke stockQuoter, I guess you mean what SOAP message to send? In the stockquoter/www directory, there should be a post.tcl file, which probably needs to be edited so it sends to the correct address. Otherwise what you are asking for is a client, and wsdlpull looks like the easiest to use.
Otherwise, move the stockquoter procs to another file under pageroot and use the <ws>* API.
If you installed from the rpm then this could be a problem
Apparently there is an issue with the rpms.Try compiling from source or better still checking out from CVS.I was able to invoke the webservice without any problems
wsdl http://www.xmethods.net/sd/2001/CurrencyExchangeService.wsdl getRate usa India
44.08
Hope this helps
If you still face a problem post it on the wsdlpull mailing list
I've started work on a few new APIs for TWiST. The first is <ws>type. This API will make it a little easier to add simpleTypes.
To demonstrate the use of this new API, I have included another service called datetime. This service demonstrates code for validating the many XML-Schema date types. There is also a procedure/service for adding/subtracting a duration from a dateTime.
Services are available at: http://junom.com/ws/ and code at:
svn checkout http://twsdl.googlecode.com/svn/tags/release-0.8/ twsdl
a) Documented procedures
b) Easier to install / use / maintain with OpenACS
I am stumbling across a couple of issues though, which mystify me a little bit 😊
1) Why do I need to source the wsdl-definitions-procs.tcl procedure in addition to have it sourced by OpenACS. You make a special call for sourcing and viola it works. Out of curiosity: Why ?
2) You are using global variables in the namespace. Would it make more sense to translate these into NSVs ?
3) Is it okay if I commit my work to OpenACS CVS repository?
4) In the checkout which I did two days ago from SVN I could not find the <ws>type code you mentioned for making it easier to add simple types. Maybe I am missing something ?
Probably more once I actually start the coding instead of (at the moment) trying to get all together in an OpenACS package, while understanding the dependencies of the code.
Please respect that TWiST and tWSDL are part of an independent project. I intend to be responsive in maintaining the code and providing examples of use and documentation. However I use a very simple framework to do this, and it works with Tcl, AOLserver and OpenACS as-is.
Project Home: http://code.google.com/p/twsdl/
1) I'm not sure why you would need to directly source this file, what exactly did you do, or what instructions did you follow? What is the special call you tried? What were you expecting to happen? You should only source the top level init.tcl file, which will handle startup:
http://twsdl.googlecode.com/svn/trunk/init.tcl
You might edit a local configuration file to remove any unneeded packages:
http://twsdl.googlecode.com/svn/trunk/packages/tws/tcl/tws-local-conf.tcl
For instance, you can remove the tdom package if it is already loaded in OpenACS. (Just edit the package list in the above file.) Also StockQuoter isn't needed.
2) Which global variables? Do you mean namespace vars? TWiST and tWSDL operate on a database of information that I call WSDB. In short there is lots of information and procedures behind the scene. In fact, a lot of code is autogenerated from the API calls. This info/procs are 'stored' in Tcl namespaces. NSVs lack structure, they are just shared arrays. The only way to understand the WSDB is to look at it in a running server. How? Link the inspect package somewhere where you can look at what is in the database (::wsdb::*)
http://twsdl.googlecode.com/svn/trunk/packages/inspect/www/
3) No, please don't do this. tWSDL and TWiST are library code, and a completely separate project. An OpenACS package should use the API, but not modify. Of course, the OpenACS package should somehow download the correct release. I can help support this option, probably this needs some best practices input. Does any other package incorporate external code? It would be nice to see how it is done. Further documentation for the project is here:
http://twsdl.googlecode.com/svn/trunk/packages/doc/
4) the <ws>type and <ws>element procs are in the main trunk:
http://twsdl.googlecode.com/svn/trunk/packages/wsapi/tcl/ws-procs.tcl
Example of use is the stockquoter example (converted from tWSDL):
http://twsdl.googlecode.com/svn/trunk/packages/wsapi/www/stockquoter/index.tcl
When you get TWiST sourced, you can link the wsapi/www directory under pageroot to test things out.
Also note that I have some promised upgrades to tWSDL and TWiST in dealing with procedures which take or return arrays or lists, or lists of arrays or lists. I thought this was going to be a simple process, but I'm taking the opportunity to simplify and generalize the code. It's looking good, maybe the end of this week. Once this is done it should be very simple to extend this to cover ns_sets as input/output (One short procedure). Documentation overhaul will start once this is finished.
Here are some of the issues:
* Namespace conflicts
* Reliance on a multi server setup
* Incorrect loading of wsdl-procedures
* Server errors due to "log" not being a recognized command
4) Yes, I see they are there, but you mentioned an example using dateTime, which seems to be missing.
Good to know that you are working on that though, if you do, then we should hopefully have a less burdensome installation process. And then we could have a wrapper package for OpenACS which provides documented procedures for the main functions used. This could then call "package require twist" and hopefully all is working. Because, if you provide this as an indepent API, then having package require working would be a major hit 😊.
Again let me emphasise that I have no intention of taking over your code. I just wanted to get the current version and stuff it into an OpenACS package which is easy to use within OpenACS as we do it a lot of times e.g. with XinHA, AJAX and other tools that we find useful. But I do respect your wish not having it there, though then I would ask you to make the installation easier, which includes not touching the config.tcl file but be able to use "package require". Then I would also install it under /usr/local/aolserver/lib and not the TCL root of AOLserver (as it does not really belong there, in an OpenACS context).
To use it I manually have to link the WWW directory. This is not possible with an OpenACS package as you cannot know where the TWIST is installed in the first place. Why not break your package up into something that you unzip in the root directory of the server and which splits into the /tcl directory just as now, but also unzips relative links into the www directory, so at least you can access the files. Something like /www/twsdl/wsdl (as an example).
And for the documentation, at the moment I always have to manually go into the tcl directory to find the procedure. This is why I would love to exchange all occurances of "proc ::" with "ad_proc ::", so at least the procedure is avialable for browsing in the ACS API and I do not have to go back to the shell, go to the directory, open the file,search for the procedure, look what it does and what variables it expects and what variables it returns. This makes development really burdensome 😊. This could be quickly fixed by you however. Just have a wrapper in init.tcl like:
if [lsearch [info procs ad_proc] ad_proc] <0} {
# insert the pass through definition for proc here
}
and then replace all instance of proc with ad_proc.
And for the documentation, at the moment I always have to manually go into the tcl directory to find the procedure. This is why I would love to exchange all occurances of "proc ::" with "ad_proc ::", so at least the procedure is avialable for browsing in the ACS API and I do not have to go back to the shell, go to the directory, open the file,search for the procedure, look what it does and what variables it expects and what variables it returns. This makes development really burdensome 😊. This could be quickly fixed by you however. Just have a wrapper in init.tcl like:
You will not get very far with tWSDL or TWiST if you rely on the _static_ definition of ad_proc. You should use the inspect package which is part of tWSDL. You mount inspect anywhere you wish, by creating a symlink. I don't expect everyone to agree with me, but inspect gives you the current state of namespace variables and procedures, not what you think they are, or were at one point in time. Since most of the code in TWiST is autogenerated, you have to use something like this.
I know that as of right now, <ws>type and <ws>element are not documented, they are still in development and subject to minor changes. This does not mean that I need to rush out and use ad_proc, which would help nobody. I'm not really sure why anyone would recommend this. Did someone really do such a thing with tDOM, or with tcllib? How about Tcl itself? The world does not revolve around the OpenACS API, and complaints of this sort don't help anyone or solve any real problem. In fact, if I followed your instructions, the package would immediately only work for full installations of OpenACS. But, my goal is to abstract out even the dependencies on AOLserver, which will take very few hours as it only involves _one_ procedure in all of tWSDL! In the same way, I'm sure we can come up with a very thin layer to integrate TWiST with OpenACS, but not by hacking away on code which works.
Please keep in mind that TWiST is a _user_ API, tWSDL is a developer API. Unless you are very comfortable with how both work, it is way premature to start 'fixing' any code.
My suggestion for ad_proc also and solemnly was referring to the procedures which we are actually using to create the webservices, not the auto generated code. Obviously that would be a little bit harder to achieve. Though I still think having support for < wsdl:documentation > would come in handy (but maybe it already does this and I just did not find in the documentation where). The way I would probably find ist useful:
< ws >type pattern datetime::year $yearAnchored xsd:integer documentation:"Year of the datetime"
Or use the way your hated (sorry 😊 ) ad_proc does it for procedures
< ws >proc ::datetime::DayNameFromNumber {
{DayNumber:datetime::dayNumber}
} {
Returns the day of the week for a given number
} {
return .....
} returns {DayName:datetime::dayName}
Please be not offended by this suggestion. If you don't like it throw it away. I would be pleased to see your suggestions for creation and integration of < wsdl:documentation > tags, especially using the < ws > procedure.
Have you read the installation instructions? This installs just like an AOLserver module, because it is an AOLserver module. I think what is confusing you is that I have used some OpenACS conventions in file naming (pkg-procs.tcl, pkg-init.tcl, etc.) These should be ignored by the user.
But really, you are asking design/development questions and criticisms prior to installation. Maybe you will find the functionality useless, but to be thinking about improving it for your own use at this point will be a big waste of your time and mine, and will give the impression that there is some issue which doesn't exist. The code is working, and you can test it for yourself at:
If you are thinking about changing the API and incorporating it into OpenACS, please don't do that. At the very least it is somewhat rude to take over this role. But in fact, I have a stake in seeing that any OpenACS package works as expected, and continues to work into the future.
Otherwise please ask a specific question about installation, and explain _exactly_ what you are doing, and what instructions you are following.
As mentioned before, I have no intention of taking over your code. I want to use it and make it easy to use in the environment I work with, which happens to be OpenACS. If you find the fact that I did this rude, please accept my apologies, as I have no intention of doing anything that might offend you (though, judging from the tone in your replies you are offended, for which I am sorry and just hope you will be able to accept my sincerest apologies).
1) Are the following two equivalent (one found in /stockQuoter/tcl/stockQuoter-procs.tcl):
::wsdl::types::simpleType::new $sqns name xsd::string
and
<ws>type simple stock::name (found in /wsapi/www/stockQuoter/index.tcl)
2) If I use the <ws> syntax, do I still need to have the definition of the type in the -procs.tcl ?
3) Is my assumption correct that each <ws>proc definition will be turned in an Operation which you can choose from, adaptly named ${proc_name}Operation ?
4) Which type types do we have. What is ENUM supposed to be, and do you need the "<ws>type pattern stock::Code {[0-9]{4}} xsd::integer" in the example provided? (and if yes, will you be able to define this way that ::Code is a variable of type integer that consists of 4 characters? Is pattern always a regexp (as I could deduct from looking at datetime/index.tcl).
5) It seems that the datetime/index.tcl does not seem to work, at least no option 1 and 3. When I use 3 (get day name from number), I always get an incorrect input value (which I realize is due to the fact that you where using enumeration with xsd:integer, which does not seem to work. I managed to get it to work with "wsdl" pull by just using enum with the optional values {0 1 2...6}, though that lacks the type definition.
6) How can I reload a service definition. At the moment, whenever I change e.g. /wsapi/www/datetime/index.tcl I have to restart the server. Is this due to the fact that I have a symbolic link from an OpenACS packages directory or is this for real. if it is for real, how can it be prevented ?
7) Would it be possible to wrap the service description at http://e4.cognovis.biz/twsdl/wsapi/datetime/?WSDL into some <code> tags, as Safari is happy in displaying ... nothing (I found out that it actually works by opening the page with FireFox).
8) Is there a "cleaner" documentation than the pure service description, where we could story a little bit more information what the variables are supposed to do. Somewhat like ad_proc, if you like. There does exists a <wsdl:documentation ...> tag, maybe use that one ?
The more I look at the examples (and fix some issues e.g. with datetime) the more I like it. Sadly I fear the ease of <ws> procedure will be lost on me as the other side expects a certain naming convention, but I will see.
Maybe your interest is in fixing stuff instead learning of how TWiST works? There are no known bugs, at least with the things you are talking about. Please stop making requests for code changes until you actually get this installed. TWiST is a simplified API on top of tWSDL. tWSDL is an attempt to satisfy the requirements of the WS-I Basic Profile for web services, it is not an attempt to satisfy you or the OpenACS community. Therefor the way things work, has a lot to do with the WS-I Basic Profile. Criticisms directed at the functional requirements are appreciated, those directed to a matter of preference are not.
Personally I think it is cool that an unrelated tool (wsdlpull, etc.) can grab a wsdl file generated by TWiST and construct C++ code to create a SOAP request and parse the result document. It is cool that for simple types, and enumerations it can validate inputs prior to sending. But this is not software that does everything, or should do everything. Type definition is a great example. Types are constructed according to the way they are derived in XML-Schema. The reason is obvious: in order to build a validation script inside a client, or server, you have to know how the types are derived from the basic types. Patterns are regular expressions, because that is what they are in XML-Schema. If you are uninterested in type definition, type validation, etc, then don't use WSDL, that is the whole point.
But it is not cool to have an application which satisfies some short term interest at the expense of interoperability. Anyone wishing to hack on tWSDL or TWiST should read all of the documentation first, and the references, especially those listed in:
http://twsdl.googlecode.com/svn/trunk/packages/doc/www/current-standards.html
Otherwise very many misunderstandings will result.
noname:~ malte$ wsdl http://junom.com/ws/datetime/?WSDL
1.AddDurationToDateTimeOperation
2.CheckDateTimeOperation
3.DayNameFromNumberOperation
Choose one of the above operations [1-3] :3
DayNumber: 2
Incorrect input value 2
noname:~ malte$
On my system, after appying a change to /datetime/index.tcl as mentioned above it looks like this:
noname:~ malte$ wsdl http://e4.cognovis.biz/twsdl/wsapi/datetime/?WSDL
1.AddDurationToDateTimeOperation
2.CheckDateTimeOperation
3.DayNameFromNumberOperation
Choose one of the above operations [1-3] :3
DayNumber: 2
DayName:Wednesday
noname:~ malte$
Now, if you say that "wsdl" from "wsdlpull" is not a program to use for your code, then this is fine, but it was mentioned by you, so I do assume that I can use it for testing purposes.
What I am interested though is understanding how I can export pm::task::new as a webservice, so I could use e.g. wsdl to create a new task. That is my ultimate goal, and this is why I am asking all those questions which I hope you will answer eventually, unless you say it is all obvious from the documentation, then I will admit defeat and humbly ask you where it is answered in the documentation (as I read through a lot and might have missed it).
Last but not least, I have not intention to hack on tWSDL or TWiST. Unless you define exchanging "proc" with "ad_proc" for your proc definitions as hacking. And you made your wishes *very* clear and I respect that. Still I feel the installation and documentation could be improved, what you make of that is entirely up to you, this is *my* personal opinion and not meant as an insult.
1) The Service Description does not get updated. I guess this is because of "
# eventually must add check to delete prior namespace and wsdb code
# also might put in variable to freeze additional changes (better word?)
# this 'freeze' would operate in <ws>proc as well to silently abort and log event" in <ws>namespace init ?
Either way, rediting, I found out that I need to unfreeze my namespace 😊. So I assume that, when I want to change your demo code, I need to make sure to either not freeze the namespace or at least unfreeze it before working on it.
2) I cannot use xsd:: types directly but have to make a type declaration first. Though I am not sure why in your example for stockQuoter you have been generating identical types (::quote, ::dailyMove, ::lastMove). What am I missing here? Wouldn't it be enough to just define stock::float and then use that for the sequence definition? I mean, it worked when I tested it, but maybe there is a deeper meaning behind it which I am unaware of yet.
3) I found the documentation (more or less) for the < ws> functions. Not linked from anywhere, but it seems autogenerated using the man command. I linked it up at http://e4.cognovis.biz/twsdl/doc/api-index.html.
4) I might be wrong, but does < ws>proc check if the data type returned matches the value? I am not sure if it should, but I would probably prefer a server error on my machine than on the querying machine in case the returned type does not match the definition of the to be returned type (as an example, take the dateTime example. If you set the dateTime manually to "TWiST" then it is still returned and I could not find any error /Notice that what I was returning was not of the datetype specified.
5) How can I give a default value that does not have to be filled in. Or, lets be precise, how can I test it? I tried with the stockQuoter example of yours not to pass in the "verbose" value and failed, both with wsdl as well as the web interface. At the moment I have to fill it in, regardless what it says in the definition and this makes me wonder where the trick is.
After working with the toolkit, here are the answers to my questions.
1) Yes
2) No
3) Yes
4) Given by Tom
5) This was a problem due to my misunderstanding of duration and dateTime. Once I looked both datatypes up in the WSDL specifications and provided correct values it works. Read the specification on Duration though, it has a special format
6) Make sure not to freeze your webservice. This is the default in all examples given, probably something that should be noted.
7) Okay, this question was incorrectly asked. I wanted to have a second place where I could get the generated service definition wrapped in <code> tags so I could read the output (full) for debugging purposes. It obviously does not make any sense to wrap the ?WSDL in those tags. For the time being, just open the link in your browser and click "Show Source" to see the outcome.
8) No, and Tom has no intention to provide it. Just in case you wonder, I am talking about extend <ws>proc by a documentation block which will result in <wsdl:documentation ..> tag being inserted.
This last issue I take very seriously, and I don't think it is appropriate or helpful for the TWiST/tWSDL project to be forked _in any way_. But let me explain how I had planned for the re-use of tWSDL so that anyone who wishes to provide a different layer on top of tWSDL is can do so.
There is a package in tWSDL called wsapi. The wsapi-procs.tcl file has this comment:
# Linked files may not carry the same copyright as this file. # Files linked here should present higher level APIs which: # 1. must use the twsdl api # 2. act as a layer above the twsdl api # 3. provide useful defaults ...
No this isn't in any external documentation, but then again, nobody will be able to provide a higher layer without becoming very familiar with all of tWSDL.
So TWiST is _but one_ higher level API above tWSDL which uses the tWSDL API, and provides useful defaults.
If any person or organization wishes to extend tWSDL, they are free to do it, and will hopefully contribute and maintain their code so that it is available either on Google Code (in twsdl or otherwise), or somewhere else.
But please, please, do not modify the TWiST API, or extend it and offer an extended version calling it TWiST, etc.
In a number of places I also clearly identify TWiST as an _example_ use of tWSDL. Anyone wishing to develop an separate layer on top of tWSDL should follow this example. The tWSDL API must be invoked in a more or less fixed sequence, or at least a number of things must be done prior to some important step.
Another important point about TWiST, and why it must remain free from enhancements is that I use it to test and debug tWSDL. For instance, TWiST allowed me to quickly rewrite some of the tWSDL code so that it was immune from certain user errors or practices. One of these is the ability to reinvoke the same commands without messing up the web service.
Another: although it would be nice to imagine that all code will remain bug free into eternity, this doesn't seem to ever occur. Page level debugging is very difficult to perform, and usually requires insertion and eventual removal of ns_log statements. Instead of this, tWSDL and TWiST can be debugged using a Tcl shell. The AOLserver layer is so thin that it only takes a few set statements to setup the TWiST service. If an error occurs, you can immediately investigate the current state of the process (because there is only one state and you are right in the middle of it).
Also note that if available, the tWSDL will load libnsd.so so that normal ns_log statements are printed, if not, the text of the log statements are printed, minus the timestamp.
Documentation!
Malte is of course correct that the documentation could be improved, or easier to find. But there are different forms of documentation aimed at different kinds of users. The TWiST API was documented as an AOLserver module, which uses nroff files which are compiled into man and html files. I'm not sure why Malte thinks that tWSDL/TWiST is not a proper AOLserver module, but it seems to be. I don't think it is correct to provide tWSDL/TWiST as a Tcl package, since in general it requires the AOLserver API for normal usage. So this is why the primary documentation for TWiST is in provided in this form. I'm not particularly fond of nroff, but that is the current state-of-the-art with AOLserver.
Also, as Malte points out the <ws>namespace freeze command is poorly documented. He correctly discovered that you should only use it when you complete your development of the TWiST configuration file. In addition, if you don't use the freeze subcommand, you need to either wait for your connection thread to exit, or restart your AOLserver _if you are changing an existing simpleType or complexType_, either directly or via a changed <ws>proc call.
WSDL documentation. The tWSDL database (WSDB) contains internal representations of the WSDL XML file for each web service. These are child namespaces of ::wsdb::definitions. You can browse this using the inspect package. Using the ::xml::* commands, you can add to this internal representation, and it would be pretty easy to write a higher level API to insert documentation. The question is: what would the tWSDL API look like, and what about the TWiST API?
Personally, I like the documentation method used in SQL: separate commands which identify the thing to be documented, followed by the documentation. I do not like the concept of adding an additional block to the TWiST API. Here's why: you should be able to document types. Types can be reused, in fact, they can't be repeated in the WSDL file. Therefore, if you document the types, documentation shows up in one place. Operations are a higher level concept. Documentation for Operations identify why you would invoke. This is kind of like an advertisement. Until you are convinced to invoke, the exact input and output signatures are unimportant. If the documentation mixes type info with 'why to use' info, it will not be as easy to read.
Additionally, it might be useful to separate the documentation API from the decision to include the documentation in the WSDL, or elsewhere. Providing a separate switch-like API for use in the TWiST configuration file would provide a place for documentation even if this is not published.
Internal API documentation: as Malte has pointed out, it appears that I think I'm the only one working on the internal API. At the moment this is true. But there is external documentation of what the expected API for tWSDL would be. Guess what? They don't match up. The best way to explain this, or make an excuse for it, if you wish, is that there is a vast difference between what you think is going to work, and what actually ends up being a better way. I do not subscribe to the view that it is best to document things first and then stick with this regardless of new discoveries. This doesn't mean that documentation should not follow up when the code matures, but if there is no code to document, or if the code needs cleanup and reorganization, which should come first? Code which does not work does not need documentation, it need work. Code which works but is difficult to understand or of limited re-usefulness, does not need documentation, it needs work. At some point, code becomes clean and useful, which means that others will use it. This is when documentation should be produced. So at the moment, the TWiST API should be the focus of documentation, and it does have some. I will update this with the next release, which will add significant functionality to tWSDL and TWiST.
Another idea comes to mind: is there no way to provide documentation for external software and integrate it with OpenACS? That seems like a good idea, especially for those packages which provide Tcl API.
with regards to the wsdb documentation I would love to see both documentation for types and operations. But to my understanding the standard WSDL 1.1 only supports the documentation tag in the operation, not in the type defininition. If I am wrong, then please find a way to describe the types, this does make much more sense (as you noted).
How could the WSDB documentation be represented: In general I would have the WSDB be enhanced by documentation fields, so the inspect functions could display the documentation as well. Here having additional commands for the documentation might be a good thing.
When displaying the webservice itself (using ?WSDL or your internal display method) the generated < documentation > tags can then be viewed when you call your page (w.g. at e4.cognovis.biz/twsdl/projectmanager), which list all the operations, along with the corresponding documentation. Once you click on one operation, you will see all the entry fields along with their documentation. Probably you would have to display what is expected to come back with documentation as well. Not sure how this works.
If using < ws >, I would really not have a separate procedure for the documentation, both for type as well as for proc (and namespace, if that makes sense). Let the < ws > proc handle the dual calls to insert the webservice along with the documenation into the WSDB. Not so sure that having a different command to desribe the documentation afterwards is the way to go, but probably I am just too much an OpenACS guy.
I'm somewhat confused by this. Are the wrapper procs merely to provide documentation for the <ws>* APIs? Looking at the internals of the <ws>* APIs, indicates that the wrapper have to have a similar internal structure, because there are subcommands. Therefore the evolution of the wrapper would have to concern itself with the internals of the wrapped APIs. This same handholding would be required for every other external package. Additionally, this duplicates the documentation effort, even if it is copy/paste, the wrapper API provides nothing new in functionality. But it does provide another layer which must be maintained and debugged and traced through in case of errors.
If using <ws>, I would really not have a separate procedure for the documentation, both for type as well as for proc (and namespace, if that makes sense). Let the <ws>proc handle the dual calls to insert the webservice along with the documenation into the WSDB. Not so sure that having a different command to desribe the documentation afterwards is the way to go, but probably I am just too much an OpenACS guy.
I'm trying to maintain some civility here, but this is somewhat interesting way of looking at things. First a minor issue. The API is written <ws>*, there are no spaces. It is like constantly mispronouncing someone's name, on purpose due to laziness. Second, I'm not interested in how you think it should be done or where the information should be stored (I am interested in _what_ should be documented). Anything like this must first have a corresponding API in tWSDL. If you read my next-to-last post, I included a comment I put in a file when I first wrote TWiST: higher level APIs must use the tWSDL API. The meaning of this is that higher level APIs cannot mess with the internal storage of the WSDB. This forced use of the API helps identify deficiencies in tWSDL as well as hacks which need code cleanup. I have done several rounds of this already in tWSDL and TWiST.
Having a separate documentation API keeps the two functions separate. I can't imagine how it would be possible to not have massive duplication of code if every <ws>* API had to have its own documentation handler. I have already outlined the deficiencies with the ad_proc method. First is that I would have to duplicate what ad_proc does. But which ad_proc do I follow? There are several versions, and I doubt that we have seen the last version. Changes in version require redoing previous documentation blocks, or maintaining backwards compatibility. If similar to, but not quite the same as ad_proc, a documentation block will lead to inevitable confusion for those who use ad_proc and expect it to be identical. For those unfamiliar with ad_proc, they must learn how the documentation block works. Finally, <ws>proc does more than define a procedure. It defines simpleTypes, complexTypes, input and output messages and an operation.
But most important, and this is really important: I cannot think of a single use of documentation blocks in API calls. Does anyone call db_multirow with a documentation block? I guess you could call ad_page_contract an example. It is a great example of one procedure doing too many things. I'm sure the code for it is difficult to maintain. But just a clue to my method of API development: I write monolithic procs which contain too much code at first. I get things to work, and then I chop up the internals and produce single function APIs to support the original one. I don't keep adding functions to existing API unless it is 'functionally necessary'. Of course I don't always succeed, but it is a valid criticism for code which needs cleanup. Okay one more thing: documentation is data. I don't know if you can include documentation in the schema of a WSDL, but since the namespace of the schema is the xml schema, the rules are the same as that namespace. But even if the WSDL standard doesn't allow the inclusion of documentation in a particular place, this has no importance with respect to the decision to document, at least internally, particular components. The documentation inside the operation might contain a url to an html document, compiled from more complete internal documentation.
My original comment was that it would be nice if there was an independent OpenACS package which allowed you to access the documentation of an external package as if the APIs were part of OpenACS. For instance: Tcl, AOLserver, tDOM. Does anyone wrap these procs? If you could create a package for AOLserver, then the documentation for TWiST would also show up in the same place. Of course my inspect package shows the current state of the code and variables in that thread, but this is not documentation, it is for debugging and analysis.
In an ideal world I would be able to register all procedures with their names within the API Browser, so that searches for "string" would turn up the commands for the TCL string function, but this is not how the API Browser works *yet*.
Writing a package for OpenACS only makes sense once the API has stabilized though, which holds true for any wrapper functionalities.
I have to correct myself, if someone is interested in a tighter integration between tWSDL / TWiST and OpenACS then write procedures similar to the <ws>* procs in OpenACS "style" which make use of functionalities that only make sense if you have installed OpenACS. We have ample of examples where OpenACS wraps existing external API into it's own functionality blocs, like ad_return* or ad_proc .... So do not write a simple wrapper which does not add any new functionality except documentation, but I guess from my posting it was not obvious that this advantage comes in hand as well 😊.
However, if I were to write a quick 'integration' package for OpenACS, it would import the correct version (for the OpenACS package) from the google code repository. After that import, it can then be packaged up into an .apm file for distribution, or maybe the package would simply do the import upon installation. The package would then provide the TWiST API. The next step would be a wizard type interface which would produce a TWiST configuration file.
You know what, you are so confused about how this software works, and why it works the way that it does that I'm just going to pass on providing answers. This is typical troll behavior.
For anyone else interested, I'll provide a simple OpenACS package to this community and help maintain it. I have other improvements that are nearly done, and I'd rather spend my time doing those than answering list after list of garbage questions.
Anyway, nice to know that Malte has finally found the documentation, in the inconspicuous 'doc' package. I did provide a link to that about 36 hours ago, in my _first_ reponse to his questions. I also advised reading the docs, hopefully that advice will sink in a little. Good day.
(Maybe I should ask if there is any way to move Malte's off topic discussion (and my responses) to another thread so as not to confuse anyone else looking for real information?)
Now I am off to figure out how I can query a remote webservice. xotcl-soap?
You have no idea what you are talking about. Your assumption is that because an external tool is unable to figure it out that it is broken. This is similar to your mistaken fix for wsdlpull. Yes wsdlpull is broken in certain respects. But instead of trying to understand the 'bug', you simply changed the datatype so that wsdlpull no longer complained about it.
So the headsup: Malte is making unsubstantiated claims, he has no idea what he is talking about, and he provides zero information about what he is doing.
I ask again that these useless comments be moved to another thread, clearly labeled as misinformation.
Better yet, Malte: just move the discussion to your website, put up your modified code, change the name and have fun.
Btw, If you had provided more information prior to this, I might have been able to help. I actually did find a bug while tracking down the wsdlpull issue. One line of code was added, but wsdlpull still doesn't work. However, it will likely fix the Eclipse issue, but who knows since you have provided no information, just a headsup. So Malte, this is what I call hacking. You are hacking away and calling it fixing. Instead of reporting the behavior you see, you take credit for fixing something without understanding the issue.
Also, note that your headsup helps nobody. Which is why I have asked that your comments be removed to another thread. In the future, if you really want to improve the software, why not report information that would allow me to track down the issue? And if you report a fix, why not include it so that someone else can use it? If it is unimportant to you that everyone benefit from your fix, then why even mention it in the first place?
I think we need another thread:
Why TWiST Sucks, and What I'm Going to do to Fix It.
In the above definition you can see there exists an "xmlns:projectm". This is not automatically added by TWiST to the definition (::wsdl::definitions::new), but I manually ammended the wsdl::definitions::new procedure and added it there using:
lappend attributes "xmlns:projectm" "$targetNamespace"
right after
lappend attributes "xmlns:tns" "$targetNamespace"
My assumption would be that TWiST is adding the Namespace which I use (projectm), especially if used in the type name for the elements in a sequence, automatically in the definition. That is not the case when using the stock examples provided by Tom and cause Eclipse to break. After I inserted the code which resulted in the insert of xmlns:projectm="..." in the WSDL definition, Eclipse worked and the partners where able to access the webservices provided.
http://junom.com/ws/datetime/?WSDL
So what is the bug? TWiST creates a mapping of the xml namespace to the prefix of your choice. However, it fails to trim off the leading ::, it passes ::projectm to tWSDL when creating the map. That is probably the bug. However, tWSDL could just trim this, because the leading : are illegal. So in the future, if another user API is developed, if the same mistake is made, the mapping will still work.
Now, during writing the WSDL file, the targetNamespace is your xml namespace. When you include types which are in the targetNamespace, the prefix is replaced with 'tns'. If an incorrect map is used, the replacement doesn't take place.
In the future, multiple namespaces will be supported, and this will be handled by creating another schema in the wsdl file and importing it into the targetNamespace. But, the schema for the targetNamespace will always use either the tns prefix or the built in xml-schema namespace.
The wsdlpull issue is a problem with wsdlpull, not TWiST. The problem will go away with the next update. Hopefully very soon. wsdlpull does not understand renaming of simpleTypes, at least within the same namespace. The problem will go away because the renaming isn't necessary anymore. I plan, eventually, on moving to a directed graph construction to plan the order of writing out xml schemas, but an intermediate solution is on the way.
Your current fix with the lappend is not actually evil, just in the incorrect place. I think you can achieve the same effect by setting:
::wsdb::definitions::projectmServer::definitions::.ATTRS(xmlns:projectm)
to the targetNamespace. This will only affect the projectm wsdl.
You can also set the targetNamespace instead of accepting the default 'urn:tcl:yourns':
<ws>namespace init ::stock <ws>namespace schema ::stock "http://junom.com/stockquoter";
But a simple fix is here:
in wsdl/tcl/wsdl-schema-procs.tcl
in proc ::wsdl::schema::new, add a trim of schemaAlias:
if {!$initialized} { initDatabase } # trim schemaAlias for buggy user APIs set schemaAlias [string trim $schemaAlias :]
As for naming the schema using "<ws>namespace schema ::projectm "http://e4.cognovis.biz/twsdl/projectmanager" ", I remembered it not adding an additional xmlns::projectm="..." to the definitions, just setting the targetNamespace differently. (Look at the source code of http://e4.cognovis.biz/twsdl/wsapi/stockquoter/?WSDL which is using your example). Sadly this still is an issue Eclipse is choking at (but do not ask me why...). So, if you could add the first call (adding the Namespace as an xmlns: definition) when doing the second (defining a custom schema), I assume it would work. Though I think for most implementations the fix you provided should be just fine.
One thing in your posting caught my attention.
"wsdlpull does not understand renaming of simpleTypes, at least within the same namespace. The problem will go away because the renaming isn't necessary anymore."
I was wondering about that. If I use <ws>proc ::projectm:GetTask {task_id:integer}, you are creating the custom "simpleType" with the name "project_id" and the base="xsd:integer". When I (during testing) made sure that not the custom type but the original xsd:integer was used as the type in the xsd:element, it worked. Obviously that is not a solution if you want to have further restrictions like enumeration or patterns. But for strictly the simple types I don't think you need to create a custom simpleType, but again, I haven't read the specs thoroughly enough to say that is allowed by the standard. I can only speak for wsdlpull behaviour in this particular instance.
You are right, it doesn't fix the bug, it replaces the default targetNamespace with your choice. Note that a lot of defaults exist as variables in <ws>namespace.
The problem with wsdlpull is that there is another layer of derivation for the types which it isn't handling correctly by wsdlpull. I believe the schema is correct. One way to test is to remove the intermediate type and point directly to the global simpleType. I'm guessing, but I think that wsdlpull is interpreting the definition as empty:
<xsd:simpleType name="DayNumber"> <xsd:restriction base="tns:dayNumber"/> </xsd:simpleType>
As the developer of wsdlpull,somebody brought my attention to
this bug. as far as i understand its a bug in simple types.One such bug is caused by the creating another type DayNumber which is just a wrapper for "daynumber" note the case. Due to this the check for valid integers in the enumeration fails for the 3rd operation in the http://junom.com/ws/datetime/?WSDL because wsdlpull does not check validity recursively up the type tree yet. I am workin on fixing this but an immediate fix would be to remove the (redundant ,IMHO)wrapper
type "DayNumber". Feel free to contact me if you do have issues in wsdlpull
vivek Krishna mailto:krishna.vivek@gmail.com
Thanks for keeping track of this discussion. Personally I like wsdlpull, which is why I provided links to your software. I immediately recognized limitations for the built in client 'wsdl'. These limitations are to be expected in this field, especially with a simple example use of the API. The point is that the WSDL is correct, but individual clients may not take into account every conceivable representation of data types. This is not necessarily a bug for the client. In my opinion, XML Schema provides so many ways to represent the same structures, that it is unreasonable to expect independent client and server software to 'understand' each other.
I agree with Vivek that there is an unnecessary, although legal, layer (at the moment) in the type definitions of TWiST. This will be removed as I have stated before. But this fix will not make wsdlpull interoperate fully with TWiST. There will still be limitations with types based upon patterns and complexTypes beyond what a simple array could represent. My understanding is that the 'wsdl' client of wsdlpull is merely an example use of the underlying API. I like it, and it helps me check for certain errors. But it is unreasonable to expect that any client or server will ever support much of the XML Schema standard for type definition. There are a handful of companies investing billions into this problem with little success. XML Schema is something of a disaster in flexibility. In response, I have placed well defined limits on how types are represented in tWSDL.
As Vivek has mentioned, a tool in the cure is to represent types as a tree going back to the built in types. I wish to use a tree so that the XML Schema is as simple as possible, but this might also help the clients. At least clients can apply a rational and consistent process in reading the WSDL and developing the type validation code. However, as an intermediate step, or maybe a final step I have decided to flatten the type hierarchy. Every complexType is at a global level, allowing it to be reused as a child of other complexTypes. Elements with a simpleType content will be protected inside their parent elements, but their simpleType will be exposed as a global type in the case that the type is a derived type. But as I said, even this simplification will not guarantee that the wsdlpull client will interoperate in every case.
You always have the option to write to the OCT at oct "" openacs.org if you find comments made offensive and would like them to be removed from the forum, or moved or labelled. I actually pondered about asking for that for some of your comments, but maybe this is just your communication style.
Other than that I am delighted to read that my comments do have an impact on TWiST, after all it seems you did find an outstanding issue. And despite dimissing me as stupid you actually take the time to look into things 😊. As you seem to prefer me not to give any feedback on TWiST, I will duely respect your wishes. After all the feedback was supposed to help you to improve the software, not to help you insult me.
On the other hand, I'm very receptive to anything indicating a bug. I think the main issue here is fairly simple: TWiST and tWSDL are for machine communication. This is vastly different than human communication. A guiding principle is to make things as simple as possible for clients (software), without regard to humans. But by simple, I don't mean accommodating. The best starting point for that is the WS-I Basic Profile. However, tWSDL has limitations above and beyond those outlined in that document.
OpenACS packages tend to be standardized for another purpose: pleasing humans. I chose to please the machines first, because even though I can argue with humans, machines refuse to even respond if they are not addressed correctly. I also have to please myself. I would rather TWiST do one thing very well, without errors, than to take on another layer of development too soon. I'm also starting to think that any OpenACS package should start over at the same level as TWiST, not above it. Then the OpenACS API could amount to a graphical interface to develop and deploy and monitor web services. Essentially you would have one page per equivalent TWiST API and use the guts of this API as a script on a Tcl page, with minor changes.
<?xml version="1.0" encoding="utf-8"?>
<soap-env:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsd="http://www.w3.org/2001/XMLSchema">
<soap-env:Body>
<GetProjectResponse xmlns="urn:tcl:projectm">
<Title>Electric Fry</Title>
<Description>The smaller kitchen of the "Vier Jahreszeiten" restaurant
needs an electric fry<br /></Description>
...
</GetProjectResponse>
</soap-env:Body>
</soap-env:Envelope>
See the "<br />" tag.
Using a different tool for testing I got "exception: org.xml.sax.SAXException: SimpleDeserializer encountered a child element, which is NOT expected, in something it was trying to deserialize.", while WSDLPULL complains "An Exception occurred ...@7:7 Expected a closing tag for Description Error validating schema instance".
I solved it for my application by returning [ad_quotehtml $description] instead of just $description as earlier. If this is universally true, then it would be an enhancement if TWiST would do that work for me. If it is not, then obviously you should not do it.
This question is like asking why your application doesn't quote the description before returning it to TWiST, this is a programmer's decision when crossing the boundary of an application. TWiST takes the data you give it, assumes that you have given it the correct data, in the format you have chosen, and then handles it.
BTW, this issue provides a perfect example of why I setup TWiST as a 'configuration', not as an on/off switch for a proc like you get with .NET. This provides for maximum code reuse.
Imagine if you had to go back and define another internal API just to quote some output so TWiST could use it. The TWiST API <ws>proc is designed to allow you to call your internal API, do what you need to the normal inputs or outputs, and return the result in a format of your choice, completely independently of the exact internal API signature.
This idea should become very useful for procedures defined via ad_proc, where there are optional switches and the order of attributes isn't well defined. You can use <ws>proc to handle common defaults, and to prevent complete exposure of the internal API. For instance, you would not want to allow an application to choose the current user_id without validation. (OOB validation isn't handled by TWiST since it is OOB and application dependent.)
Also, you asked earlier if you could validate an output prior to sending. You do this inside of a <ws>proc.
There is no built in way to do this because the server should know that it is constructing a valid document. But it can be done inside the <ws>proc.
Once you have the data ready, feed the same thing you would return from <ws>proc (a list), to create a return message, then feed this reference to the validation routine. If you get an invalid result, you should return a server error (500).
Offline, you can do the same thing. Call your newly defined <ws>proc (in your new namespace) to get the return value list, create your XML message document, and then validate. This setup skips all the SOAP stuff, network interaction, etc. This is also a good place to do your unit tests.
If you have examples of what you are trying to do, it would benefit everyone if you would post the TWiST configuration, or parts of it. Real world examples are the best way to explain how things should work, and, as I said, they benefit everyone.
http://e4.cognovis.biz/twsdl/projectmanager/index.txt.
I can post the whole examples, but this is going to grow, so a link is probably better.
With regards to your answers, I understand your idea, my logic was simply: If the standard (WSDL 1.1) does not allow it, then it would be great if TWiST "catches" it, fix it (in the case of HTML within the returned value) or give a server error, without the developer having to do manual catching and checking within the <ws> proc.
With regards to the on/off switch of a procedure, I understand why you would like TWiST to be more flexible and I like it that way. Yet my projectm::GetTask is a perfect example where I as the user would prefer a simple way to have TWiST take the procedure, plug it in it's pieces and return them accordingly. But that might as well be an extension written purely for OpenACS, as we do have more information about the procedures (or could retrieve it from the (proper) documentation.
Maybe this clarifies my point that from a user perspective I would like to have a couple more things taken care for me. But I completely understand if you prefer that not to be part of TWiST as it would limit the configurability.
"for a type "float" it's not possible to have an empty value, set to 0 instead
<EstimatedHours></EstimatedHours>
put:
<EstimatedHours>0</EstimatedHours>"
Not sure if this is according to standard WSDL or application specific on the other's side. I can put all of that on a separate page for other developers what I had to keep in mind when dealing with WSDL in general and TWiST in particular, as I am sure I will miss a couple of more things.
Or you can define a new type which allows the empty string. All built in numeric types in TWiST and tWSDL do not allow an empty string (like string is -strict), for obvious reasons.
So to be clear: there is no way to "catch" this. Do you even remember the 'noquote' discussions which have taken place over the years right here on OpenACS? This is exactly the same issue.
Another mistake you are making is your belief that ad_proc somehow has more information about types, especially _return types_, than TWiST, especially with regard to the requirements of WSDL/SOAP. Data typing, construction and validation are way beyond what is available in OpenACS, it has exact information on every type. It can validate every type it handles. It can construct documents of a given type. But Tcl itself does not carry type information. There is no way to tell if incoming data has been quoted once, twice or never. Maybe this is available in some language, but not Tcl.
TWiST and WSDL are about 'interoperability', not 'bells and whistles', especially cracked bells and broken whistles.
It is somewhat amazing that you fixed this for yourself in exactly the correct way, by adding the typical method for handling this, and yet that isn't enough for you. You shouldn't be bothered to think at all. You wish for TWiST to use magic to guess what you want it to do.
Maybe another example: XML or XHTML. The more you restrict the functionality of a component, the more generally useful it becomes. The software becomes very simple when it doesn't have to guess at what the humans wanted to do, but couldn't figure out how to do. TWiST isn't smart, and that is simple. You are the one which has to be smart.
Here's an idea: if OpenACS truly has all the information you need to do this, then there is no issue at all, simply write your procedure to return what you want! Or, add a quote flag to your own internal API if it doesn't have this information, then use the flag inside the TWiST <ws>proc to return the correctly quoted text, or optionally a CDATA wrapped, but unquoted text.
I apologize for my tone, but you keep bring up examples of your own failings and blaming them on TWiST. If you want my tone to change, look a little deeper into the issue prior to revealing your thoughts. That's my feedback to you. TWiST is not a cure for not thinking.
I've already come up with the title of the thread you need to start about your improvements to TWiST (Why TWiST Sucks, and What I'm Going to do to Fix It). I promise not to complain in that thread about your ideas about your software. But can we stay on topic here: I'm not soliciting feedback on improvements, I have a whole list of them myself that are first in line. I'm very interested in bugs and if you give enough details I can fix them, or offer a temporary work-around.
Additionally, how could I do this for a complexType as described at http://www.smrtx.com/RS/xsd_documentation.htm
Otherwise, you can save the wsdl and put in your own documentation tags. Please start your other thread of your proposed improvements to TWiST, so I don't have to respond to them. This is off topic.
And I'll offer my services as bookie for anyone interested in laying odds on the outcome!
Tom, lighten up a bit. Open criticism, deserved or not, is part and parcel of the open source world.
It seems that you think I should just allow Malte to make whatever comments he wants without response. He can comment all he likes. If I find his reasoning suspect, why can't I comment on that? The logic of a criticism is even more important than the exact substance because it applies to other specific cases.
Malte's first criticism was that TWiST didn't use ad_proc. Somehow he misses the fact that TWiST is not an OpenACS package. Also missed is the fact that another organization paid for the development of TWiST. They use a different version of ad_proc. I argued against making TWiST specific for their software, _exactly so_ that it could be immediately used by OpenACS and any version of ad_proc. But the fact that I chose to write an API which would be useful to everyone is completely lost on Malte.
But simply saying that it would be nice if the software used ad_proc is one thing. Comments are just comments. But Malte suggested in his first post that the situation be immediately corrected. The TWiST module should be converted to use ad_proc, and the result should be imported into the OpenACS CVS repository. Do you consider this criticism? Don't you have module owners here at OpenACS? Even if the software itself is open source, even if it doesn't belong to me, or the organization which paid for the development, is there any respect _at all_ for the work of those who decided to provide this software?
Later on we find that these initial criticisms were made _prior to_ correct installation. In fact many comments were made prior to understanding the subject of the comment.
Believe me, I know that there are many valid criticisms of this software. I have asked for them. I'm waiting for them; I have a whole list of my own as well. But the standard for criticism should at least require that they enlighten those wishing to use the software. If a criticism amounts to misinformation, or suspect reasoning, or is the result of simply not understanding what is going on, does that help anyone? I submit that it simply calls into question the usefulness of the software for others who might be interested, but will get the incorrect idea that the software has functional problems.
Another side effect of this might be that other developers, not wishing to subject their software to Malte's criticisms will simply not release it. Who needs to spend their time defending unfounded or misguided comments when they can be improving the product? Additionally, I risk suppressing helpful comments and questions when I respond. I thought about that prior to my responses. If Malte was not such a respected and long standing member of the OpenACS community, there would be little need to brush off his comments. But that is not the case, and I decided to take the risk.
I encourage open criticism, and open criticism of that open criticism, etc. I also encourage understanding the topic you wish to criticize. It is much more effective.
It seems that you think I should just allow Malte to make whatever comments he wants without response.Well, no, actually, I just said ... lighten up. It's just a suggestion. Life's short, after all.
No, it is not related. I knew of this use of the acronym as I was choosing a name. Note that there is a small 'i' in the tWSDL wrapper TWiST.
The TWIST standards are a very nice example of a (somewhat) open source data model for business processes represented as XML documents (messages). I'm not sure if there is an associated WSDL or not, but the messages could be reused in any client server application.
TWiST could be used to represent the types in TWIST, and to read and validate messages. But currently it can't directly create new messages using the high level API.