Forum OpenACS Development: TWiST OpenACS package

Collapse
Posted by Tom Jackson on
I've completed a number of improvements to TWiST, including a wrapper OpenACS package. The package is available, at least for now at:

http://junom.com/openacs/twist-0.9.3.apm

The source code is not included in the package. Instead, the specified version of TWiST is downloaded from Google Projects. This make upgrading relatively simple: change the version parameter in the Application instance and restart the server.

If subversion is not available, the source code can be moved into place by hand. Any error message during install should indicate what svn command to execute on a different server, and where the current configuration will look for the files.

Users can optionally link in the example web services and the documentation by setting the link* parameters.

TWiST Improvements:

<ws>doc allows documentation of types, elements and operations. This wraps the already existing tWSDL documentation procs. Documentation does not show up in the WSDL file at the moment.

One problem with displaying documentation of a running service is that services are dynamically generated on a per-thread basis. If a service configuration file hasn't been visited by a thread, there is nothing to show. I solved this problem by showing documentation below the list of operations. All of the auto-generated code for a service is displayed, or linked to. This feature can be turned off on a per service basis, but the current default is to show documentation. See an example here:

http://junom.com/ws/stockquoter2/

As shown with the stockquoter2 example, <ws>element now supports both simpleType and complexType children, and each child can occur 0 or more times. This removes the limitation of only children of simpleTypes occurring 0 or 1 times.

As an interesting coincidence, the web form interface which allows testing, can take a space separated list for any element which allows multiple occurrences. These are converted into individual elements before being POSTed. This is a consequence that if maxOccurs is greater than 1, the input data is considered to be a Tcl list.

In addition, I have updated this interface to allow omitting optional elements (by leaving the text field blank). Since the request document is generated using the 'new' procedure for the element, the default values are filled in, but SOAP clients no longer have to send optional elements. One additional note: when a complexType is included as a child of another complexType, a default can also be included. The default is interpreted as a tcl list. This is simply a consequence of the API.

Collapse
2: Re: TWiST OpenACS package (response to 1)
Posted by Tom Jackson on
One important change that I forgot to mention is a change in how to specify a <ws>proc. Anyone upgrading will need to make a few changes to their configuration files. The change is in the input and output types. In the original release, each input parameter (proc args) was limited to zero or one. This exactly matched the Tcl proc signature, and the only possibility was that if a parameter was optional, you had to provide a default value. Either one implied the other. So this argument list looked identical to a Tcl procedure arg list, with the addition of a type specifier stuck on to the parameter name. The new features required that this format change to be the same format as used in <ws>element and in the return spec in <ws>proc.

Here is an example:

<ws>proc ::stock2::Stock {
    Symbol:stock2::symbol
    {Verbose:stock2::verbose "1"}
} {
  ...
} returns { ... }

Is now formatted like this:

<ws>proc ::stock2::Stock {
    {Symbol:stock2::symbol}
    {Verbose:stock2::verbose {minOccurs 0 default "1"}}
} {
  ...
} returns { ... }

The first list element is unchanged with format "Name:typeSpec", and at least the name is required here. The second element is any additional information as a name value list, exactly what would be used to create a tcl array. Unless otherwise listed in this list, minOccurs = 1, maxOccurs >= minOccurs > 0, no default value and nillable is not set. So the following are equivalent:

{OptionalArg {default "abc"}}
and
{OptionalArg {minOccurs 0 maxOccurs 1 default "abc" nillable false}}

This change was pushed all the way through to tWSDL which used a different format. When TWiST was being developed, the complexity of the original format was discarded, and the result was a lot of additional code in TWiST to convert to the old format. Now all the logic has been pushed into a small procedure, greatly simplifying the TWiST code, and providing a single data structure to maintain.

Collapse
3: Re: TWiST OpenACS package (response to 1)
Posted by Claudio Pasolini on
Hi Tom,

Installing TWiST on my OpenACS 5.2 with Aolserver 4.5 I got the following error from packages-install-4.tcl:

Failed to install TWiST, version 0.9.3. The following error was generated:

Database operation "0or1row" failed (exception NSDB, "Query was not a statement returning rows.")
ERROR: -20000: The specified package 0 AND/OR parameter initialized do not exist in the system
SQL: select apm__set_value( '0', 'initialized', '1]' );

Are there any specific requirements to install?

Collapse
4: Re: TWiST OpenACS package (response to 3)
Posted by Tom Jackson on
I hope someone can answer this question, but my guess is that the parameters are not added to the database before the tcl scripts (twist-procs.tcl and twist-init.tcl) are run.

Oh, wait! the default appears to be '1]', I'll see where that shows up in the source.

Collapse
6: Re: TWiST OpenACS package (response to 4)
Posted by Malte Sussdorff on
The problem does not lie in the 1, it lies in the fact that you run "init" in the before-install block. As the name implies, it is executed "before install", meaning the parameters are not yet written into the database as the package is not yet installed. Remove it and you are fine at this point :)

Furthermore you need to have SVN installed, which is not a good thing to require. Why not provide it at a download location (url) where one can download a .tar.gz?

After solving this riddle I hit the road again with the init in the after_install and here I am not sure why the parameter is not yet installed, but well. Do we really need it there? After all you are restarting the server after an install of a package anyway, and init runs upon server start, so ....

Just a minor thing though. As per convention the callback procedures (to which before_install and after_install belong) are put into a file $package_name-callback-procs.tcl or apm-callback-procs.tcl (new / old convention). But this is not documented in the developers guide, I think 😊.

Collapse
8: Re: TWiST OpenACS package (response to 6)
Posted by Tom Jackson on
There is a new version up at http://junom.com/openacs/ (0.9.3.2).

I finally figured it out, I hope. Before-install means that a Tcl proc is run prior to loading any database code. After-install means after the database code has been run. Both seem to run before APM considers the package to be installed, so it is before there is a package_id, so you can't use parameter::set... in either of these.

But there is another issue as well: I was thinking that the package parameters would be initially taken from the twist.info file, but they are not. This means that if you don't like the defaults (except the mount point, which works), you need to change them in twist-procs.tcl in the proc ::twist::install::init.

Although the normal install requires subversion, it isn't necessary. I don't have subversion on the server where I created the package; on the other hand, I don't have wget on my local machine, so coming up with an 'always sure to work' solution may not be possible. Here is the reasoning behind the way the package works:

If you want to upgrade TWiST, go to the application instance and change the version parameter. Restart your server. The old version remains in place, and you can revert back to it if there is some issue with the new version, just change the version number back and restart the server. You can remove the old version if you wish. The upgrade of TWiST version doesn't affect the OpenACS package.

The new OpenACS package removes the 'initialized' parameter and replaces it with an 'installed' parameter. The default in twist.info is '1', but until the package is installed, the 'init' procedure sets it to '0'. Once this becomes '1', which means that the package is installed, the sourceDirectory parameter is updated with the Tcl default. This parameter is set in Tcl code to [ns_info tcllib], so if you want the TWiST source somewhere else, you have to edit the Tcl code prior to installation, or change the sourceDirectory parameter and restart the server, after installation. This will force a re-download of the code using svn ... or just move it by hand into place and restart.

Collapse
5: Re: TWiST OpenACS package (response to 3)
Posted by Tom Jackson on
Claudio,

I've added an update to http://junom.com/openacs/

I uninstalled the package from my OpenACS and tried it from scratch, 0.9.3.1. I get a different error, but it installed anyway, something about a missing url, like set_url? You might need to restart the server, but the error you found could prevent it from ever loading. Note that I don't have svn on my server, so I had to place the files in [ns_info tcllib]/twist/twist-$version/, but I have tested the download code on my laptop. These errors are unrelated. Maybe my apm is out of date?

I'm still not sure exactly how this works. I went to the list of applications for the main site, after the restart and added TWiST. Is this necessary? But it also added it at the url /twist-2/.

One way to test the installation is to see if the documentation is showing up under /twist/doc/, or whatever directory it ends up at, plus doc/.

I'll be out for a few hours, but back later this evening.

Collapse
7: Re: TWiST OpenACS package (response to 5)
Posted by Claudio Pasolini on
Following Malte's suggestion and downloading your updated TWiST version I managed to install the package, even if I got another error (that I forgot to annotate).

As a result the package was created and the twist code was correctly checked out by subversion, but:

  • the package was not auto mounted on the site map
  • the links to the documentation and to the example web services were missing, even if I had set the corresponding parameters to "1" before installing
  • the parameter "sourceDirectory" was empty
I manually mounted the package, executed the proc ::twist::install::after-install to create the missing links and set the parameter "sourceDirectory" (not sure if it is necessary) and restarted the server.

Now all is working fine, but naturally only after modifying the config.tcl adding the line:

    ns_param   twsdl              tcl
to the list of Aolserver modules: perhaps you could add a simple reminder about this into the package /doc.

Thank you very much for your work on complex types and the examples provided!

Collapse
9: Re: TWiST OpenACS package (response to 7)
Posted by Tom Jackson on
There seemed to be two causes for errors:

First I tried to use parameter::set... before the package was installed. The bottom line result is that you can't access the parameter defaults in the twist.info file prior to the package being installed. I finally figured out how to use this to my advantage.

Second: a mount point like '/twist/' was rejected, I think due to the leading '/'. Changing it to 'twist' seemed fo fix the error.

Third: code in twist-init.tcl should not run until the package is installed. The installed and upgradeNow parameters default to 0 before installation, so this code doesn't run until you restart the server.

Claudio,

You should not have to modify your config.tcl file, the TWiST code is sourced by the twist-init.tcl file during startup.

If this change actually works at all, it would not be good, because the twist source should be at tcl/twist/twist-$version/. If AOLserver is finding and executing subdirectories of tcl/twist, then the next time you upgrade twist, both versions will execute. Hopefully your install is just working as expected and not because of the change to config.tcl.

If you decide to test the new package, you might need to delete the site-node created. At least once in my testing I ended up with twist-2/ as the auto-mount point.

I am still working to improve this so it is easier to install and upgrade, and follows OpenACS conventions.

So far I have these suggestions (some my own):

1. alternative download option, using http/tar.gz. Which client should I standardize on?

2. maybe move the install/callback code to other files.

3. provide some OpenACS package specific documentation.

4. make it easier to setup defaults prior to package installation.

5. examples using the OpenACS API.

6. some method for discovering or registering twist configuration files.

Collapse
10: Re: TWiST OpenACS package (response to 1)
Posted by Tom Jackson on
I think the basic installation code for the TWiST package is stablized, thanks for enduring the process, and let me know if there are still problems, I'm sure that there are some.

I started working on examples using the OpenACS API, and the first thing I ran into is that the testing client needed to copy the cookies sent by the user's browser. So I needed to update <ws>return to do the copy. This works for all versions of AOLserver and OpenACS. This also allows services to be moved into the admin directory, at least until the security measures are put in place. The updated version tag is twist-0.9.4, and the OpenACS package is at http://junom.com/openacs/twist-0.9.4.0.apm

The first examples are posted at http://junom.com/openacs/testoacs.txt

Collapse
Posted by Claudio Pasolini on
Hi Tom,

I'm trying to build a TWiST service using the new API that you provided to deal with complex types and I've posted for reference the code at http://demo01.vestasoft.com:8001/twistsample.txt and the responses at http://demo01.vestasoft.com:8001/twistresponses.txt or you can try them live at http://demo01.vestasoft.com:8001/twist/ws/parties

The first operation parties::PartiesQuery mimicks your stock2::Stocks example and returns the complex type parties::OneParty made up of a fiew simple types. Actually, given an input request that specifies a party type, the operation queries a table and returns some attributes for every party of the given type.

With the second operation parties::PartiesQuery2 I'd like to return the complex type parties::Result made up of the simple type parties::partiesNumber, wich holds the number of parties retrieved, and the already existing complex type parties::OneParty.

The response contains only the first party of the list, ignoring the others and I don't understand if this depends from the way I constructed the returnList within the proc or from some other reasons.

I tried several variations, one of wich is parties::PartiesQuery3 where I built the resultList in a
different way, but then the system complains about "Missing value for required Element CreationDate with no default value calling ::namespace inscope
::wsdb::elements::parties::OneParty::CreationDate new".

What I'd like to do next is to create a service that accepts a parties::Parties complex type made up of a sequence of
parties:OneParty and returns a list of party_id after creating them. Do you think that this is possible with TWiST? I tried to define {OneParty:elements::parties::OneParty} as the argument of the proc, but the system complains about not finding the corresponding ::validate proc.

A last question about the automatically generated names of the Request and Response elements: do you think that they could be parameterized?

TIA,
Claudio

Collapse
Posted by Tom Jackson on
Claudio,

You want to return a list, the first element of the list is your first child element partiesNumber, the second element of the list is your list of parties:

    return [list $partiesNumber $partiesList]

Also, on your return type, you have two colons after the name, I haven't checked if this matters or not, or if it should matter or not:

{PartiesNumber::parties::partiesNumber}

About parameterizing the names of the request and response elements: yes this needs to be dealt with. There is a slightly larger problem to fix or clarify in this area, I think the issue is that you can pre-create request/response types if you name them ${procName}Request and ${procName}Response, but by pre-creating them, they are not again created when you use <ws>proc, although you can do things like change the default value from what was specified in the complexType. In tWSDL, you can name simpleTypes, complexTypes, messages, operations, ports, etc. all with the same name without any problem. However, it is sometimes confusing to track down errors if you don't put a trailing chunk of text which helps identify what the name applies to.

Anyway, I had thought of parameterizing these, and I believe I put some namespace variables in the <ws>namespace to use, but then I didn't use them. So I'll look at that to see if it can be quickly fixed.

Collapse
Posted by Tom Jackson on
BTW, after considering the potential complexity handling inputs of greater depth than list_of_lists, or more complex outputs, I think the next addition to TWiST will be <ws>operation. This will allow the developer direct access to writing the 'Invoke' command for the operation, which takes an inputXMLNS (where to find the input document) and fills an outputXMLNS (with the return document), and returns a reference to the documentElement. You could use this, for instance, to send all the data for an invoice and process the different complexTypes involved in a single transaction. (You can do this now by writing a proc and setting a variable pointing to it, but not using the TWiST API.)
Collapse
Posted by Tom Jackson on
Claudio,

The new version of TWiST is 0.9.5, hopefully if you are using the OpenACS wrapper, you can update by changing the version number and restarting OpenACS. The older version will remain in place, and you can switch back if you find a bug.

Here is what the new version adds: with <ws>proc, the ability to use a complexType element name in place of the procArgsList and/or the returnList.

Example (from stockquoter):

# Create input element complexType
<ws>element sequence stock::StocksToQuote {
    {Symbol:stockquoter::symbol {maxOccurs 8 default "MSFT"}}
    {Verbose:stockquoter::verbose {minOccurs 0 default "1"}}
}

# Create output element complexType <ws>element sequence stock::StocksQuoted { {StockResponse:elements::stock::StockResponse {maxOccurs 8}} }

# Use complexType element name in place of lists <ws>proc ::stock::Stocks { StocksToQuote } { set resultList [list] foreach symbol $Symbol { lappend resultList [Stock $symbol $Verbose] } return $resultList } returns StocksQuoted

The result is that the user now has control of the names of the input and output elements.

Collapse
Posted by Claudio Pasolini on
Tom,

the upgrade to version 0.9.5 worked very well and the ability to use an arbitrary complexType element name as input/output of the procs is very useful.

For a number of simple web services this is enough, while as you already pointed out we need something as the announced <ws>operation to deal with more complex cases.