Forum OpenACS Q&A: SOAP Gateway Package ready

Collapse
Posted by William Byrne on
You can fetch the soap-gateway package at:

http://www.byrnelitho.com/SOAP-Gateway-0.1d.apm

Direct install works ok.

The package should sit well on top of a v4.6 install.

The package features:
 - basic SOAP/HTTP 
 - implements security
 - provides a 'published methods' paradigm
 - management for multiple service categories
 - publishes WSDL in multiple formats
    - include/exclude documentation
    - force return values on one-ways (helps AXIS)
    - endpoint rerouting for trace/debug
 - grafts ad_proc docs into published method docs
 - restricts method entry points to ::sg::* namespace
 - includes a few demos: MSSOAP Toolkit & AXIS
 - method binding analysis (diff)

Collapse
Posted by Ola Hansson on
William,

Did you use the package manager to create the .apm?

I ask because there seems to be a little problem when uncompressing it via "/acs-admin/apm/package-load":


Accessing /tmp/SOAP-Gateway-0.1d.apm...

      The follow error occured during the uncompression process:

child process exited abnormally

Collapse
Posted by Ola Hansson on
William,

Pardon me. It was just a silly permissions thing...

I don't know much about SOAP but it installs on 4.7 HEAD and looks real cool and it has documentation, too!

I shall go back to play with it. Thanks for your work.

Collapse
Posted by Ola Hansson on
In soap-gateway-create.sql I had to change "timestamp" to "timestamptz" in the param declaration for sg_namespace__new() and sg_method__new() to be able to import the "interop" and "workspace" services in my PG 7.3.1 installation of OpenACS 4.7.

This might break support for PG 7.2*, though, but I don't know for sure.

I also get:

'Request Error

can't read "diffdata": no such variable'

...when I follow the "Create new namespace" link.

The code says something about "debug"...

/Ola

Collapse
Posted by William Byrne on
Ola, thanks for the early input. I made corrections for the issues you described. Timestamptz took without problems. For now, I removed the link to create a service namespace. Since each WSDL/service/namespace relies heavily on a tcl source code, it was virtually useless to expose the feature at this point.

If you wish to create your own service, take a look at the supplied services 'Workspace', 'Interop', and 'Demo' to see what you need to do. You can park your tcl source files anywhere the server can source files. You simply need to use a namespace like ::sg::myservice and declare your methods with -public access.

Collapse
Posted by Dan Wickstrom on
William, this looks nice.

This package seems to be configured only for supporting the server end of a soap connection, and I don't see how one goes about generating the client stubs needed for invoking a service?  Are there some utilities for generating the client stubs, or do I need to do this part manually?  I haven't looked through the whole package, so I may be missing it.

Does the IDL only support basic types like int, float, string, etc?  If so, how difficult would it be to extend the IDL to support more complex types?

Overall, the package looks to be very nicely done.

Collapse
Posted by William Byrne on
>>Are there some utilities for generating the client stubs
You bet, there are plenty. From within SG, go to Help and navigate to the references section where I list a few. The SG package also includes client demos for MSSOAP Toolkit and AXIS which is a java based solution. You'll find them under 'Tests'. I'm waiting for Don Baccus to chime in and make reference to TclSOAP<g>. As he's correctly pointed out, it's a viable solution for thing related to tcl and soap. As far as client stuff goes in SG, I thought about it. Something will eventually surface. I must admin, all client connectivity was performed from a win32 box (hmmmm, testing interoperability to my linux server from win32 - pretty cool.) Server to server connectivity would probably be helpful for scaled implementations. Despite a potential SG client solution, I'd encourage other types of client implementations since that's what SOAP's all about.

Concerning complex types, I'll be handling that within the near future. I've been through complex types in other SOAP services I manage; therefore, I'm not concerned about its complexity.

Thanks for the compliments. I hope the package provides the connectivity we're hoping for. My personal goal was to get automated access to the ecommerce package. I'm looking forward to getting there.

Collapse
Posted by Don Baccus on
BTW unfortunately timestamptz is not recognized in PG 7.2.  This is a minor but nagging instance of backwards compatibility being broken by PG 7.3.

Which remind me ... I bet the work that was done to get 4.6 to install under PG 7.3 totally ignored the timestamp/timestamptz issues.  I bet none of the time manipulation code in acs-reference works unless the datamodel is modified to use timestamptz.

Someone's going to have to sit down and evaluate the impact of timestamp no longer carrying timezone information before we support PG 7.3.  We may be stuck dropping PG 7.2 support for 4.7.

Collapse
Posted by Dan Wickstrom on
I was thinking of the case for server-to-server communication,  and for such a case, having the ablity to generate the client stubs would be a big win in simplifying the development process for soap-based services.

For example, I've been working on a news-feed package that  uses soap services to transfer news items from a news instance on one server to news instance on another server. Basically it is just a specialized service for copying CR items from one server to another.  To develop the soap service, I prototyped it using my web-services package (based on nsjava and apache-axis), and then when I was happy with the design, I converted the soap calls to tcl.  In the tcl implmentation, I store message templates with placeholders for interpolating the data necessary for creating the soap message, and then on the receiving end, I use ns_xml to parse the received messages. The soap messages are parsed by a specialized request processor that is hard-coded for the news-feed service.  The tcl implementation is lightweight and, other than the network, there is no perceptible delay.

Having your package would have saved me some work in writing the soap request processor, but I would still be left with quite a bit of work for creating the client stubs.  As you probably already know, the axis implementation allows you to write a java interface class and use a utility to generate a wsdl file.  Then, there also is another utility that takes the wsdl file as an input and generates all of the client stubs and server skeleton code based on the wsdl description.  So all a developer has to do, is write a java interface class, generate the wsdl, client stubs and server skeleton code, and the fill in the server skeleton implementation to create a web-service.

For your package, the approach seems to be to write the server implementation utilizing @Idl tags in the document section of ad_proc to describe the interface, and then using information from those tags you generate a wsdl description that soap toolkits can use to generate the client stubs.  Possibly tcl-based client stubs could be generated by tclsoap from the wsdl description?  Has anybody looked into this?

I'm thinking that the server-to-server type applications will proabaly be the most valuable, so if you can add support for generating client stubs and complex types, your  soap gateway package will become really useful for building openacs-based web services.

Collapse
Posted by William Byrne on
Support for the Tcl client stubs is an interesting one in terms of persistence and dynamic loading of the stubs. And, there's the cost of generating the stubs and who'll do the work and how often (client or server.) Certainly, the code implementation for stub generation is nothing more than an excercise. In light of your level of interest and the benefits you point out, I'll give the issue greater attention.

For now, if you're interested in the soap-gateway package, I recommend using whatever techniques you now employ to generate your stubs. Most likely, I'll provide another parameter in the WSDL fetch that allows a developer to specify stub generation. I've done this for C++ in another project and I can certainly do it for this one as well. The first blast would target the OpenACS/ns_xml/tcl combo. Upon fetching the WSDL generated code, you'd park it somewhere in your OpenACS code base. I'll extend the SG's source library management pages to include stub management.

Out of curiosity, what flavor of complex types do you require? Basic arrays, structures, arbitrary ranges within multi-dimensional arrays....

Ah, and there's still the issue of attachments...  what are your thoughts on the DIME spec?

Collapse
Posted by Don Baccus on
Talk to Bart Teuwisse about tDOM.  Our plan is to replace ns_xml with it and Bart's already done so for one private project and has volunteered to do the work to make the switch for OpenACS 4.7.

It's not much work to incorporate tDOM - the author has kindly provided, and supports, an AOLserver module that exposes it.

Collapse
Posted by Dan Wickstrom on
Just being able to generate the stubs would be nice, but having Stub management would be a plus.

As far as complex types, I'm currently using arrays of structures (java beans), but I'm sure that having other complex types like multi-dimensional arrays and structures that contain nested structures would also be useful.

I haven't used attachments, other than to try a few attachment based demos, but I definitely think they would be worth having.

With regards to the use of tDom, using the xml abstraction procs in openacs would probably be the way to go for now, so you don't have to rewrite everything when openacs goes from ns_xml to tDom.

Collapse
Posted by William Byrne on
Funny you should mention XML abstraction. I almost created an abstraction in SG when I rolled back the ns_xml version from 1.5 to 1.4 a couple of days ago. So, there's a XML abstraction somewhere?

I'd be more than happy to use tDOM. I noticed it delivers namespace info which ns_xml didn't do very well.

I agree Bart would be a good person to contact since this entire SOAP Gateway package began with the need to connect to the ecommerce package.

Collapse
Posted by Robert Locke on
This sounds great!  Quick question though: should this be uploaded to openacs.org/new-file-storage?

This seems to be the de facto location to place packages which haven't yet been blessed for inclusion in the official distribution.  Though it's not the best solution, it seems to be the current one and it does make it easier for users to locate "other" packages.

On a more general note, we need to think of a way to better organize user-contributed modules.  Any ideas?  (Maybe a question for a new thread.)

Collapse
Posted by William Byrne on
I suspect the implementation will undergo some review. If it makes the grade, I imagine I'll be contacted.

Gee, I'm beginning to feel a bit vulnerable<g>.

Collapse
Posted by Don Baccus on
Actually this has been discussed ... I have a package for generating certain types of test data that I used when investigating scalability of permissions that I'll be adding.

To do this, I'll make a contrib/packages subdirectory to the openacs-4 project.

We will gingerly start opening this up as we start thinking about criteria for inclusion of user-contributed packages (we want *some* level of quality assurance, don't we???)

Anyway ... the SOAP stuff's extremely important and as William's stuff gets tested, vetted, polished etc I can't imagine it won't go into our main CVS tree.  Assuming he wants it to, of course!

Collapse
Posted by William Byrne on
Don,

  I'm interested in including the SOAP Gateway into the OpenACS distribution. Though the project initially meant to solve my problems, I was hoping to contribute and maintain the SOAP Gateway within the world of the OpenACS community.
Collapse
Posted by Bruno Mattarollo on

Hello William,

First, thanks a lot for contributing this package...

I downloaded this version, installed the APM and mounted the application on a folder called soap. When I try to access the service, I get this error:

06/May/2003:14:46:42][742.45288448][-sched:16-] Notice: Running scheduled proc sg_boot_libraries...
[06/May/2003:14:46:42][742.45288448][-sched:16-] Debug: NO FULLQUERY FOR dbqd.soap-gateway.tcl.soap-gateway-db-procs.sg_library_get_paths. --> using default SQL
[06/May/2003:14:46:42][742.45288448][-sched:16-] Notice: Querying '
                select path
                from sg_libraries;'
[06/May/2003:14:46:42][742.45288448][-sched:16-] Error: Ns_PgExec: result status: 7 message: ERROR:  Relation "sg_libraries" does not exist

[06/May/2003:14:46:42][742.45288448][-sched:16-] Error: dbinit: error(localhost::development,ERROR:  Relation "sg_libraries" does not exist
): '
                select path
                from sg_libraries;
        '
[06/May/2003:14:46:42][742.45288448][-sched:16-] Error: Database operation "select" failed (exception NSDB, "Query was not a statement returning rows.")

ERROR:  Relation "sg_libraries" does not exist

SQL: 
                select path
                from sg_libraries;

Database operation "select" failed (exception NSDB, "Query was not a statement returning rows.")

ERROR:  Relation "sg_libraries" does not exist

SQL: 
                select path
                from sg_libraries;

    while executing
"ns_pg_bind select nsdb0 {
                select path
                from sg_libraries;
        }"
    ("uplevel" body line 1)
    invoked from within
"uplevel $ulevel [list ns_pg_bind $type $db $sql"
    invoked from within
For the record, I am using OpenACS 4.6.2/PostgreSQL on Mac OS X ... all the rest works fine ... Have you released a new version btw?

Thanks.

Collapse
Posted by Bruno Mattarollo on
Ok, posting to my own post ... It looks like when I installed the APM the -create.sql wasn't ran properly ... loaded it manually and this error is gone ... Sorry :)
Collapse
Posted by Khy H on
Is there a version that works with Oracle?
Collapse
Posted by William Byrne on
Bruno,

  I missed your messages. Sorry for the late reply. I installed the APM a few times and haven't experienced your symptoms; though, the OS was Linux. Any ideas as to why create.sql gave you a problem the first time around?
Collapse
Posted by Bruno Mattarollo on
Hello William,

Thanks for the reply. Unfortunately I don't know why the "-create.sql" file wasn't run properly. I will look into that when I am back in the office (today, monday, is a holiday here in the Netherlands).

The good thing is that it runs perfectly after loading the datamodel. I haven't had time to look more in details into it but I think I will have to as we are having some talks of using SOAP to communicate between some databases and our OpenACS installation... I managed to debug some SOAP messages with XML-SPY so that worked nicely. Generated some simple messages as well with Java (Apache AXIS and JAXP as well).... so this shouldn't be too complicated :)

Thanks again.

Collapse
Posted by Nick Carroll on
Hi all,

I've ported the soap-gateway package from ns_xml to tdom.  Why wasn't this package added to the OpenACS CVS repository?  Just wondering if there is some reason for this, otherwise I can check-in the tdom version.

There are still a few bugs that I need to work out, as well as further testing.

Also is it worth supporting SOAP 1.1?  My recommendation would be to follow the suggestion made in the 1.2 specs by returning a fault message that states SOAP 1.1 isn't supported.

Cheers,
Nick.

Collapse
Posted by William Byrne on
Hey Nick,

  The code is unchecked for no particular reason. I'll look into it. Regarding the v1.2 spec, it might be worthwhile to support the older v1.1 as a config option in the event someone requires it.

- Bill

Collapse
Posted by Rocael Hernández Rizzardini on
please commit your work you guys ...
Collapse
Posted by Jade Rubick on
When it is checked in, please announce it as well, so that we can set it up on bug-tracker and on the project page.
Collapse
Posted by Nick Carroll on
I'm in the process of refactoring the code with namespaces for tcl functions.  I will check the code in once I've finished this work.

Bill, I will take your advice and support SOAP 1.1 as well.  Also will you have time to maintain this package in the future?  If not I can volunteer to take on this responsibility as I will be working with this package for the duration of my phd, which is about another 2 more years.