Forum OpenACS Development: XOTcl remoting services

Request notifications

Collapse
Posted by Gustaf Neumann on
Dear Community,

There have recently been a couple of enticing announcements about
SOAP-based remoting solutions available to the OpenACS community,
namely TWiST by Tom Jackson and SOAP-DB by Rocael.

As many of you know from Stefan's presentation in Boston, we are doing
as well a substantial development at our institute in Vienna. We do
not want to fall into a "mine is better" discussion, but emphasize the
architectural approach we are following and sketch here the provided
services of mostly Stefan's SOAP and ORB implementation. Therefore, we
try to give a broader picture of the developed (and developing) XOTcl
based remoting services.

Motivation:

These XOTcl based remoting services are not meant to reproduce the
"classic OpenACS scenario" of proposing "just-another" package to the
community, as pointed out correctly by Don, but the intention is to
develop a more generic and framework-integrated solution (not
necessarily ours), capable to work with several
marshaling/demarshaling packages (not necessarily ours) and naming
services.

Towards a remoting architecture for OpenACS

Stefan is developing a generic remoting framework for OpenACS / .LRN
as part of his thesis for more than a year now. This project started
with a thorough review of the existing "half a dozen" OpenACS based
remoting packages (as Andrew Piskorski points out in
http://www.openacs.org/forums/message-view?message_id=316610), in
particular soap-gateway, the tsoap port and the XML-RPC package. With
that input and the observation, that there is an an obvious
correspondence between WSDL and Service Contracts, it became clear
that we have in OpenACS a good chance to come up with a highly
generic solution, which could make the framework attractive
for highly flexible middleware applications. As we are academics,
our profession is to investigate and develop new approaches with
the potential for publication.

Stefan's work currently contains two parts:

  • XOTcl-request-broker: a CORBA-style request broker (aka xorb).
    the request broker as a plug-in infrastructure for various
    remoting protocols (SOAP, XML-RPC, ...)
  • xotcl-soap: is the currently most dominantly used service
  • What is a request broker (xorb) all about?

    Procedure call abstraction is a well-established concept in OpenACS
    through service contracts. Request brokerage is available in many
    different middleware flavors, e.g. CORBA, .NET Remoting, Web Services (in their
    SOAP variant), DCOM, Java RMI etc. The developed request broker is
    basically designed as a generic object-oriented broker for abstract
    procedure calls, i.e. local and remote ones.

    Request brokerage is a generic form of request redirection, which is
    fairly common in OO frameworks. XOTcl itself provides several novel
    means for providing chains of redirectors (e.g. through mixins). One
    research goal is to rethink these redirection approaches in terms of
    remoting and middleware. A part of this discussion came up in
    (http://www.openacs.org/forums/message-view?message_id=588213).
    Note that functions/methods can be in general called explicitly,
    or via URLs (the request processor) or via remoting services.

    Publishing services by annotations:

    OpenACS supports declaring ad_procs as public or private. Similar to
    this, XOTcl supports the same flags. The declarations introduced via
    the remoting infrastructure through mixins allows to go one step
    further and to publish methods for remoting purposes through
    annotations. This allows great reuse of code and supports all OO
    features such as sub-classing (specialization) etc., such one can
    provide a customized remoting services, where everything is the same
    as in a more generic one except a few differences.

    Role of Service Contracts

    Service contracts - as they are available in OpenACS today - are a
    very powerful construct somewhat similar to service announcements in
    middleware and software-bus architectures, but they are not developer
    friendly (as they break locality) and they are not compatible with the
    needs of todays remoting SOAs. The request broker addresses these
    issues and provides an easier to use and more local interface to
    service contracts based on XOTcl objects. These service definitions
    can be transformed automatically into WSDL and can be more easily
    published.

    The overall idea we are trying to follow is to enrich existing
    concepts rather than to replace them. Object orientation in terms of
    XOTcl is not meant to provide a new language to replace existing TCL
    code, but rather to deliver advantages of OO abstractions to the
    OpenACS developers. It is fully compatible to existing TCL code. The
    ideas of extending service contracts for remoting applications is
    analogous. Essentially, one can replace occurrences of acs::sc::invoke
    uses by its xorb counterparts.

    The current features of Stefan's packages include:

    • support for Tcl procs and XOTcl classes and objects
    • server and client-side implementation of SOAP 1.1
    • architectural support for maintaining state
      between invocations (one might call a specific
      instance of an object (e.g. a certain shopping basket)
      and modify it)
    • publishing interface (including dynamic WSDL 1.1 generation,
      based on service contracts)
    • SOAP 1.1 support (SOAP marshaller / demarshaller on top of tdom)
    • Modular / extensible request handling: xosoap allows for registering
      interceptors / handlers that add functionality (logging,
      authentication/ authorization, caching, custom) for various scopes
      (per-service, explicit sequencing, ...)
    • Status:

      The framework is successfully deployed in the context of European
      Research Projects (www.icamp.eu, www.prolixproject.org) and is
      currently evaluated for EU4ALL by Emma.

      Although the mentioned features are fully functioning, changes in the
      naming of the concepts, or in the composition of the components are
      not unlikely. For example, in the icamp project, we are developing a
      lors mediator interfacing with a SQI based repositories for federated
      search of learning resources. Queries are distributed via SOAP to
      different servers, the results are collected and transformed to RSS
      and delivered to the asking client. In order to work in larger
      quantities, we had to use asynchronous queries (asking simultanueos n
      repositories, not blocking connection threads), which lead to strange
      problems in aolserver with vwait. Now we have rewritten the basic
      communication infrastructure, and things work smoothly. However, the
      handling of asynchronous queries and the new basic infrastructure lead
      to some interface changes as well. So, we want to gain more experience
      in larger applications before we release the package to the general
      audience.

      What remains to be done

      • more thorough test cases (currently just about 100 tests)
      • Complete compliance tests with SOAPBuilder Interoperability Lab
        (20% done)
      • as always after larger changes, code cleanup
      • Enhance and test publishing mechanism
      • Legacy test: replace ALL acs::sc::invoke uses by its xorb equivalent
      • Polishing consumer-side stub creation
      • Rewrite support for XSD complex data types based on XOTcl slots
        (50% done)
      • Provide for WS-I compliance of WSDL generation
      • Updating documentation and providing more examples
      • We should be able to provide a public prerelase withing the next weeks (early Mach)
        and organize a tutorial at the Vienna OpenACS conference, if there is interest.

Collapse
Posted by Haynes Horne on
Did I miss the announcement of this conference, Gustav? Can you point me to details? Thanks!
Collapse
Posted by Don Baccus on
Details are still under discussion, currently we're thinking last week in april but please don't do anything rash like book an airplane ticket yet.
Collapse
Posted by Don Baccus on
This is very interesting stuff.

There are a couple of issues that will result in this code being a bit marginalized unless they're resolved.

1. I recognize the need for Stefan to be able to continue to view this code as a research project. However, OpenACS and .LRN are production systems and folks need to be able to depend on code stability. So at some point you're going to have to be able to say "here is our published API, it will remain stable at least in regard to backwards compatibility with previous releases".

2. Oracle support. Everyone knows that I've suggested we drop this twice now. Both times, the community has shouted me down. At the moment, at least, anything that's incorporated into OpenACS core or the mainstream .LRN stuff needs to support Oracle. I don't expect U Vienna to do this but at some point either a) the community becomes convinced to no longer support Oracle or b) Vienna partners with people using Oracle for .LRN/OpenACS in order to get a port working (and this is really impossible until the code stablizes).

I'm not trying to be negative with either of these points, just trying to state the reality within which we operate if we want our code to be adopted into the framework.

Collapse
Posted by Stefan Sobernig on
Don, thx for these two comments.
Btw, I don't consider your attitude negative at all, _honi soit qui mal y pense_.

First, I want to make sure that there is no misunderstanding: We certainly do call for a more generic and viable solution in terms of call abstraction & remoting for OpenACS. However, this is not necessarily to be found in our work alone! All I want to stress at this point (before we can play our card = upcoming prerelease) is that any decision (either by the single developer or the OCT) should not be limited to a rough "SOAP-goes-Core".

ad-1 :: published API & code stability

As for the API, this is -- as prominently mentioned by Gustaf Neumann -- our top priority for the prerelease. As long as you refer by "code stability" to a stable interface, we certainly have to meet this fundamental requirement. However, the interface is already matured (though still not stable) through incorporating the experiences from the projects mentioned above and beyond. Moreover, the interface will certainly be familiar to many of you as we tried to stick with established notational forms, for instance, when declaring a contract the OOish interface resembles the good old arraylist.

ad-2 :: oracle support

I cannot comment on the fundamental issue here, nor the overall opinion in the community. Currently, the entire request broker operates on the established collection of db calls offered by acs-service-contract with only minor extensions. However, there is currently not a fundamental rewrite at this level (which might though be a good thing to do at some point) and therefore the Oracle issue should not be a barrier in this respect.

Collapse
Posted by Gustaf Neumann on
concerning 1: what you are addressing is one of the reason, we were not publishing the code say, a year ago, when it started to do something useful. Code stability is reached best, when nobody develops the code further. note, that we are not taking about making it part of acs-core, or even xotcl-core. Once a package is fully documented, it reaches some natural state of code stability, since changes become more costly for the developer.

concerning 2: most parts of xotcl-core don't depend on the database at all, mostly the oo layer for the content-repository does. Stefan's code does not depend on this. Only the xotcl request broker uses the database to interface with the service contracts.

Technically, we could factor out the database dependencies for xotcl-core and make it different packages, which will complicate bootstrapping and increase the number of small packages. Alternatively, we could define a separate project (e.g. ooacs) which shares large parts of the code base with oacs, but which does not have the oracle constraints.

Collapse
Posted by Jade Rubick on
It might be useful for someone to look at all of the SOAP and WSDL related solutions and summarize the pros and cons of each.
Collapse
Posted by Tom Jackson on
Information on tWSDL is here:

http://junom.com/document/tWSDL/

I believe that the software has achieved the specified goals for the server. The example API docs are a little old, but the example services work and are the best example of how to use the API. Also TWiST is an example of how to create a higher level API for a specific purpose, such as wrapping the database API.

tWSDL has a number of layered abstractions which are not fully explained in the documentation (yet). These abstractions will not likely change as they reflect the standards.

Collapse
Posted by Stefan Sobernig on
Jade, I heavily support this idea. Such a comparison proofed useful for our work and it is certainly a necessity for the community.

One intention of on our report above was to contribute to a more structured discussion on the state of (a)
abstract dispatch mechanisms in OpenACS and (b) of remoting support for OpenACS.

As a first step, we might come up with a state-of-the-art report (xowiki-based), requirements gathering based upon the results of the existing solutions, review of available code, ... we can then distil this into a consumable comparison scheme.

Again, this would also be an interesting task for a thematic workshop during scheduled Vienna meeting, such a comparison being the joint work artefact.

Collapse
Posted by Don Baccus on
note, that we are not taking about making it part of acs-core ...
I'm anticipating a bit ... web services are becoming more and more important, and at some point it's going to be hard for us *not* to provide one central, approved solution possibly supported as an acs-core package.

Thanks for pointing out that Stefan's work does very little db querying on its own. Sounds like Oracle support won't be a big deal if the decision is made to do so at some point.