Forum OpenACS Development: Remoting (Formerly: The future of OpenACS)

I forked the thread so not to distract from a discussion that is more than just some tech-heads elaborations on what is called "Web services". Hope, you agree ...
Now, what are the issues with OpenACS? The main issue is that most OpenACS applications depend upon page views. By that I mean that pages serve as a procedure. This makes it nearly impossible to reuse this code by a different protocol.
If this would be the major concern in the realm of OpenACS then any solution centered around WSDL breaks a fly on the wheel. First, the is a more lightweight, streamlined alternative available to WSDL, called WADL (somehow sponsored by Sun, by now) which is even readable and graspable by human minds. Nevertheless, it realises a basic resource model that conceptualises all, e.g. pages, remote operations under this notion of callable/invokable resources. Note, however, if this your or OpenACS's concern, the question is why not going for a generic "active web objects framework" (remember the fashionable debate couple of years ago; see also ActiWeb done by our group here in Vienna) and providing connectors/ bridges to some to those protocol families that have a completely different use and history, e.g. WSDL, SOAP, XML-RPC, ...?
Another issue specific to Tcl is important to recognize: Tcl does not maintain type information, ad_proc actually complicates the situation, and we know nothing about certain things like return types, use of uplevel/upvar, etc. Many programming languages know all this information and so it is 'easy' to serialize a data structure to XML. This is why there are so many rpc packages in every language.
I disagree on a couple of statements here. How could Tcl (as such) maintain something like "type information" if it only exposes TCL Strings, by design (apart from internal conversion to c integers and floats, and tcl objects as internal representations, however, never exposed)? Btw., serialization or marshaling is not exclusively build around introspective capabilities of a language, many bits and pieces needed can not be captured this way. This is why there is the notion of "interface description" around, the making-explicit of what needs to be exposed. The family of interface descriptions (and their styles) is long and old, just think of desig-by-contract based approaches allowing for the definition of invariants enforced throughout the lifecycle of object states etc.
Second is that it is unlikely that anyone anymore really wants to expose an internal procedure to the network. The best method for handling this is to write a wrapper procedure to restrict use or perform any additional checks that usually take place prior to execution.
Again, I disagree. If you have a uniform dispatcher/invoker (e.g. a message redirector) at hand then you should not require someone to write a wrapper proc for each operation to be exposed (and therefore duplicate efforts), no, you go for something that hooks in the dispatch, allow for defining "invocation policies", allow for conditions + privileges and scopes (all procs, single proc, channels, ...). This should be easily realisable in your case, if your query-writer package has such a central dispatch mechanism anyway. These policies per se are re-usable, in contrast to every wrapper proc needing to enforce atomic 'checks' as you call it.
One of these 'checks' is validation. This is where the wheels fall off of rpc and it is why nobody, not .NET or AXIS performs validation out of the box! And by validation, in this situation I only mean simple validation like is 4 an int? Derived types and complex type validation...this isn't even supported if you wished to do it.
Here, I assume, you refer to your experience with tWSDL/TWist being realised around what is called actually referred to as WSDL Document/Literal (SOAP as specification is not related to this in any way, per se and by standard, so to speak). Type constraint enforcement is not only a tricky thing (due to the ambiguity of XML as such etc.), but sometimes it is simply not useful or appropriate. I'm afraid I'm not quite sure about what you really mean by "validation out of the box" is not supported?! Type validation is definitely supported, at various levels and under major restrictions, but it is. Primitive types are usually checked against a value space and notational forms (decimal, ...), complex types are first matchable during serialisation/deserialisation and again when being mapped to object types in the target system. May be you can clarify your statement, I might have misunderstood it somehow, but I just wanted to make sure. But my key statement should be another and refers to
Just make up your own method of figuring out if the input is valid, test, debug and hope.
. What you refer to is more or less a passive programming style, leaving the processing decision to either end. This are not evil spirits, but a valid and advisable programming style anyhow. It is also the idea of "message- or document exchange of remoting" which is not the same as WSDL (or SOAP) Document/Literal: Exchange XML documents (after a schema handshake) and leave all validation and processing (eventual mapping into local data constructs etc.) to the decision of the endpoint developers. But, again, this is a valid approach and has nothing to do with mere support for WSDL Document/Exchange.

(Sorry, Tom, but I think discussions should be kept open to the interested public, so from time to time we should either describe details discussed or refrain from going into a certain level of detail) For those not so familar with these terms (and still interested and following): WSDL basically describes four aspects: 1-) A remote interface (operations, parameters etc.), 2-) "types" for these parameters (parameters are actually grouped in input and output messages), 3-) rules how to create the XML messages used to exchange operation calls between two ends, and 4-) . The term "Document/Literal" refers to this rule set, and describes two aspects:

  • "Document" vs. "RPC": Document means, the XML message to expect from either side is "well-formed and valid XML document" you can not only throw in XML parsers but also validate against schema descriptions (RelaxNG, XML Schema). The opposite style is RPC, and in fact, the only difference (looking at XML messages in Document and RPC styles) is that in RPC, the name of the remote operation is part of the message, as an XML element. In Document style, it is not!!!! So what is the key difference now, well, how should the receiving end of this message now which local operation to invoke on??? So what the key difference here is that dispatch resolution needs to be handled differently (taking http header information, etc.)
  • "Literal" vs. "Encoded": Literal translates to "literal meaning" and simply says take the XML message that comes in as what it is, a completed and self-contained XML document. Encoded, on the contrary, means that it is not generic XML, but a dialect of XML that one should expect. For instance, associatives arrays can be written differently (serialised differently).

  • One thing has become clear over the last few years: developers are starting to recognize that interoperability is the key. Interoperability is achieved by restricting the number of options and requiring public descriptions of services which don't require out of channel communications.
  • "Restricting the number of options" ... This sounds like the fallen promise of standardisation comittees. Interoperability is more a human (and therefore social) phenomenon than a technical. So I rather support your initial comment of public descriptions which are not WSDL or the like. From my perspective, developer support is not simply given by accessing some interface descriptions, but by means of framework design. If you look at all those middleware platforms (DCOM, CORBA, Web Services, .NET Remoting, ...) co-exisiting and competing over the years, all of them follow or employ very similar framework sketches (design patterns, actually). There is great work out there that tries to find and describe these patterns, and the major gain is, that perceived complexity is reduced by common metaphors. Perceived interoperability happens in human minds by anticipating the remote system and not by reading an interface description issued by these systems.
    Collapse
    Posted by Stefan Sobernig on
    Sorry, I found some typos or left-outs:
    • Item "4-)" on WSDL should be "explicit invocation styles"
    • The last bullet when talking about Document vs RPC, Literal vs. Encoded should be a blockquote.
    Collapse
    Posted by Tom Jackson on
    Stefan,

    Why don't we compare apples to apples. I am talking about real software. TWiST, .NET, AXIS, as they exist today. The purpose of this software is to solve a problem, not philosophy. The tiny problem I am interested in solving is exposing useful operations over a network and making it as easy as possible for remote users or programs to figure out how to actually invoke the operations.

    At any rate, TWiST exists and works. Does the software you are speaking of exist? Could you give examples? The comments I have made about ASP.NET and AXIS are well documented. The reason I know is that after I wrote TWiST, I started hunting around for how these other solutions did the same thing as TWiST. Well, very poorly and not out of the box. However, I could be wrong, and I would really and truly be interested in parallel examples, of even the simplist web service. But the requirements are that the developer:

    1. Doesn't have to write their own WSDL file or move it into place by hand.
    2. Can abstractly describe simpleTypes not based upon a given programming language.
    3. Can describe how to restrict simpleTypes from built in abstract types.
    4. Can describe how to combine simpleTypes into complexTypes
    5. Doesn't have to write validation code or type creation code as a separate step from description.
    6. Can name their input and output parameters as they wish without rewriting their internal code.
    7. Don't have to edit source code files to make configuration changes to the service.

    Okay, that is enough, Actually if you find examples of how to do ANY of these with either .NET or AXIS, please pass them along. I am stuck on writing a howto to find such examples.

    There is one fact which always stands out for me on this topic, I'll restate it and hope that someone can set me straight, because if it is not a true fact which always must be kept in mind, I need to rethink my whole way of writing code. Here is the fact: data cannot be created out of thin air. Maybe this is one of those laws of thermodynamics.

    The situation is this: some code exists in memory or in a file. This code cannot 'remote' itself without some help. If this fact is not fully appreciated, those who fail to appreciate it will wait in vain for this 'remoting' to happen all by itself. If things were otherwise, we would have programming languages which consisted of one word, because everything else has already been figured out.

    Skip the one word. There is proposed a great remoting program. You install it and suddenly it does exactly what you want without any input. It speaks whatever language or protocol you can think of. Not only does it speak, but anyone or any program which contacts it understands everything. Communication is perfect.

    Until I find this program, or it finds me, I'll try to focus on figuring out what information is needed to move from an internal code to a useful service. What language will I use to do this? WSDL. Will I follow any advice from others, or just consider my creative genius enough? No, I'll follow the WS-I Basic Profile.

    I'll use a WSDL file to hold information, data, created by a person in order to communicate what I wish to be a service. By following a certain format, the WSDL format, I reduce the amount of information I need to create. The form of the WSDL file is fixed by certain standards. The form of the messages is fixed, the description of the types and operations follows a fixed pattern. All I have to do is provide a few pieces of real data. Maybe this is called work, maybe thinking, but it doesn't come out of thin air. This is what folks get paid for. If we could remote a procedure by adding a switch, then remoting would be a one bit operation. One bit of data. We can argue all the way around this for years, but the information has to come from somewhere. All TWiST does is provide a format for collecting this information in one place so that tWSDL can do all the boring code writing which doesn't contain any more content than the TWiST file. This is what has to be seen: a few lines of a configuration file is expanded into a working web service. Developers don't need to spend hours or days writing code which essentially has no more data content than the description of the service.

    Collapse
    Posted by Stefan Sobernig on
    Sorry for not continuing the discussion any earlier, I was out of reach of any internet access for the last two days.
    I am talking about real software [...] At any rate, TWiST exists and works. Does the software you are speaking of exist? Could you give examples?
    I'd wish you wouldn't have played this card, as most of the issues listed and mentioned can easily be discussed without referring to YOUR and MY piece of work. If I remember correctly, the idea of the thread (as it was started by Malte) was to collect ideas and requirements for remoting solutions for OpenACS, and not whether YOURS or MINE are better suited. This is a valid level of discussion, but a different. Anyway, yes there is such a MY, and it is already in use by a smaller community. I did not release it to a wider public simply because documentation was so awfully lacking that i did not dare to do so. This has changed, I will be happy to put you on my list of addressees of the upcoming announcement. In the meantime, you might refer to the slide set of a talk given at the last OACS conference. At least, this should convince you that is not pure philosophy, although philosophy is an important ingredient. As for your requirements list below, I need to state that initially, you just referred to "out-of-the-box validation not being supported" by existing framework, what you are looking for by far more than that. Anyway, let me try to restate your quest, just to make sure i get it right: I have a tcl code block (a proc) and want to expose it as a SOAP-enabled service the easiest way possible. the infrastructure should take care of gathering service-declaring information from the proc declaration. how to achieve this in a tcl environment without annotating/extra-declaration efforts by the developer.
    1. Doesn't have to write their own WSDL file or move it into place by hand.
    Ok, auto-generation of wsdl files out of existing code. there are quite a few examples available. However, the resources you posted below indicate them already. So, I am wondering why you are still desperately looking for more sources. The main difference between AXIS an ASP.NET (in all implementations, MS and mono's) is that AXIS uses static stub creation (incl. java > wsdl) while ASP.NET does this dynamically with some annotation sugar (it is reflection-heaviy in implementation).
    • AXIS: there is a wsdl compiler available, called Java2WSDL. this achieves what you seem to be looking for, however, under heavy constraints of (a) compiler languages as such and (b) java's limitations. there need to be an Interface construct, java primitive types are mapped to xsd built-in types, java types are mapped to xs complex types. Achieving this on the fly (as you intend) is hard in compiler languages, as reflection is a costly thing in these environments.
    • ASP.NET uses annotators and internally reflection to register web methods etc. with the dispatcher framework. again, .NET primitive types become xsd simple types, typed arguments or return values become complex xs types, etc. If you are interested in the details, a look in the source code is necessary which might be hard. But you might refer to mono's code base which is open to the public domain and comes with an asp.net implementation. there are also some archived discussions on mono's devel list.
    I don't quite understand why you are so interested in examples from compiler language environments. Scripting languages are much more powerful in this respect!
    2. Can abstractly describe simpleTypes not based upon a given programming language.
    This I don't quite understand, provided that I understood your guiding scenario. You want to have a language-bound code block turned into a service + service description, but you want simple types (xs based types) not defined in the very language the code block was written? this is conflicting and contradictory to your scenario. If I want to deploy my tcl proc as service, I don't want to specify a constraint derived type "myInteger" based on the base type xsd:interger used when the service description is built etc. Was it you saying "interoperability is result of the restriction of options"? This is not separation of concerns as I understand it, but I might be wrong in my reading of your scenario. Btw., describing data types in abstract manner usually is referred to as abstract data types (ADTs) but I think you are purely referring to what in type creation in XML Schema world here.
    3. Can describe how to restrict simpleTypes from built in abstract types.
    The same goes here. A Tcl developer does not want to bother with these issues which are XML Schema specific.
    4. Can describe how to combine simpleTypes into complexTypes
    This is usually done (in the OO world) by mapping object types (class schemes and their primitive-typed attributes) to complex types. In procedural world, this requires some add-on abstraction.
    5. Doesn't have to write validation code or type creation code as a separate step from description.
    The question is rather what is the realm of validation: The streamed data represention (XML element against element description) or the language-level representation (Tcl string that is validated to be conforming to a notational form, character set and value space). In the streamed scope, use schema validation. In the language level, you need to provide your own validator. C'est la vie, but as you are centred on WSDL/XML Schema, you need to think of option (1).
    6. Can name their input and output parameters as they wish without rewriting their internal code.
    Again, separation of concern is not clear to me. I read the above requirement as follows: My exposed proc has a different signature (internal code, as you call it) as I want it to be exposed. This is a matter of framework design and, again, the employment of design patterns: Provide signature adapters with the adapters being exposed as service.
    7. Don't have to edit source code files to make configuration changes to the service.
    This is a basic requirement and the solution in most frameworks is referred to as "deployment descriptors". They come in various flavours, in AXIS they are XML-based descriptors, in ASP.NET they come as annotation tags, ...

    I will take up you questions/ requirements and include them (however, somehow clarified in my reading) in my documentation so the interested reader can then figure out the design decisions for MY remoting fetish.

    Collapse
    Posted by Tom Jackson on
    I'd wish you wouldn't have played this card, as most of the issues listed and mentioned can easily be discussed without referring to YOUR and MY piece of work. If I remember correctly, the idea of the thread (as it was started by Malte) was to collect ideas and requirements for remoting solutions for OpenACS, and not whether YOURS or MINE are better suited.

    Ummm, you started this thread.

    I have never looked at your software, don't know where to get it or how it works. I use my software as an example for several reasons, but I'm not promoting it. Anyway, the main reason for discussing software which actually exists is because it is obvious how things are done, while talking about stuff which could work or might be better is a waste of time. Until something is boiled down to code, everything sounds possible, that is the nature of programming languages: anything can be done, in any language.

    Another reason to talk about existing code is that code has behavior. If you want to discuss a future requirement, it is good to know how other people have actually solved the problem, what is involved. That is why I was looking for examples from other major players in 'remoting'. I talked about, and provided links to the ones I know about. Personally I'm not interested in re-inventing the wheel. When someone tries to code to a standard, they are giving up that part of the process. I have said I tried to follow the recommendations in WS-I Basic Profile. If you don't wish to follow a standard, it becomes very difficult to compare it to anything else. It does make it easier for you to claim that your software works and does what you want.

    I don't quite understand why you are so interested in examples from compiler language environments. Scripting languages are much more powerful in this respect!

    I listed the examples I could find, I asked for better examples. I'm still waiting.

    This I don't quite understand, provided that I understood your guiding scenario. You want to have a language-bound code block turned into a service + service description, but you want simple types (xs based types) not defined in the very language the code block was written? this is conflicting and contradictory to your scenario.

    Ahhh! Yes! This is exactly the advantage of Tcl over most other languages in this area. First consider the following: Tcl is best as a glue language. In OpenACS it is a glue language. OpenACS doesn't store information as a Tcl structure, it takes in some string and passes on a string to another application. Usually stuff goes into a database, which does have type requirements. Most procedures don't care what they take in, so you claim that the typing of this input or output doesn't make sense. Actually what doesn't make sense is caring about constructing types in Tcl just to pass them on to another application. But if you can test the type information, you can pass back information to the client in a useful form. Once the procedure executes, you pass from client error to server error, which is much less useful. This is why we have filters in ad_page_contract, not because the Tcl layer necessarily cares, but it is easier to return error information to the client.

    Besides simple typing and derivation, more important is structural typing. Structural typing is very important. Every langauge is slightly different and the point of WSDL using XML-Schema to describe how to construct types is that it is theoretically easy to build software which could validate a type (simple or complex). Does XML-Schema suck: big time. But one of the biggest sucking parts is that it tries too hard to accommodate optimized validation routines and fuzzy types.

    Anyway, I'm not soo sure what is contradictory: WSDL defines an interface, at some point there has to be code which translates from an XML/SOAP document to Tcl. This is required.

    The question is rather what is the realm of validation:

    The question is what helps the client, targeted error messages which relate to the data input, or a stacktrace from the server? Of course, validation also protects the server from attack. I'm not too impressed with MS releasing code which is all too happy to stick any data into any variable. My general rule is to validate as soon as possible, why wait? Oh, and it is a requirement with SOAP/WS-I that client errors should lead to no change in state on the server. The best way to ensure that is by validation before execution.

    This is a matter of framework design and, again, the employment of design patterns: Provide signature adapters with the adapters being exposed as service.

    Yes again! <ws>proc is an adapter! Actually it is a description of an adapter. The adapter is written by tWSDL based upon the input to <ws>proc. Very complex adapters have to be written by hand, although I haven't thought of a realistic example which would require this.

    This is a basic requirement and the solution in most frameworks is referred to as "deployment descriptors". They come in various flavours, in AXIS they are XML-based descriptors, in ASP.NET they come as annotation tags, ...

    Yes, this is how they do it. All the examples I have found are for very simple services. I know they can do more, obviously, but how easy it is? More importantly, how do you learn how to do it, where are the complex examples? I'm interested in example code. Another thing that is still missing is using annotations _and_ using complexTypes or derived types and not having to edit a bunch of files.

    I would like to promote something, two things: efficiency and simplicity. If you can boil out the information content which distinguishes one remoting service from another, you can write code which uses this tiny amount of information to write the actual code. One example of what information is required for a service is a WSDL file. It isn't the only example, but it is a good one. It doesn't contain all the information needed since the adapter information is missing. A more complex solution will require more information, more software to handle the complexity, etc.

    Collapse
    Posted by Stefan Sobernig on
    Ummm, you started this thread.
    Nope, if I remember correctly, it was Malte starting the thread on remoting, by collecting some rough scenarios. I continued by summarizing what had been discussed during the OpenACS Spring Conference, still focussing on a general road mapping. It was you going into design + implementation details (of TWiST/tWSDL) which I somehow found inappropriate in a Future-of-OpenACS thread and therefore forked the original. However, my idea was still a general road mapping. C'est tout.
    I listed the examples I could find, I asked for better examples. I'm still waiting.
    Mmmh, if you are looking for "better examples" to the AXIS/.NET/... examples posted by you, simple googling won't be enough. I had good experiences with browsing the code repositories (especially the test suites contained in there) and getting my hands dirty with re-implementing and extending test cases for my purposes. If this is no option for you, mailing to the developer lists may help (that's why I indicated an example thread from the mono-devel list previously). At least, either of these options is better than posting a request to this forum.
    What can be done? Have the most experienced people with regards to webservices (Stefan and Tom) along with interested OCT members come up with a generic strategy how to support webservices in OpenACS.
    Taking up Malte's original statement, I would like to call a spade a spade. Tom, would you be willing to contribute (a) to a phase of general + detailed requirements gathering/ road mapping and, subsequently, (b) realising implementation efforts, even though they go beyond the scope of TWiST/tWSDL and might even replace these two as self-contained solutions?
    Collapse
    Posted by Tom Jackson on
    I really wish I could figure out what I did that was inappropriate. Discussing code? This isn't a tea party.

    Anyway, I'm not interested in your remoting whatever. TWiST does what it does and doesn't need to be integrated with some ambiguous uber-code. Why not just release the code you have right now?

    Collapse
    Posted by Stefan Sobernig on
    Tom,
    Anyway, I'm not interested in your remoting whatever.

    I got this strong impression, indeed! However, you constantly become set on MY, YOURS, THEIRS etc. I was never referring to MY fetish necessarily being THE solution for OpenACS, I just wanted to know whether there is esprit for co-ordinated action.

    TWiST does what it does and doesn't need to be integrated with some ambiguous uber-code.

    At least in this respect, I got an honest answer. Thanks for making this clear to me.

    Why not just release the code you have right now?
    Why? The only thing that prevents me from doing so is the fact that I am currently on holidays and that I am not fully available to support (early) adopters. I will be back to office around the end of next week, so expect the release around that weekend.

    I, therefore, close this enlightening tea party from my side ...

    Collapse
    Posted by Tom Jackson on
    Here are the best descriptions I have found on ASP.NET and AJAX. If anyone has better references, please post them here if possible:

    Basic ASP.NET description:
    http://www.stardeveloper.com/articles/display.html?article=2001071401&page=5

    Adding validation (note what isn't shown, how to validate):
    http://msdn.microsoft.com/msdnmag/issues/03/07/XMLSchemaValidation/default.aspx?loc=&fig=true#fig10

    AXIS:
    http://ws.apache.org/axis/java/user-guide.html

    IBM on JAX-RPC/JAX-WS (configuration vs annotation):
    http://www.ibm.com/developerworks/library/j-cwt08025.html

    ASP.NET with WS-I (Is there more info somewhere?):
    http://msdn2.microsoft.com/en-us/library/ms230196%28VS.90%29.aspx

    IBM: compare JAX-RPC to JAX-WS (hello world example):
    http://www.ibm.com/developerworks/webservices/library/ws-tip-jaxwsrpc3/index.html?S_TACT=105AGX04&S_CMP=HP

    Collapse
    Posted by Stefan Sobernig on
    Tom, I had a quick look at your source collection,
    and especially from http://msdn.microsoft.com/msdnmag/issues/03/07/XMLSchemaValidation/default.aspx I believe that I better understand your scenario now. It is what I have been referring to the ambiguity of xml and the ambiguity/heterogeneity of xml handling (parsing, serialisation) in a previous posting.

    However, this is more related to the layer of marshaling/demarshaling than others, higher layers such
    as type bindings (stream -> language type), invocation
    dispatch etc. That's what the authors of the
    article and the described extension mention themselves.

    The reason for this behavior has to do with XmlSerializer, the underlying plumbing that takes care of object deserialization. XmlSerializer is very forgiving by design. It happily ignores XML nodes that it didn't expect and will use default CLR values for expected but missing XML nodes. It doesn't perform XML Schema validation and the consequence is that details like structure, ordering, occurrence constraints, or simple type restrictions are not enforced during deserialization.

    This is an implementational issue and not one that was designed or forgotten as feature. Why, because in a final stage (whether the value parsed for CalcArea->{W|w}idth correctly or not) will be tried to be assigned to the double-typed attribute of an object CalcArea . And this can fail or it won't.

    But there are more drastic examples as the lower/major case example if we talk about XML marshaling: What about either streaming based on XML attributes or XML elements. Semantically equivalent (or at least, undecided), but syntactically completely different.

    There is certainly a role for strict schema validation in terms of XMl Schema, RelaxNG etc., but not necessarily for examples as those given above.

    I would go for a double strategy (in fact, this is what I started for MY fetish):

    • provide an extension mechanism (interceptors) that can hook in schema-based validation on various scopes. But then it is also schema validation to the developer, i.e. he has to provide a schema fragment to be used as validation rule. See e.g. http://www.sosnoski.com/jibx-wiki/space/axis2-jibx/validation. I don't think that providing validation rules as language extension or annotation elements to the language are appropriate here, simply introducing a new complexity and still requiring basic knowledge/anticipation of schema validation.
    • Provide validating message parser. To a certain level but not complete level of complexity, these problems (cases, attribute vs. element) can be handled at this level.
    • Validating type binding procedure: When parsed message elements become elements of object states, than provide some validation and conformence checks. Oh, wait, this would require some OO! Indeed, OO principles are quite handy here. XOTcl comes with some nice, extensible mechansim to provide these functionalities.