Forum OpenACS Q&A: Tcl function call to remote AOLserver via xmlrpc ?

Folks, for those of you more familiar with xmlrpc, soap, and their various implementations for AOLserver or Tcl, I have a question for you:

Say I have AOLserver or tclsh process A, and an entirely seperate AOLserver process B. Is there any way for me to have process A call a Tcl function running in process B?

Security issues and the like aside, it would be very cool if I could have process A make an arbitrary Tcl function call to B, and have it all work pretty much the way any normal Tcl function call would work, so that most of my code in A wouldn't even have to know that it's doing a remote rather than local function call.

Would XMLRPC or SOAP let me doing something like that? Or is there any any other tool that would make this feasible? And what would I have to do to make it work? Thanks!

Andrew,

XMLRPC or SOAP will do this for you. Depending on the library you use, you might have to write a wrapper tcl proc to hide the xmlrpc/soap calling code.

This is all done over HTTP, so you could also just map a URL to a tcl proc, and parse the URL parameters.

I can give you more information if you need it.

Dave,<br>it would be very fine to have some practical <i>how to</i> about SOAP. I check periodically the nssoap project at SourceForge, but it is yet unfinished, while the ability to call a web service or to expose one is becoming a must!
Ah. NSSOap is pretty much stopped development. TclSOAP is what it was based on. I am working to get TclSOAP to load nicely with AOLserver, but it is still a struggle.

TclSOAP does work very well from tclsh, so it is a good way to learn about it. The latest version of TclSOAP supports exporting procs under xmlrpc and SOAP, so it is very useful.

The main issues are AOLserver's handling of Tcl namespaces and it's lack of support for "package require".

It would be possible to modify TclSOAP to load in AOLserver, but I thought it would be best if we could just use the TclSOAP tcl package as is, instead of rewriting it every time the code changes.

Cool! Dave, writing wrapper procs doesn't sound like a problem, but, which of the various xmlrpc or soap packages would you recomend I use?

Since I definitely want to be able to make remote function calls to AOLserver from both AOLserver and a tclsh script, should I start by trying to use tclSOAP, Steve Ball's xmlrpc, or some other Tcl-based toolkit for everything, both in tclsh and AOLserver? Or should I just use ns_xml and ns_xmlrpc inside AOLserver, and pick some entirely separate tool to use from tclsh?

I'd greatly appreciate comments from people with the experience to compare/contrast the different implementations available...

Andrew,

The fastest way to get going is to use ns_xmlrpc in AOLserver, and TclSOAP in tclsh (using the xmlrpc implementation).

ns_xmlrpc has very little error checking code built in though, so you might want to look into that. It would be fine for testing. I really don't plan on developing it furhter, because TclSOAP is much more mature, and support xmlrpc and soap using the same code, so it is a much better long-term solution.

Hi Dave,
The main issues are AOLserver's handling of Tcl namespaces and it's lack of support for "package require".
Could you please explain what the namespace problems are and specifically with the package? Thank you, Hanjo.
The problems are in how AOLserver instantiates interpreters. Quoting Zoran Vasiljevic in the "new AOL-based thread-happy mem allocator in 8.4" thread in the AOLserver list:

"The AOLserver does seed the the connection
interpreter with commands introspected from within loaded initialized
startup interpreter. It just copies (blindly) all command structures
found in the initial interp, which causes quite a few problems with
Tcl packages planting commands with associated delete callbacks.

So the problem is not AOLserver<->Tcl integration. The problem is
how the AOLserver does handle Tcl extensions and their registered
commands. There is nothing Tcl (you) can do about it. But there is
something AOLserver (Kris/Jim/community) can do about.
I've made a patch for 3.4 which corrects above problems. I'm just
qurious to find out if/how/when can it be included in the core
server distro."

To me this is one of the serious problems with the architecture right now because it basically alienates AOLserver from the Tcl world to a certain extent. We can't use the many available Tcl modules that would provide significant functionality, in AOLserver without some ugly hacks.

I and Dave have tried reviving that thread and trying to get people to discuss those issues, but we have failed miserably. I'm not knowledgeable enough of AOLserver and Tcl internals to fix the problem, but it appears that this is not an issue for the DCI (AOL team that works on AOLserver) folks, since I haven't seen them worrying about it.

If anybody would like to maybe form some sort of "task force" to study and fix this problem, this would be a good time do it. I was hoping AOLserver 4 would have that fixed, but so far it has not.

Collapse
Posted by Andrew Piskorski on
I just installed nsxml, ns_xmlrpc, and ran it through Userland's XML-RPC Validator. All the tests passed except the manyTypesTest.

Now, that test failed due to "Error: Poorly formed XML text, string constant is improperly formatted. (At character #213.)", because AOLserver didn't send back XML, the ns_xmlrpc code died and AOLserver sent back it's usual Server Error HTML message. The stack trace was:

[02/Jul/2002:20:02:24][17809.8][-conn0-] Error: value "Fri Jan 01 09:23:58 EWT 1904" 
for option "-date" is not a valid date 
(unable to convert date-time string "Fri Jan 01 09:23:58 EWT 1904")
value "Fri Jan 01 09:23:58 EWT 1904" for option "-date" is not a valid date 
(unable to convert date-time string "Fri Jan 01 09:23:58 EWT 1904")
    while executing
"xmlrpc_construct $data value $datum"
    ("foreach" body line 2)
    invoked from within
"foreach datum $value {
                set result [xmlrpc_construct $data value $datum]
                if {[llength $result]} {
                    r..."
    ("-array" arm line 4)
    invoked from within
"switch -- $option {
            -string -
            -text {
                ns_xml node setcontent  [xmlrpc_createContext $node $context]  $value
  ..."
    (procedure "xmlrpc_construct" line 18)
    invoked from within
"xmlrpc_construct $value_id {} $data"
    (procedure "xmlrpc_respond" line 8)
    invoked from within
"xmlrpc_respond $result"
    (procedure "xmlrpc_invoke" line 33)
    invoked from within
"xmlrpc_invoke $content"
    (procedure "xml_rpcdispatcher" line 12)
    invoked from within
"xml_rpcdispatcher"

The dateTime part of the request validator.xmlrpc.com is sending my server is this:

<value><dateTime.iso8601>19040101T08:23:58</dateTime.iso8601></value>

The validator1.manyTypesTest proc is taking its incoming dateTime variable and doing this to it:

[list -date [clock format $dateTime]]

And the Tcl error is coming from this code in xmlrpc_construct:

-date {
    if {[catch {
        clock format [clock scan $value] 
              -format {%Y%m%dT%H:%M:%S}
    } datevalue]} {
        return -code error 
                "value "$value" for option "$option" is not a valid date ($datevalue)"
    }
    ns_xml node new_child 
            [xmlrpc_createContext $node $context] 
            "dateTime.iso8601" $datevalue
}

So, I'm not sure what's going on there, but it looks to me like the clock command proc validator1.manyTypesTest is probably messed up.

Is the intention there to convert the incoming dateTime.iso8601 format into Tcl's internal integer representation? I don't think so, because then it'd be doing a clock scan, not clock format. And anyway, I tried the obvious fixes of changing it to either [list -date $dateTime] or [list -date [clock scan $dateTime]], and neither of those worked, they just changed the nature of the error.

So, what are these different bits of code supposed to be doing? Could someone knowledgeable please explain how these bits of ns_xmlrpc are supposed to work? I suspect it's simple but I don't see it...

Andrew,

I have previously tested the nsxmlrpc code at validator.xmlrpc.com and it worked fine.

That said, I just ported the code from using TclXML to nsxml and haven't looked at it since. I have learned alot about XMLRPC since then, so it probably can be fixed up quite a bit.

I'll take a look and see if I can figure it out.

Andrew, I just installed nsxmlrpc at www.thedesignexperience.org and it validated fine. This is a strange problem. Which AOLserver/Tcl versions are you running? I am using Tcl8.3 Aolserver 3.3+ad13 Maybe we can track this down. Here is the request and response for the manyTypesTest: Request
POST /RPC2 HTTP/1.0
User-Agent: Frontier/7.0.1 (WinNT)
Host: www.thedesignexperience.org
Content-Type: text/xml
Content-length: 686

<?xml version="1.0"?>
<methodCall>
	<methodName>validator1.manyTypesTest</methodName>
	<params>
		<param>
			<value><i4>12874</i4></value>
			</param>
		<param>
			<value><boolean>0</boolean></value>
			</param>
		<param>
			<value>Minnesota</value>
			</param>
		<param>
			<value><double>10202.33333333</double></value>
			</param>
		<param>
			<value><dateTime.iso8601>19040101T05:23:44</dateTime.iso8601></value>
			</param>
		<param>
			<value><base64>R0lGODlhFgASAJEAAP/////OnM7O/wAAACH5BAEAAAAALAAAAAAWABIAAAJAhI+py40zDIzujEDBzW0n74AaFGChqZUYylyYq7ILXJJ1BU95l6r23RrRYhyL5jiJAT/Ink8WTPoqHx31im0UAAA7</base64></value>
			</param>
		</params>
	</methodCall>

Response
TTP/1.0 200 OK
Set-Cookie: ad_session_id=1710038%2c0%20%7b582%201025663719%201D5A4B84D580197B61BA6EB179B0937355AAED37%7d; Path=/; Max-Age=1200
Content-Type: text/xml; charset=iso-8859-1
MIME-Version: 1.0
Date: Wed, 03 Jul 2002 02:15:19 GMT
Server: AOLserver/3.3.1+ad13
Content-Length: 514
Connection: close

<?xml version="1.0"?>
<methodResponse>
<params>
<param>
<value>
<array>
<data>
<value><i4>12874</i4></value>
<value><boolean>0</boolean></value>
<value>Minnesota</value>
<value><double>10202.33333333</double></value>
<value><dateTime.iso8601>19040101T05:23:44</dateTime.iso8601></value>
<value><base64>R0lGODlhFgASAJEAAP/////OnM7O/wAAACH5BAEAAAAALAAAAAAWABIAAAJAhI+py40zDIzujEDBzW0n74AaFGChqZUYylyYq7ILXJJ1BU95l6r23RrRYhyL5jiJAT/Ink8WTPoqHx31im0UAAA7</base64></value>
</data>
</array>
</value>
</param>
</params>
</methodResponse>

This is code I just checked out from the openacs.org CVS. Where did you get your code from?
Dave, I'm also using 3.3+ad13 with Tcl 8.3, in my case on Solaris (SunOS 5.8). My ns_xmlrpc is a cvs checkout from the OpenACS trunk:

cvs -d:pserver:anonymous@openacs.org:/cvsroot co ns_xmlrpc

My nsxml is from AOLserver on SourceForge:

cvs -d:pserver:anonymous@cvs.aolserver.sourceforge.net:/cvsroot/aolserver co nsxml

My libxml appears to be version 2.3.9, and was installed on this box back in July 2001.

One thing I forgot to mention before, is that when I run the nsxml tests/index.tcl, I get the following error:

[02/Jul/2002:19:59:35][17809.8][-conn0-] Error: Instead of erroring as expected, "set doc_id [ns_xml string parse xml -validate $xml_string]" returned: Entity: line 1: error: Validation failed: no DTD found !
<bloop><blop></blop><schism/><cantankerous moldy="true" flaky="false"/></bloop>
      ^
t1
Instead of erroring as expected, "set doc_id [ns_xml string parse xml -validate $xml_string]" returned: Entity: line 1: error: Validation failed: no DTD found !
<bloop><blop></blop><schism/><cantankerous moldy="true" flaky="false"/></bloop>
      ^
t1
    while executing
"error "Instead of erroring as expected, "$cmd" returned: $return_val""
    (procedure "report_error" line 8)
    invoked from within
"report_error {set doc_id [ns_xml string parse xml -validate $xml_string]}"
    ("eval" body line 1)
    invoked from within
"eval $line"
    (procedure "show_and_tell" line 6)
    invoked from within
"show_and_tell {
    set xml_string {<bloop><blop></blop><schism/><cantankerous moldy="true" flaky="false"/></bloop>}
    set doc_id [ns_xml string par..."
    (file "/web/aol3-src/nsbbapi/www/nsxml-tests/index.tcl" line 23)
    invoked from within
"source $file"
    invoked from within
"ns_sourceproc cns6 {}"
Here's another weird ns_xmlrpc error. I set up a little page to have my AOLserver hit itself with an xmlrpc call. My little test/rpc.tcl page was:

set url {http://localhost:8075/RPC2}
append body [xmlrpc_call $url ns_info name]
ns_return 200 text/plain $body

From the ns_logs and stack trace below, it's pretty clear that this combination can't possibly work right. AOLserver seems to be handling and returning the rpc call ok, but once the xml gets to the client, the 'ns_xml node name' is 'methodResponse', which immediately breaks stuff because xmlrpc_parse is only set up to handle 'params', not 'methodResponse'.

These errors make it seem like I need to actually learn to use ns_xml and heavily debug ns_xmlrpc. But, the mysterious part is, everybody else seems to be saying that this same code works fine for them. What could be wrong?

The error from my test/rpc.tcl page:

[03/Jul/2002:04:05:34][28484.8][-conn0-] Debug: xmlrpc_callResponse: response: '<?xml version="1.0"?>
<methodResponse><params><param><value>AOLserver</value></param></params></methodResponse>
'
[03/Jul/2002:04:05:34][28484.8][-conn0-] Debug: xmlrpc_parse: ns_xml node name: 'methodResponse'
[03/Jul/2002:04:05:34][28484.8][-conn0-] Error: XMLRPC-ERROR remote procedure call failed: "invalid server response ()"
XMLRPC-ERROR remote procedure call failed: "invalid server response ()"
    while executing
"xmlrpc_callResponse $response"
    (procedure "xmlrpc_call" line 33)
    invoked from within
"xmlrpc_call $url ns_info name"
    invoked from within
"append body [xmlrpc_call $url ns_info name]"
    (file "/web/aol3-src/nsbbapi/www/test/rpc.tcl" line 6)
    invoked from within
"source $file"
    invoked from within
"ns_sourceproc cns0 {}"
Andrew,

I haven't tested that latest ns_xml code with validation with ns_xmlrpc, or actually tested it at all.

So that might be a culprit.

On the client issues, I don't know how well tested is it as a client, validator.xmlrpc.com only tests the server functions.

I'll take another look at the code.

Andew, It looks like the client code was not fixed to handle the newer versions of nsxml without a bug that returned the first child instead of root for ns_xml doc root $doc_id. Add this code right after the set response [ns_xml doc root $doc_id]
if {[ns_xml node name $response] != "methodResponse"} {
            set response [lindex [ns_xml node children $response] 0]
        }
I have checked this in, so you can also cvs update to test it. This code checks if the root node returned refers to the methodRepsonse node or the root node. If it refers to the root node, it goes to the next node to get the actual methodResponse. This should fix the code to work with any version of ns_xml. Should fix it.
Dave, yep, that fixed it. But now that I can actually make rpc calls, I've run into some other issues.

The server messes up whenever it needs to return any string (including lists) with whitespace in it - it returns only the last non-whitspace item. Is this a bug?

It also doesn't play nice with using the optional parameters of ad_proc. The following code fails in xmlrpc_construct with the error: no value for option "abc" (see below for my test proc):

[xmlrpc_call $url bb_rpc_string -quotes_p 1 abc]

Also, the XML-RPC spec. defines scalars (of different types), structs, and arrays. Structs have key value pairs like Tcl arrays and do not preserve order, but (unlike Tcl arrays) struct values may themselves be scalars, structs, or arrays. Arrays contain values only (not key value pairs), and each value may be also be a scalar, struct, or array. I assume arrays preserve order, although the spec. doesn't say.

So, I guess Tcl arrays should be represented as XML-RPC structs, and Tcl lists could be represented by either scalar strings or XML-RPC arrays. If the client and server are both using Tcl, I just using the Tcl string representation of lists might be most convenient, but XML-RPC arrays would be correct cross-language way to represent lists.

I don't think ns_xmlrpc is attempting to do any magic to automatically convert Tcl data structures to XML-RPC data structures. So, I assume I'm supposed to write a wrapper around each of my procedures to do these data structure conversions for me? Are there any good examples somewhere of doing that?

validator.tcl has a lot of stuff like this:

set struct(times10) [expr $number * 10]
return "-struct [list [array get struct]]"
And it looks to me like the xmlrpc_construct proc expects to get all data as a Tcl list containing values, plus those little "what type of data structure or scalar value is this?" prefix tags, like -struct, -array, or -date.

So, am I on the right track here? I have to write wrappers that explicitly specify all my data types in XML-RPC terms? Has anyone considered or tried having ns_xmlrpc automagically convert Tcl to XML-RPC data structures and back, instead?

Here are some examples showing the whitespace and parsing of '-' (like in ad_proc) bugs:

ad_proc bb_rpc_list_1 {{}} {} {
   return [list a b c d]
}
ad_proc bb_rpc_list_2 {{}} {} {
   return ""[list a b c d]""
}

ad_proc bb_rpc_string_1 {{}} {} {
   return {"a b c d"}
}
ad_proc bb_rpc_string_2 {{}} {} {
   return {a b c d}
}

ad_proc bb_rpc_string {{
   -quotes_p 0
} str } {} {
   if { $quotes_p } {
      return ""$str""
   } else {
      return $str
   }
}

append body "
local:  [bb_rpc_list_1]  	remote:  [xmlrpc_call $url bb_rpc_list_1]
local:  [bb_rpc_list_2]  	remote:  [xmlrpc_call $url bb_rpc_list_2]
                           
local:  [bb_rpc_string_1]  	remote:  [xmlrpc_call $url bb_rpc_string_1]
local:  [bb_rpc_string_2]  	remote:  [xmlrpc_call $url bb_rpc_string_2]

local:  [bb_rpc_string {a b c}]  	remote:  [xmlrpc_call $url bb_rpc_string {a b c}]
local:  [bb_rpc_string {abc}]    	remote:  [xmlrpc_call $url bb_rpc_string {abc}]

local:  [bb_rpc_string -quotes_p 1 {abc}]    	remote:  FAILS
"

## Fails in xmlrpc_construct with:  no value for option "abc"
#[xmlrpc_call $url bb_rpc_string -quotes_p 1 {abc}]

ns_return 200 text/plain $body

And here are my results from the commands immediately above:

local:  a b c d    remote:  d
local:  "a b c d"  remote:  a b c d
                      
local:  "a b c d"  remote:  a b c d
local:  a b c d    remote:  d

local:  a b c      remote:  c
local:  abc        remote:  abc

local:  "abc"      remote:  FAILS

Andrew,

Yes. nsxmlrpc takes a list and turns it into the appropriate xmlrpc format.

The other parts I will look at.

It is probably some improper list converion or quouting I did when I converted from TclXML to nsxml.

I'll look at it.

Thanks for testing.

Over at validator.xmlrpc.com there are serveral AOLserver implementations, I suspect they are nsxmlrpc, but noone else really has admitted to using it except Jerry Asher.

Is anyone else using or has anyone tried and had a problem with nsxmlrpc?

Andrew,

I think i fixed the -prefixed arg problem. xmlrpc_contruct was taking all args starting with - as a switch to define the variable type.

I think I see why your lists are getting screwed up.

Nsxmlrpc is not smart enough to know that you are returning a tcl list. It does not guess the xmlrpc datatype.

Now the handy part of xmlrpc and soap is that you can call the same proc locally or remotely. This fails with nsxmlrpc. The proc needs to return the proper switchs to build the xmlrpc datatypes.

You are returning [list a b c d]
What you need to do is return that as an xmlrpc datatype. ie:
return [list -struct [list a b c d]]

Another option is to build in datatype guessing into nsxmlrpc. This would be a great feature! There is probably some code that can do this in TclSOAP.

If you are serious about using xmlrpc in aolserver, I really think that TclSOAP will give you a better foundation to build on. Also, it gives you SOAP and XMLRPC in the same code.

It would need some tweaking to fix the problems with package require and aolserver, but it is certainly doable.

Hmmm - all a bit messier than I had been hoping! I just pumped a cvs version of ns_xmlrpc into my oacs dev server, and when I tried to validate the server with validator.xmlrpc.com all I got was a log full of:

no value for option "times1000 47000 times100 4700 times10 470"
    while executing
"xmlrpc_construct $value_id {} $data"

and similar garbage.

bin/nsd -V
AOLserver/3.4.2 (nsd_v3_r4_p2)
   CVS Tag:         $Name: nsd_v3_r4_p2 $
   Built:           Jun 30 2002 at 22:47:26
   Tcl version:     8.3
   Thread library:  pthread
   Platform:        linux

I'm using what appears to be the most current nsxml (1.4), and libxml2 2.4.10. Since I actually don't need the xml server it's not specifically a problem, but I haven't tried the client yet... Anyone got an easy idea to test the client? (ie. a public test server that works...)

I prefer the sount of TclSOAP, but at the moment recompiling aolserver with patches on the client server isn't particularly appropriate - and I'm in kind of a rush (as always).

Oh well - I'll let everyone know how I go. If things get real ugly I might even construct the XML by hand.. urgh...

Mark, I haven't gotten around to trying TclSOAP either, but why do you say that using it would require patching and recompiling AOLserver??

From whatI've read in other threads Andrew, AOLServer has some nasty shortcomings in loading Tcl packages. Having said that, I just found this thread:

https://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0005Bz&topic_id=11&topic=OpenACS

Where DaveB says it's not too hard - I'll give it a shot and let you know how I go..