Forum OpenACS Q&A: Tcl function call to remote AOLserver via xmlrpc ?

Folks, for those of you more familiar with xmlrpc, soap, and their various implementations for AOLserver or Tcl, I have a question for you:

Say I have AOLserver or tclsh process A, and an entirely seperate AOLserver process B. Is there any way for me to have process A call a Tcl function running in process B?

Security issues and the like aside, it would be very cool if I could have process A make an arbitrary Tcl function call to B, and have it all work pretty much the way any normal Tcl function call would work, so that most of my code in A wouldn't even have to know that it's doing a remote rather than local function call.

Would XMLRPC or SOAP let me doing something like that? Or is there any any other tool that would make this feasible? And what would I have to do to make it work? Thanks!

Dave,<br>it would be very fine to have some practical <i>how to</i> about SOAP. I check periodically the nssoap project at SourceForge, but it is yet unfinished, while the ability to call a web service or to expose one is becoming a must!

Cool! Dave, writing wrapper procs doesn't sound like a problem, but, which of the various xmlrpc or soap packages would you recomend I use?

Since I definitely want to be able to make remote function calls to AOLserver from both AOLserver and a tclsh script, should I start by trying to use tclSOAP, Steve Ball's xmlrpc, or some other Tcl-based toolkit for everything, both in tclsh and AOLserver? Or should I just use ns_xml and ns_xmlrpc inside AOLserver, and pick some entirely separate tool to use from tclsh?

I'd greatly appreciate comments from people with the experience to compare/contrast the different implementations available...

Hi Dave,

The main issues are AOLserver's handling of Tcl namespaces and it's lack of support for "package require".

Could you please explain what the namespace problems are and specifically with the package? Thank you, Hanjo.

I just installed nsxml, ns_xmlrpc, and ran it through Userland's XML-RPC Validator. All the tests passed except the manyTypesTest.

Now, that test failed due to "Error: Poorly formed XML text, string constant is improperly formatted. (At character #213.)", because AOLserver didn't send back XML, the ns_xmlrpc code died and AOLserver sent back it's usual Server Error HTML message. The stack trace was:

[02/Jul/2002:20:02:24][17809.8][-conn0-] Error: value "Fri Jan 01 09:23:58 EWT 1904" 
for option "-date" is not a valid date 
(unable to convert date-time string "Fri Jan 01 09:23:58 EWT 1904")
value "Fri Jan 01 09:23:58 EWT 1904" for option "-date" is not a valid date 
(unable to convert date-time string "Fri Jan 01 09:23:58 EWT 1904")
    while executing
"xmlrpc_construct $data value $datum"
    ("foreach" body line 2)
    invoked from within
"foreach datum $value {
                set result [xmlrpc_construct $data value $datum]
                if {[llength $result]} {
                    r..."
    ("-array" arm line 4)
    invoked from within
"switch -- $option {
            -string -
            -text {
                ns_xml node setcontent  [xmlrpc_createContext $node $context]  $value
  ..."
    (procedure "xmlrpc_construct" line 18)
    invoked from within
"xmlrpc_construct $value_id {} $data"
    (procedure "xmlrpc_respond" line 8)
    invoked from within
"xmlrpc_respond $result"
    (procedure "xmlrpc_invoke" line 33)
    invoked from within
"xmlrpc_invoke $content"
    (procedure "xml_rpcdispatcher" line 12)
    invoked from within
"xml_rpcdispatcher"

The dateTime part of the request validator.xmlrpc.com is sending my server is this:

<value><dateTime.iso8601>19040101T08:23:58</dateTime.iso8601></value>

The validator1.manyTypesTest proc is taking its incoming dateTime variable and doing this to it:

[list -date [clock format $dateTime]]

And the Tcl error is coming from this code in xmlrpc_construct:

-date {
    if {[catch {
        clock format [clock scan $value] 
              -format {%Y%m%dT%H:%M:%S}
    } datevalue]} {
        return -code error 
                "value "$value" for option "$option" is not a valid date ($datevalue)"
    }
    ns_xml node new_child 
            [xmlrpc_createContext $node $context] 
            "dateTime.iso8601" $datevalue
}

So, I'm not sure what's going on there, but it looks to me like the clock command proc validator1.manyTypesTest is probably messed up.

Is the intention there to convert the incoming dateTime.iso8601 format into Tcl's internal integer representation? I don't think so, because then it'd be doing a clock scan, not clock format. And anyway, I tried the obvious fixes of changing it to either [list -date $dateTime] or [list -date [clock scan $dateTime]], and neither of those worked, they just changed the nature of the error.

So, what are these different bits of code supposed to be doing? Could someone knowledgeable please explain how these bits of ns_xmlrpc are supposed to work? I suspect it's simple but I don't see it...

Andrew, I just installed nsxmlrpc at www.thedesignexperience.org and it validated fine. This is a strange problem. Which AOLserver/Tcl versions are you running? I am using Tcl8.3 Aolserver 3.3+ad13 Maybe we can track this down. Here is the request and response for the manyTypesTest: Request

POST /RPC2 HTTP/1.0
User-Agent: Frontier/7.0.1 (WinNT)
Host: www.thedesignexperience.org
Content-Type: text/xml
Content-length: 686

<?xml version="1.0"?>
<methodCall>
	<methodName>validator1.manyTypesTest</methodName>
	<params>
		<param>
			<value><i4>12874</i4></value>
			</param>
		<param>
			<value><boolean>0</boolean></value>
			</param>
		<param>
			<value>Minnesota</value>
			</param>
		<param>
			<value><double>10202.33333333</double></value>
			</param>
		<param>
			<value><dateTime.iso8601>19040101T05:23:44</dateTime.iso8601></value>
			</param>
		<param>
			<value><base64>R0lGODlhFgASAJEAAP/////OnM7O/wAAACH5BAEAAAAALAAAAAAWABIAAAJAhI+py40zDIzujEDBzW0n74AaFGChqZUYylyYq7ILXJJ1BU95l6r23RrRYhyL5jiJAT/Ink8WTPoqHx31im0UAAA7</base64></value>
			</param>
		</params>
	</methodCall>

Response

TTP/1.0 200 OK
Set-Cookie: ad_session_id=1710038%2c0%20%7b582%201025663719%201D5A4B84D580197B61BA6EB179B0937355AAED37%7d; Path=/; Max-Age=1200
Content-Type: text/xml; charset=iso-8859-1
MIME-Version: 1.0
Date: Wed, 03 Jul 2002 02:15:19 GMT
Server: AOLserver/3.3.1+ad13
Content-Length: 514
Connection: close

<?xml version="1.0"?>
<methodResponse>
<params>
<param>
<value>
<array>
<data>
<value><i4>12874</i4></value>
<value><boolean>0</boolean></value>
<value>Minnesota</value>
<value><double>10202.33333333</double></value>
<value><dateTime.iso8601>19040101T05:23:44</dateTime.iso8601></value>
<value><base64>R0lGODlhFgASAJEAAP/////OnM7O/wAAACH5BAEAAAAALAAAAAAWABIAAAJAhI+py40zDIzujEDBzW0n74AaFGChqZUYylyYq7ILXJJ1BU95l6r23RrRYhyL5jiJAT/Ink8WTPoqHx31im0UAAA7</base64></value>
</data>
</array>
</value>
</param>
</params>
</methodResponse>

This is code I just checked out from the openacs.org CVS. Where did you get your code from?

Here's another weird ns_xmlrpc error. I set up a little page to have my AOLserver hit itself with an xmlrpc call. My little test/rpc.tcl page was:

set url {http://localhost:8075/RPC2}
append body [xmlrpc_call $url ns_info name]
ns_return 200 text/plain $body

From the ns_logs and stack trace below, it's pretty clear that this combination can't possibly work right. AOLserver seems to be handling and returning the rpc call ok, but once the xml gets to the client, the 'ns_xml node name' is 'methodResponse', which immediately breaks stuff because xmlrpc_parse is only set up to handle 'params', not 'methodResponse'.

These errors make it seem like I need to actually learn to use ns_xml and heavily debug ns_xmlrpc. But, the mysterious part is, everybody else seems to be saying that this same code works fine for them. What could be wrong?

The error from my test/rpc.tcl page:

[03/Jul/2002:04:05:34][28484.8][-conn0-] Debug: xmlrpc_callResponse: response: '<?xml version="1.0"?>
<methodResponse><params><param><value>AOLserver</value></param></params></methodResponse>
'
[03/Jul/2002:04:05:34][28484.8][-conn0-] Debug: xmlrpc_parse: ns_xml node name: 'methodResponse'
[03/Jul/2002:04:05:34][28484.8][-conn0-] Error: XMLRPC-ERROR remote procedure call failed: "invalid server response ()"
XMLRPC-ERROR remote procedure call failed: "invalid server response ()"
    while executing
"xmlrpc_callResponse $response"
    (procedure "xmlrpc_call" line 33)
    invoked from within
"xmlrpc_call $url ns_info name"
    invoked from within
"append body [xmlrpc_call $url ns_info name]"
    (file "/web/aol3-src/nsbbapi/www/test/rpc.tcl" line 6)
    invoked from within
"source $file"
    invoked from within
"ns_sourceproc cns0 {}"

Andew, It looks like the client code was not fixed to handle the newer versions of nsxml without a bug that returned the first child instead of root for ns_xml doc root $doc_id. Add this code right after the set response [ns_xml doc root $doc_id]

if {[ns_xml node name $response] != "methodResponse"} {
            set response [lindex [ns_xml node children $response] 0]
        }

I have checked this in, so you can also cvs update to test it. This code checks if the root node returned refers to the methodRepsonse node or the root node. If it refers to the root node, it goes to the next node to get the actual methodResponse. This should fix the code to work with any version of ns_xml. Should fix it.

Andrew,

Yes. nsxmlrpc takes a list and turns it into the appropriate xmlrpc format.

The other parts I will look at.

It is probably some improper list converion or quouting I did when I converted from TclXML to nsxml.

I'll look at it.

Thanks for testing.

Over at validator.xmlrpc.com there are serveral AOLserver implementations, I suspect they are nsxmlrpc, but noone else really has admitted to using it except Jerry Asher.

Is anyone else using or has anyone tried and had a problem with nsxmlrpc?

Hmmm - all a bit messier than I had been hoping! I just pumped a cvs version of ns_xmlrpc into my oacs dev server, and when I tried to validate the server with validator.xmlrpc.com all I got was a log full of:

no value for option "times1000 47000 times100 4700 times10 470"
    while executing
"xmlrpc_construct $value_id {} $data"

and similar garbage.

bin/nsd -V
AOLserver/3.4.2 (nsd_v3_r4_p2)
   CVS Tag:         $Name: nsd_v3_r4_p2 $
   Built:           Jun 30 2002 at 22:47:26
   Tcl version:     8.3
   Thread library:  pthread
   Platform:        linux

I'm using what appears to be the most current nsxml (1.4), and libxml2 2.4.10. Since I actually don't need the xml server it's not specifically a problem, but I haven't tried the client yet... Anyone got an easy idea to test the client? (ie. a public test server that works...)

I prefer the sount of TclSOAP, but at the moment recompiling aolserver with patches on the client server isn't particularly appropriate - and I'm in kind of a rush (as always).

Oh well - I'll let everyone know how I go. If things get real ugly I might even construct the XML by hand.. urgh...

From whatI've read in other threads Andrew, AOLServer has some nasty shortcomings in loading Tcl packages. Having said that, I just found this thread:

https://openacs.org/bboard/q-and-a-fetch-msg.tcl?msg_id=0005Bz&topic_id=11&topic=OpenACS

Where DaveB says it's not too hard - I'll give it a shot and let you know how I go..