Forum OpenACS Development: Re: XML-RPC String limit

Collapse
3: Re: XML-RPC String limit (response to 2)
Posted by Iuri Sampaio on
Hi Gustaf,

The invoked method? Do you mean the new ad_proc that I created (i.e. iurix_robot::parse_json or [xmlrpc::invoke] ?

I haven't post the actual content here. It was just an example. Plus, the new method has nothing yet but logs. It only prints the arguments within the file error.log. Furthermore, "<, &, >, etc" special tags are not missing.

Certainly, the xml received has no sintax problems. I split the huge request (i.e. the one with 100000 chars) into two requests of 50000 chars and both worked just fine.

The second argument of the new xmlrpc ad_proc is a json array. See bellow.

ad_proc -public ix_robot.parse_json {
token
jsonfile
} {
It parses the json sent through xmlrpc

} {
ns_log Notice "Running ad_proc ix_robot.parse_json ..."
ns_log Notice "$token"
ns_log Notice "JSON \n $jsonfile"

return "OK"
}

I believe the size limit constraint is related to XMLRPC Lib http://xmlrpc-c.sourceforge.net/doc/libxmlrpc.html, but I haven't learned yet how to handle it yet.

Collapse
4: Re: XML-RPC String limit (response to 3)
Posted by Gustaf Neumann on
By the invoked method i meant the method named $method_name ... the suspicion was just triggered by the line number that you provided.

´What are the last 50 characters of the variable $xml in xmlrpc::invoke in line 648 [1]? To me, the error message "error parsing request: error "Unexpected end" at position 0" looks like a incorrect XML. have you checked, that the python library you are using does not truncate the output?

Was is the connection with libxmlrpc?
-g

PS: I've fixed the exception about the non-existing variable 'doc' [2], but this will not solve your problem.

[1] https://github.com/openacs/xml-rpc/blob/master/tcl/xml-rpc-procs.tcl#L648
[2] http://cvs.openacs.org/browse/OpenACS/openacs-4/packages/xml-rpc/tcl/xml-rpc-procs.tcl?r1=1.8&r2=1.9

Collapse
5: Re: XML-RPC String limit (response to 4)
Posted by Iuri Sampaio on
Gustaf,

xml-rpc-proc.tcl#L648 has been properly updated but the error still remains. To avoid potential syntax errors, as a json file would contain, I isolated the problem. I've written a chunk to generate 100k X's appending to string and send it to RPC2.

The logs point to syntax errors in a TCL file which is well written. Furthermore, if the string has less than 95 thousand chars the requests work just fine.

-] Error: xmlrpc::invoke: error parsing request: error "Unexpected end" at position 0
""
[14/Apr/2017:02:32:06][1460.7ff7f68ea700][-conn:1contact:1-] Notice: error "Entity parsing error" at position 1903
" + '?locale='+l+'&return_url='+return_url;
        top.location.href=url;
    }
--Error-- /script
   script type="text/javascript" src="https://code.jquery.com/jquery-"
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Notice: checking entry 127.0.0.1 from host_node_map  
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Warning: ignore untrusted host header field: '127.0.0.1:8000'
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Notice: ignore non-existing or untrusted host header, fall back to 1c.1contact.ch
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Error: can't read "doc": no such variable
    while executing
"xml_doc_free $doc"
    (procedure "xmlrpc::invoke" line 38)
    invoked from within
"xmlrpc::invoke $content"
    ("uplevel" body line 15)
    invoked from within
"uplevel {
    # /packages/xml-rpc/www/index.tcl
ad_page_contract {
    Accept XML-RPC POST requests and processes them. GET requests are shown
    lin..."
    (procedure "code::tcl::/var/www/1contact/packages/xml-rpc/www/index" line 2)
    invoked from within
"code::tcl::$__adp_stub"
    ("uplevel" body line 12)
    invoked from within
"uplevel {

        if { [file exists $__adp_stub.tcl] } {

            # ensure that data source preparation procedure exists and is up-to-date
      ..."
    (procedure "adp_prepare" line 2)
    invoked from within
"adp_prepare"
    invoked from within
"template::adp_parse [file rootname [ad_conn file]] {}"
    (procedure "adp_parse_ad_conn_file" line 6)
Collapse
6: Re: XML-RPC String limit (response to 5)
Posted by Gustaf Neumann on
i do no understand what you mean by "xml-rpc-proc.tcl#L648 has been properly updated": the committed change to rpc-procs refers to line 681 (see [1]), it will remove cause the exception "can't read doc" (the secondary bug), but the primary bug will stay. To fix the secondary bug, update the xml-rpc package (from cvs or github).

To debug the primary problem: insert after line 684 the statement

ns_log notice "xmlrpc::invoke last chars: '[string range $xml end-30 end]'"
and report the results to convince me, that the XML document with the 100k size is indeed correctly terminated. So far, i am not convinced.

[1] https://github.com/openacs/xml-rpc/blob/master/tcl/xml-rpc-procs.tcl#L681

Collapse
7: Re: XML-RPC String limit (response to 6)
Posted by Iuri Sampaio on
I downloaded and replaced the TCL file https://github.com/openacs/xml-rpc/blob/master/tcl/xml-rpc-procs.tcl. Curiously, my file was different than that one.

I've pasted the results in the previous post. There's no errors within that JS code. Plus, I've removed that chunk of code and the log notice threw another chunk from the same TCL in a different part of the file. It seems that the chunk gets returned in the error.log as a side effect of the parsing execution.

-] Error: xmlrpc::invoke: error parsing request: error "Unexpected end" at position 0
""
[14/Apr/2017:02:32:06][1460.7ff7f68ea700][-conn:1contact:1-] Notice: error "Entity parsing error" at position 1903
" + '?locale='+l+'&return_url='+return_url;
top.location.href=url;
}
--Error-- /script
script type="text/javascript" src="https://code.jquery.com/jquery-";
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Notice: checking entry 127.0.0.1 from host_node_map

Collapse
8: Re: XML-RPC String limit (response to 7)
Posted by Gustaf Neumann on
what is the result of the "nslog notice ..." statement of
https://openacs.org/forums/message-view?message_id=5358474 ?
Collapse
9: Re: XML-RPC String limit (response to 8)
Posted by Iuri Sampaio on
To knock your suspects off see bellow the xml request file of a successful attempt. Then, make X times "100000, to get a body with 100000 X's char and break xmlrpc.

#####
Chrome/40.0.2214.111 Safari/537.36 user_id 698 peer 127.0.0.1
[14/Apr/2017:17:11:58][3231.7fef5a8db700][-conn:1contact:0-] Notice: Running TCL script test_ws.tcl
[14/Apr/2017:17:11:58][3231.7fef5a8db700][-conn:1contact:0-] Warning: ns_httpopen POST http://1c.1contact.ch/RPC2/ d4 30 {?xml version="1.0"?<methodCall><methodName>ix_robot.parseJson</methodName><params><param><value><string>gtregivrwjvoiejviowvwgvwegvwgre</string></value></param><param><value><string>XXXXXXXXX</string></value></param></params></methodCall>} is deprecated. Use 'ns_http' instead!
[14/Apr/2017:17:11:58][3231.7fef5a0da700][-conn:1contact:1-] Notice: Running ad_proc ix_robot.parseJson ...
[14/Apr/2017:17:11:58][3231.7fef5a0da700][-conn:1contact:1-] Notice: gtregivrwjvoiejviowvwgvwegvwgre
[14/Apr/2017:17:11:58][3231.7fef5a0da700][-conn:1contact:1-] Notice: JSON
XXXXXXXXX
[14/Apr/2017:17:11:58][3231.7fef5a8db700][-conn:1contact:0-] Notice:
[14/Apr/2017:17:12:00][3231.7fef5a0da700][-conn:1contact:1-] Notice: CSP violation: {"csp-report":{"document-uri":"http://1c.1contact.ch/","referrer":"","violated-directive":"font-src 'self' data:","effective-directive":"font-src","original-policy"

Furthemore, see the results of ns_log Debug

####
ted. Use 'ns_http' instead!
[14/Apr/2017:17:16:16][3231.7fef5a0da700][-conn:1contact:1-] Notice: BEFORE DEBUG
[14/Apr/2017:17:16:16][3231.7fef5a0da700][-conn:1contact:1-] Error: xmlrpc::invoke: error parsing request: error "Unexpected end" at position 0
####

And the collateral effect of XML RPC parsing

######
[14/Apr/2017:17:16:16][3231.7fef5a8db700][-conn:1contact:0-] Notice: error "Entity parsing error" at position 2006
" + '?locale='+l+'&return_url='+return_url;
top.location.href=url;
}
--Error-- /script
script type="text/javascript" src="https://code.jquery.com/jquery-";
[14/Apr/2017:17:16:16][3231.7fef5a0da700][-conn:1contact:1-] Error: can't read "doc": no such variable
while executing
"xml_doc_free $doc"
(procedure "xmlrpc::invoke" line 39)
invoked from within
"xmlrpc::invoke $content"
("uplevel" body line 15)