Forum OpenACS Development: XML-RPC String limit

Is there a limit of chars to pass through xml-rpc?

I'm trying to send a xmlrpc request and the code breaks when the content has more than 99754 characters. Less than that it works fine, but for bigger contents it breaks the code at /packages/xml-rpc/tcl/xmlrpc-procs.tcl, at line 671.

ns_log debug "xmlrpc::invoke REQUEST: $xml"
if {[catch {set doc [xml_parse -persist $xml]} err_msg]} {
ns_log Notice "ERROR SDSAFAFE"
set result [xmlrpc::fault 1 "error parsing request: $err_msg"]

Plus,
Where do I find the content limit imposition within [xml_parse] ?

p.s. OACS and NGINX are properly configured with maximum size limits.

[12/Apr/2017:12:27:04][26465.7fa469234700][-conn:1contact:0-] Notice: Running ad_proc xmlrpc::invoke
[12/Apr/2017:12:27:04][26465.7fa469234700][-conn:1contact:0-] Notice:
[12/Apr/2017:12:27:04][26465.7fa469234700][-conn:1contact:0-] Notice: ERROR SDSAFAFE
[12/Apr/2017:12:27:04][26465.7fa469234700][-conn:1contact:0-] Error: xmlrpc::invoke: error parsing request: error "Unexpected end" at position 0
""
[12/Apr/2017:12:27:04][26465.7fa469234700][-conn:1contact:0-] Notice: checking entry from host_node_map ->
[12/Apr/2017:12:27:04][26465.7fa469234700][-conn:1contact:0-] Warning: ignore untrusted host header field:
[12/Apr/2017:12:27:04][26465.7fa469234700][-conn:1contact:0-] Notice: ignore non-existing or untrusted host header, fall back to 1c.1contact.ch
[12/Apr/2017:12:27:04][26465.7fa469234700][-conn:1contact:0-] Error: can't read "doc": no such variable
while executing
"xml_doc_free $doc"
(procedure "xmlrpc::invoke" line 46)
invoked from within
"xmlrpc::invoke $content"
("uplevel" body line 15)
invoked from within
"uplevel {
# /packages/xml-rpc/www/index.tcl
ad_page_contract {
Accept XML-RPC POST requests and processes them. GET requests are shown
lin..."
(procedure "code::tcl::/var/www/1contact/packages/xml-rpc/www/index" line 2)
invoked from within
"code::tcl::$__adp_stub"
("uplevel" body line 12)
invoked from within
"uplevel {

if { [file exists $__adp_stub.tcl] } {

# ensure that data source preparation procedure exists and is up-to-date
..."
(procedure "adp_prepare" line 2)
invoked from within
"adp_prepare"
invoked from within
"template::adp_parse [file rootname [ad_conn file]] {}"
(procedure "adp_parse_ad_conn_file" line 6)
invoked from within
"$handler"
("uplevel" body line 2)
invoked from within
"uplevel $code"
invoked from within
"ad_try {
$handler
} ad_script_abort val {
# do nothing
}"
invoked from within
"rp_serve_concrete_file [ad_conn file]"
(procedure "::nsf::procs::rp_serve_abstract_file" line 60)
invoked from within
"rp_serve_abstract_file "$root/$extra_url""
("uplevel" body line 2)
invoked from within
"uplevel $code"
invoked from within
"ad_try {
rp_serve_abstract_file "$root/$extra_url"
set tcl_url2file([ad_conn url]) [ad_conn file]
set ..."
called from rp_handler
POST http://1c.1contact.ch/RPC2/? referred by '' peer 127.0.0.1 user_id 0
post-data:

X-Forwarded-For: 201.6.135.96
Host: 127.0.0.1:8000
Connection: close
Content-Length: 100001
Accept-Encoding: gzip
Content-Type: text/xml
User-Agent: Python-xmlrpc/3.5

The invoked method? Do you mean the new ad_proc that I created (i.e. iurix_robot::parse_json or [xmlrpc::invoke] ?

I haven't post the actual content here. It was just an example. Plus, the new method has nothing yet but logs. It only prints the arguments within the file error.log. Furthermore, "<, &, >, etc" special tags are not missing.

Certainly, the xml received has no sintax problems. I split the huge request (i.e. the one with 100000 chars) into two requests of 50000 chars and both worked just fine.

The second argument of the new xmlrpc ad_proc is a json array. See bellow.

ad_proc -public ix_robot.parse_json {
token
jsonfile
} {
It parses the json sent through xmlrpc

} {
ns_log Notice "Running ad_proc ix_robot.parse_json ..."
ns_log Notice "$token"
ns_log Notice "JSON \n $jsonfile"

return "OK"
}

I believe the size limit constraint is related to XMLRPC Lib http://xmlrpc-c.sourceforge.net/doc/libxmlrpc.html, but I haven't learned yet how to handle it yet.

xml-rpc-proc.tcl#L648 has been properly updated but the error still remains. To avoid potential syntax errors, as a json file would contain, I isolated the problem. I've written a chunk to generate 100k X's appending to string and send it to RPC2.

The logs point to syntax errors in a TCL file which is well written. Furthermore, if the string has less than 95 thousand chars the requests work just fine.

-] Error: xmlrpc::invoke: error parsing request: error "Unexpected end" at position 0
""
[14/Apr/2017:02:32:06][1460.7ff7f68ea700][-conn:1contact:1-] Notice: error "Entity parsing error" at position 1903
" + '?locale='+l+'&return_url='+return_url;
        top.location.href=url;
    }
--Error-- /script
   script type="text/javascript" src="https://code.jquery.com/jquery-"
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Notice: checking entry 127.0.0.1 from host_node_map  
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Warning: ignore untrusted host header field: '127.0.0.1:8000'
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Notice: ignore non-existing or untrusted host header, fall back to 1c.1contact.ch
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Error: can't read "doc": no such variable
    while executing
"xml_doc_free $doc"
    (procedure "xmlrpc::invoke" line 38)
    invoked from within
"xmlrpc::invoke $content"
    ("uplevel" body line 15)
    invoked from within
"uplevel {
    # /packages/xml-rpc/www/index.tcl
ad_page_contract {
    Accept XML-RPC POST requests and processes them. GET requests are shown
    lin..."
    (procedure "code::tcl::/var/www/1contact/packages/xml-rpc/www/index" line 2)
    invoked from within
"code::tcl::$__adp_stub"
    ("uplevel" body line 12)
    invoked from within
"uplevel {

        if { [file exists $__adp_stub.tcl] } {

            # ensure that data source preparation procedure exists and is up-to-date
      ..."
    (procedure "adp_prepare" line 2)
    invoked from within
"adp_prepare"
    invoked from within
"template::adp_parse [file rootname [ad_conn file]] {}"
    (procedure "adp_parse_ad_conn_file" line 6)

I've pasted the results in the previous post. There's no errors within that JS code. Plus, I've removed that chunk of code and the log notice threw another chunk from the same TCL in a different part of the file. It seems that the chunk gets returned in the error.log as a side effect of the parsing execution.

-] Error: xmlrpc::invoke: error parsing request: error "Unexpected end" at position 0
""
[14/Apr/2017:02:32:06][1460.7ff7f68ea700][-conn:1contact:1-] Notice: error "Entity parsing error" at position 1903
" + '?locale='+l+'&return_url='+return_url;
top.location.href=url;
}
--Error-- /script
script type="text/javascript" src="https://code.jquery.com/jquery-";
[14/Apr/2017:02:32:06][1460.7ff7f70eb700][-conn:1contact:0-] Notice: checking entry 127.0.0.1 from host_node_map

#####
Chrome/40.0.2214.111 Safari/537.36 user_id 698 peer 127.0.0.1
[14/Apr/2017:17:11:58][3231.7fef5a8db700][-conn:1contact:0-] Notice: Running TCL script test_ws.tcl
[14/Apr/2017:17:11:58][3231.7fef5a8db700][-conn:1contact:0-] Warning: ns_httpopen POST http://1c.1contact.ch/RPC2/ d4 30 {?xml version="1.0"?<methodCall><methodName>ix_robot.parseJson</methodName><params><param><value><string>gtregivrwjvoiejviowvwgvwegvwgre</string></value></param><param><value><string>XXXXXXXXX</string></value></param></params></methodCall>} is deprecated. Use 'ns_http' instead!
[14/Apr/2017:17:11:58][3231.7fef5a0da700][-conn:1contact:1-] Notice: Running ad_proc ix_robot.parseJson ...
[14/Apr/2017:17:11:58][3231.7fef5a0da700][-conn:1contact:1-] Notice: gtregivrwjvoiejviowvwgvwegvwgre
[14/Apr/2017:17:11:58][3231.7fef5a0da700][-conn:1contact:1-] Notice: JSON
XXXXXXXXX
[14/Apr/2017:17:11:58][3231.7fef5a8db700][-conn:1contact:0-] Notice:
[14/Apr/2017:17:12:00][3231.7fef5a0da700][-conn:1contact:1-] Notice: CSP violation: {"csp-report":{"document-uri":"http://1c.1contact.ch/","referrer":"","violated-directive":"font-src 'self' data:","effective-directive":"font-src","original-policy"

Furthemore, see the results of ns_log Debug

####
ted. Use 'ns_http' instead!
[14/Apr/2017:17:16:16][3231.7fef5a0da700][-conn:1contact:1-] Notice: BEFORE DEBUG
[14/Apr/2017:17:16:16][3231.7fef5a0da700][-conn:1contact:1-] Error: xmlrpc::invoke: error parsing request: error "Unexpected end" at position 0
####

And the collateral effect of XML RPC parsing

######
[14/Apr/2017:17:16:16][3231.7fef5a8db700][-conn:1contact:0-] Notice: error "Entity parsing error" at position 2006
" + '?locale='+l+'&return_url='+return_url;
top.location.href=url;
}
--Error-- /script
script type="text/javascript" src="https://code.jquery.com/jquery-";
[14/Apr/2017:17:16:16][3231.7fef5a0da700][-conn:1contact:1-] Error: can't read "doc": no such variable
while executing
"xml_doc_free $doc"
(procedure "xmlrpc::invoke" line 39)
invoked from within
"xmlrpc::invoke $content"
("uplevel" body line 15)

ns_log notice "xmlrpc::invoke last chars: '[string range $xml end-30 end]'"

What we know now is that for whatever reason, an xml-rpc request with 100k size ends up as an empty string in xmlrpc::invoke. This means, that xmlrpc::get_content (called by the www/index.tcl returns empty). I would recommend to try to replace the body of xmlrpc::get_content by the single line (when you use NaviServer):

  return [ns_getcontent -as_file false -binary false]

Let's hope this helps.
-g

[1] http://cvs.openacs.org/browse/OpenACS/openacs-4/packages/xml-rpc/tcl/xml-rpc-procs.tcl?r1=1.9&r2=1.10