Forum OpenACS Development: http POST X binary file

Posted by Iuri Sampaio on 01/17/21 01:13 AM

Hello there,

The following CURL request works fine, and I'm trying to send files, using either NS or OACS procs, to a specific endpoint. However, the file format has been an issue.

What would be the proper replacement to the following CURL post request?

curl -k -v -X POST -H "X-Auth-Token: 41fb3071-4947-48a-bf2a-e59e3062c2ff" -H "Content-Type: image/jpeg" --data-binary @/home/mixus/Work/Faces/man1.jpg http://ip:5000/4/storage/descriptors

I've tried both NS proc and OACS ad_proc. Documentation:
[ns_http] http://18.218.229.235/doc/naviserver/files/ns_http.html
and
[util::http::post] https://openacs.org/api-doc/proc-view?proc=util::http::post&source_p=1

I've also tried to read the file whether binary or not. Neither one worked
1. [open [ns_queryget upload_file.tmpfile] "rb"]
2. [open [ns_queryget upload_file.tmpfile] "r"]

I've also tried to send the files whether as the parameters -body or -body_files. None of them worked either
1.1. -body_file "[read $fp]"
1.2. -body "[read $fp]"

P.S. CURL and Postman client POST requests, both worked just fine. Please, See below the logs from a successfull request in Postman.

The source code snippets are described bellow

# Add Descriptor (portrait)
...
set req_headers [ns_set create]
ns_set put $req_headers "X-Auth-Token" "[parameter::get_global_value -package_key qt-lunaapi -parameter AccessToken -default ""]"
ns_set put $req_headers "Content-Type" "image/jpeg"
set url "${proto}://${domain}:${port}${path}descriptors"
set fp [open [ns_queryget upload_file.tmpfile] "rb"]

1. util::http:post
...
set res [util::http::post \
-headers $req_headers \
-url $url \
-timeout 60 \
-body "file=[read $fp]"]

2. ns_http
set res [ns_http queue \
-method POST \
-headers $req_headers \
-body_file [read $fp] \
-timeout 60 \
$url]

What would be the proper replacement to the following CURL post request?

Logs from Postman client request, HEADER and BODY contents. FILE format is binary!

[16/Jan/2021:20:59:11][10351.7efbf3d70700][-conn:qonteo:default:0:664-] Notice: Running REST debug upload
[16/Jan/2021:20:59:11][10351.7efbf3d70700][-conn:qonteo:default:0:664-] Notice: HEADER
: t0
[16/Jan/2021:20:59:11][10351.7efbf3d70700][-conn:qonteo:default:0:664-] Notice: HEADERS 11
[16/Jan/2021:20:59:11][10351.7efbf3d70700][-conn:qonteo:default:0:664-] Notice: Host dashboard.qonteo.com X-Real-IP 187.127.207.76 Connection close Content-Length 16925 X-Auth-Token 41fb3071-4947-48a-bf2a-e59e3062c2ff User-Agent PostmanRuntime/7.26.8 Accept */* Cache-Control no-cache Postman-Token b8ad0439-46eb-4321-bae2-f9583069aa53 Cookie {ad_locale="es_ES"; ad_session_id="27280007%2c0%2c0%2c1610841399%20{384%201610842599%2041B85ED469858F06E5E0A5B883624A7B5289E1EA}"} Content-Type image/jpeg

[16/Jan/2021:20:59:11][10351.7efbf3d70700][-conn:qonteo:default:0:664-] Notice: BODY
: \xc3\xbf\xc3\x98\xc3\xbf\xc3\xa0\xc0\x80\x10JFIF\xc0\x80\x01\x01\x01\xc0\x80H\xc0\x80H\xc0\x80\xc0\x80\xc3\xbf\xc3\x9b\xc0\x80C\xc0\x80\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05
...

2: Re: http POST X binary file (response to 1)

Posted by Iuri Sampaio on 01/17/21 01:40 AM

okay, it seems I had forgotten double quotes in -body switch 😊
as in :
...
-body "[read $fp]"
...

However, the request has failed still :(

3: Re: http POST X binary file (response to 1)

Posted by Iuri Sampaio on 01/17/21 02:01 AM

I noticed there's an extra header Content-type application/x-www-form-urlencoded, which comes by default, whether on ns_http or util::http:post

Perhaps that's what has caused the error. ( i.e. endpoint rejects requests from Naviserver/OACS.)

I thought default content-type would be overwritten once I had declared explicitly a new content-type as an extra header, within the custom snippet but it didn't. 😟

as in:

ns_set put $req_headers "Content-type" "image/jpeg"
...

set res [ns_http queue \
-method POST \
-headers $req_headers \
-body "[read $fp]" \
-timeout 60 \
$url]

set result [ns_http wait $res]
...

How do I remove that default header (i.e. Content-type application/x-www-form-urlencoded) from the request, leaving only Content-type image/jpeg ??

[16/Jan/2021:21:52:37][10351.7efbf356f700][-conn:qonteo:default:1:842-] Notice: Host dashboard.qonteo.com X-Real-IP 192.199.241.132 Connection close Content-Length 16925 X-Auth-Token 9fb6e731-b342-4952-b0c1-aa1d0b52757b Content-type application/x-www-form-urlencoded Content-type image/jpeg User-Agent NaviServer/4.99.19

4: Re: http POST X binary file (response to 3)

Posted by Gustaf Neumann on 01/17/21 05:09 PM

If you want to submit a POST request with a certain content-type to transmit a file "/tmp/foo.jpg", use:

set req_headers [ns_set create]
ns_set put $req_headers "Content-Type" "image/jpeg"
ns_http run -method POST -headers $req_headers -body_file /tmp/foo.jpg https://localhost:8443/upload

One can use as well the OpenACS API, where files can passed explicitly: util::http::post. The interface supports submission of multiple files. Probably, the data is sent via multipart/form-data, you have to double-check.

5: Re: http POST X binary file (response to 4)

Posted by Iuri Sampaio on 01/17/21 06:09 PM

Hi Gustaf,
I'm sorry, I believe I haven't expressed myself well in the previous post.

ns_http comes with the header "Content-type" "application/x-www-form-urlencoded" by default.

Thus, when I add the extra header "Content-Type" "image/jpeg", it gets doubled, instead of overwriting the default one

The reason for that is "ad_form context". That POST request happens in the middle of another POST request, while ad_form -on_submit event has been executed.

"ns_http run" snippet is placed still within the block -on_submit, the image is sent to a third/external site, right before the image gets stored in the ACS content repository.

Unfortunately, writing ns_http post request in a different file, avoiding context overlapping, then redirecting to the workflow/execution of the form, is not a solution. Ad_form has been used in the GUI, so that everything starts with the ad_form submission.

The external site functions as some sort of validation to the image being uploaded. Meaning, if the endpoint returns negatively
the users will be required to chose/use a valid image.

So, how would I remove "Content-type" "application/x-www-form-urlencoded"?

Do you see any other approach that could allow running this post request out of the ad_form context?

NS release is NaviServer/4.99.19 (tar-4.99.19) running

[17/Jan/2021:13:38:25][10351.7efbf3d70700][-conn:qonteo:default:0:2287-] Notice: Running REST debug upload
[17/Jan/2021:13:38:25][10351.7efbf3d70700][-conn:qonteo:default:0:2287-] Notice: HEADER
: t0
[17/Jan/2021:13:38:25][10351.7efbf3d70700][-conn:qonteo:default:0:2287-] Notice: HEADERS 8
[17/Jan/2021:13:38:25][10351.7efbf3d70700][-conn:qonteo:default:0:2287-] Notice: Host dashboard.qonteo.com X-Real-IP 192.199.241.132 Connection close Content-Length 16925 Content-type application/x-www-form-urlencoded Content-Type image/jpeg User-Agent NaviServer/4.99.19
[17/Jan/2021:13:38:25][10351.7efbf3d70700][-conn:qonteo:default:0:2287-] Notice: BODY
: \xc3\xbf\xc3\x98\xc3\xbf\xc3\xa0\xc0\x80\x10JFIF\xc0\x80\x01\x01\x01\xc0\x80H\xc0\x80H\xc0\x80\xc0\x80\xc3\xbf\xc3\x9b\xc0\x80C\xc0\x80\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05
: \x07\x07\x06\x08\x0c
: \x0c\x0c\x0b
: \x0b\x0b\x0d\x0e\x12\x10\x0d\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f\x17\x18\x16\x14\x18\x12\x14\x15\x14\xc3\xbf\xc3\x9b\xc0\x80C\x01\x03\x04\x04\x05\x04\x05\x09\x05\x05\x09\x14\x0d\x0b\x0d\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\x14\xc3\xbf\xc3\x82\xc0\x80\x11\x08\xc0\x80\xc3\xba\xc0\x80\xc3\xba\x03\x01\x11\xc0\x80\x02\x11\x01\x03\x11\x01\xc3\xbf\xc

6: Re: http POST X binary file (response to 1)

Posted by Iuri Sampaio on 01/17/21 06:43 PM

Wondering if that would change the environment context, thus to reset NS headers too, I've moved the whole custom chunk of code to a /tcl library file, wrapping the request post within a custom ad_proc however, ad_form context remained.

Currently, within on_submit block there's only a line calling the ad_proc, as in:

...
} -on_submit {
qt::lunaapi::upload_luna -file [ns_queryget upload_file.tmpfile]

db_transaction {
...

logs are bellow

[17/Jan/2021:14:33:55][28167.7efbf356f700][-conn:qonteo:default:1:25-] Notice: Running ad_proc qt::lunapai::upload_luna
[17/Jan/2021:14:33:55][28167.7efbf3d70700][-conn:qonteo:default:0:26-] Notice: HEADER
: t0
[17/Jan/2021:14:33:55][28167.7efbf3d70700][-conn:qonteo:default:0:26-] Notice: HEADERS 8
[17/Jan/2021:14:33:55][28167.7efbf3d70700][-conn:qonteo:default:0:26-] Notice: Host dashboard.qonteo.com X-Real-IP 192.199.241.132 Connection close Content-Length 16925 Content-type application/x-www-form-urlencoded Content-Type image/jpeg User-Agent NaviServer/4.99.19
[17/Jan/2021:14:33:55][28167.7efbf3d70700][-conn:qonteo:default:0:26-] Notice: BODY
: \xc3\xbf\xc3\x98\xc3\xbf\xc3\xa0\xc0\x80\x10JFIF\xc0\x80\x01\x01\x01\xc0\x80H\xc0\x80H\xc0\x80\xc0\x80\xc3\xbf\xc3\x9b\xc0\x80C\xc0\x80\x03\x02\x02\x03\x02\x02\x03\x03\x03\x03\x04\x03\x03\x04\x05\x08\x05\x05\x04\x04\x05
: \x07\x07\x06\x08\x0c
: \x0c\x0c\x0b
: \x0b\x0b\x0d\x0e\x12\x10\x0d\x0e\x11\x0e\x0b\x0b\x10\x16\x10\x11\x13\x14\x15\x15\x15\x0c\x0f\x17\x18\x16\x14\x18\x12\x14\x15\x14\xc3\xbf\xc3\x9b\xc0\x80C\x01\x03\x04\x04\x05\x04\x05\x09\x05\x05\x09\

7: Re: http POST X binary file (response to 1)

Posted by Iuri Sampaio on 01/17/21 07:20 PM

Good news!

Restarting Naviserver did the magic trick!

I believe there was some sort of cache, which was holding "Content-type" "application/x-www-form-urlencoded in the header. No idea.

So, The solution was: wrapping up the request POST (i.e. ns_http) within an ad_proc, which puts that action out of ad_form's context, then it cleans up NS headers.

Thanks, Gustaf, for triggering this enlightenment.

Best wishes,
I

8: Re: http POST X binary file (response to 7)

Posted by Gustaf Neumann on 01/17/21 08:49 PM

Iuri, to save your time: neither in NaviServer or in OpenACS is something i would call "cache"; and if your code works after the restart, it must have different causes to be found in your code, but not in the infrastructure.

Certainly, ns_sets (which are used for header fields) can be reused within a request for multiple usages/purposes. After a request, these are cleaned up automatically via "ns_set cleanup". So, if you use/reuse e.g. an ns_set for headers for multiple "ns_http" client requests, and you are always adding additional fields to it, you might end up with a request with multiple "content-type" header fields. So, the general recommendation is to use always fresh ns_sets for http-client requests (unless you know, what you do).

9: Re: http POST X binary file (response to 8)

Posted by Iuri Sampaio on 01/18/21 01:30 AM

Gustaf,
I have no idea what possible be causing this strange behavior in my installation.

I've upgraded to NaviServer/4.99.19 (tar-4.99.19) and acs-kernel-5.10.0d30. last week. Plus it's a fresh install. So it's too recent to get myself "forking around" anything in the core 😊

It could be NGINX cache. However, it happened only in this instance, among others, which are served by the same proxy.

Anyway, all it means is that I need more time to dig further into this problem and find the actual cause of it.

Best wishes,
I

10: Re: http POST X binary file (response to 9)

Posted by Gustaf Neumann on 01/18/21 09:41 AM

You don't have to dig deeper, but just in your own code. That was the good news.