Forum OpenACS Development: JSON Array & objects

Request notifications

Collapse
Posted by Iuri Sampaio on
Hi there,

Retrieving a JSON, as in (1), and parsing the result, as in (2) results in a huge and cluttered list os lists, as in (4)
I did even tried to convert it to an array, but it is still painful to manipulate.

Is there a clearer and more direct way to handle JSON results?

(1) set json [im_httpget $url]

(2) set parsed_result [util::json::parse $json]

(3) array set result_hash [lindex $parsed_result 1]

(4) List of lists
[31/Jan/2018:11:35:12][2066.7ffac156f700][-conn:evex:0-] Notice: _object_ {html_attributions {_array_ {}} results {_array_ {{_object_ {geometry {_object_ {location {_object_ {lat -33.8581392 lng 151.2098175}} viewport {_object_ {northeast {_object_ {lat -33.8567902197085 lng 151.2111664802915}} southwest {_object_ {lat -33.8594881802915 lng 151.2084685197085}}}}}} icon https://maps.gstatic.com/mapfiles/place_api/icons/bar-71.png id 8e980ad0c819c33cdb1cea31e72d654ca61a7065 name {Cruise Bar, Restaurant & Events} opening_hours {_object_ {open_now false weekday_text {_array_ {}}}} photos {_array_ {{_object_ {height 1134 html_attributions {_array_ {{Cruise Bar, Restaurant & Events}}} photo_reference CmRaAAAA7AvF1ITVw4Ik99bkK8TULBeYRkLW9EzbZjKn_OzS0UDld5VKeGWVA6kne9xZyOQ9jmZjmJcRIaqcBPBzjqMNlD61RZZgsZ_OFe_wCAInrRcaObqqhUS3AsBZXumqiC4gEhA7JElNjdAfFANdvKycnmykGhQtBDlNmvG2IqRoYKiOvqo5dzqDcQ width 2048}}}} place_id ChIJi6C1MxquEmsR9-c-3O48ykI price_level 1 rating 3.9 reference CmRRAAAA14SKShVcPvzOXip11-6N-69uE43uIcmxQuBh-0KZ32FoXY7QMD9ceLoZqYwWK2tDrlwrv5aNA6Z40JRO8Q-aL3GV5XmT1DuUOfyCB5TaV8j4NxVbUmsVSuaufvg0hVOrEhB0_hqJIkF_RJgzBwfybof0GhSzKWNcFTWa_w12ef0CJeq4Vy2UdA scope GOOGLE types {_array_ {bar restaurant food point_of_interest establishment}} vicinity {Circular Quay West, Overseas Passenger Terminal, Sydney}}} {_object_ {geometry {_object_ {location {_object_ {lat -33.8610965 lng 151.2097834}} viewport {_object_ {northeast {_object_ {lat -33.8597475197085 lng 151.2111323802915}} southwest {_object_ {lat -33.8624454802915 lng 151.2084344197085}}}}}} icon https://maps.gstatic.com/mapfiles/place_api/icons/generic_business-71.png id 9ea7c77cb181b1f33d19c9d76121fcc6d5246ad8 name {Australian Cruise Group Circular Quay} opening_hours {_object_ {open_now false weekday_text {_array_ {}}}} photos {_array_ {{_object_ {height 1152 html_attributions {_array_ {{Australian Cruise Group Circular Quay}}} photo_reference CmRaAAAAH8Z4q3kIdWjfcptt6hySA0uuWFttlqL3cgl17sBjCaxH9FDk4vSccye3SpW3

Collapse
2: Re: JSON Array & objects (response to 1)
Posted by Iuri Sampaio on
As JSON results may vary a lot, it seems there will be customization for every kind of result. Roughly, there will be a combination of "array sets" and "foreaches", which can be put into a recursive ad_proc.

Not sure if that will solve all cases, but at least it works for GoogleMapsAPI :: nearbies places.

Best wishes,


set parsed_result [util::json::parse $json]

array unset json_hash
array set json_hash [lindex $parsed_result 1]
array set json_hash $json_hash(results)
...

if {[info exists json_hash(_array_)]} {
set json_array $json_hash(_array_)
foreach array_elem $json_array {
set obj [lindex $array_elem 0]
set json_list [lindex $array_elem 1]

if {$obj eq "_array_" } {
# here it goes a recursive call
...
}
}
}

Collapse
3: Re: JSON Array & objects (response to 1)
Posted by Gustaf Neumann on

For processing complex JSON data, have a look at tDOM 0.9.0 (http://tdom.org/, manual) or newer. These new versions of tDOM support among other nice things JSON parsing. One can do e.g.:


set json {{"string":"bar","number":1,"boolean":true,"array":[1,2,3],"object":{"foo":"bar","baz":"boo"}}}
dom parse -json $json doc
...do something with $doc ...

to parse a JSON structure and use later XPath on the root node of $doc to extract whatever data you are interested is. This is very handy, especially with deeper nested structures.

Collapse
4: Re: JSON Array & objects (response to 3)
Posted by Iuri Sampaio on
lol! Let me do my homework then! rsrs
Collapse
5: Re: JSON Array & objects (response to 3)
Posted by Gustaf Neumann on

Have a look at rl_json, which supports parsing, accessing, iterating, ... and templating for JSON, very neat!

It also looks performance-wise very good (a comparison with other approaches from the RublyLane people):

-- parse-1.1: "Parse a small JSON doc and extract a field" --------------------
                   | This run
    old_json_parse |  241.595
     rl_json_parse |    5.540
       rl_json_get |    4.950
           yajltcl |    8.800
rl_json_get_native |    0.800

rl_json is available from https://github.com/RubyLane/rl_json
-g

Collapse
6: Re: JSON Array & objects (response to 5)
Posted by Iuri Sampaio on
Gustaf,

I've installed rl_json and called it into NS environment, as in:

package require rl_json

Then, it returned errors related to librl_json0.9.11.so
Error: couldn't load file "/usr/local/ns/lib/rl_json0.9.11/librl_json0.9.11.so": /usr/local/ns/lib/rl_json0.9.11/librl_json0.9.11.so: undefined

It seems TCL 8.6 is required to run rl_json. However, I installed Naviserver using install-ns scripts (from your GIT repository), and the script installs TCL 8.5.19 from tarball.

Now, I need to remove TCL 8.5.19 manually, and then install TCL8.6, using apt-get this time :)

Will I run into problems with NS, removing TCL 8.5.19 and installing TCL 8.6? I believe NS won't be aware of TCL installation.

If so, I wonder if I could just amend install-ns script, replacing TCL 8.5.19 with TCL 8.6, then I'd simply run install-ns again, overwriting everything. And "Voià!"

What do you think?

Best wishes

[08/Mar/2018:08:00:27][1877.7f92267af700][-conn:evex:0-] Notice: ignore non-existing or untrusted host header, fall back to

[08/Mar/2018:08:00:27][1877.7f92267af700][-conn:evex:0-] Error: couldn't load file "/usr/local/ns/lib/rl_json0.9.11/librl_json0.9.11.so": /usr/local/ns/lib/rl_json0.9.11/librl_json0.9.11.so: undefined symbol: Tcl_NREvalObj
while executing
"load /usr/local/ns/lib/rl_json0.9.11/librl_json0.9.11.so rl_json"
("package ifneeded rl_json 0.9.11" script)
invoked from within
"package require rl_json "
("uplevel" body line 44)
invoked from within
"uplevel {
ad_page_contract {

} {
{token}
{content_type}
{item_type ""}
{event_types ""}
{location ""}
{neighbourhoods ""}..."
(procedure "code::tcl::/var/www/evex/packages/evex-rest/www/itemsListReq..." line 2)
invoked from within
"code::tcl::$__adp_stub"
("uplevel" body line 12)
invoked from within
"uplevel {

if { [file exists $__adp_stub.tcl] } {

# ensure that data source preparation procedure exists and is up-to-date
..."
(procedure "adp_prepare" line 2)
invoked from within
"adp_prepare"
invoked from within
"template::adp_parse [file rootname [ad_conn file]] {}"
(procedure "adp_parse_ad_conn_file" line 6)
invoked from within
"$handler"
("uplevel" body line 2)
invoked from within
"uplevel $code"
invoked from within

Collapse
7: Re: JSON Array & objects (response to 6)
Posted by Steve Manning on
I took a slightly different approach, installing rl_json 0.9.7 which avoids the TCL 8.6 Tcl_NREvalObj problem and allows me to use it with TCL8.5. Seems to be working OK so far.

I use

package require rl_json
namespace path {::rl_json}

To get the json commands into the namespace.

Hope that helps.

Steve

Collapse
8: Re: JSON Array & objects (response to 7)
Posted by Iuri Sampaio on
It seems TDOM >= 0.9 does JSON. It parses into a DOM tree, serializes that back to the same JSON (infoset), preserving data type, order of object members, even non unique object members.

Updating TDOM seems to be an easier solution.

I will install both libraries ! rsrs
It will take more time, but it will be a good exercise to recycle skills!

Thanks Steve

Collapse
9: Re: JSON Array & objects (response to 8)
Posted by Iuri Sampaio on
Collapse
10: Re: JSON Array & objects (response to 9)
Posted by Iuri Sampaio on
Steve,

It turns out that tDOM 0.9 also depends on TCL 8.6. Thus, I solved the problem, amending install-ns.sh script to install tdom 0.9, tcl 8.6, and of course NS 4.99.16.

Furthermore, upgrading TCL to 8.6 was an old task that I had postponed from a while ago, because of an issue that not only myself but also Michael Aram had. He's posted here in OACS Forum, regarding [ns_return 200 text/plain ${v}v]

i.e. "Ain’t No ☀shine", https://openacs.org/forums/message-view?message_id=5379887

@Gustaf, rl_json is just beautiful! Thanks for the tip Gustaf!

Best wishes

Collapse
11: Re: JSON Array & objects (response to 10)
Posted by Steve Manning on
I suppose I'm only delaying the inevitable :o)

Yes, rl_json is cool. I especially like the templating, the power of which I didn't appreciate until I started to use it.

Steve

Collapse
12: Re: JSON Array & objects (response to 10)
Posted by Iuri Sampaio on
Gustaf,

As you're the author of most of install-ns.sh script, If you allow me, I can share it on my Github repository.

I also made a few more tweaks to install systemctl on Debian 8 and Ubuntu 14.04(lts)

Best wishes,

Collapse
13: Re: JSON Array & objects (response to 12)
Posted by Gustaf Neumann on
iuri, since a few months, install-ns installs per default Tcl 8.6.8. Can it be that you have downloaded the script a while ago without updating it later?
Collapse
14: Re: JSON Array & objects (response to 13)
Posted by Gustaf Neumann on
tdom 0.9.0 works at least with Tcl 8.5.19 (I've just compiled it iin this combination), i would expect it to be as well compatible with Tcl 8.4 and 8.3.

Although we had recently problems on one site with tdom 0.9, it turned out finally that the same problems were already in tdom 0.8.3. I think, tdom 0.9.0 can be recommended in general. The updated version of install-ns.sh contains by default support for 0.9.0.

Concerning Tcl 8.6: since Tcl 8.6.7 there are no known issues in connection with OpenACS.

Collapse
16: Re: JSON Array & objects (response to 14)
Posted by Iuri Sampaio on
Since I've use tDOM only on XML-RPC, I've decided to try and learn rl_json.

Best wishes,

Collapse
17: Re: JSON Array & objects (response to 16)
Posted by Gustaf Neumann on
OpenACS uses tdom for many tasks via the xml_* interface. Examples are .xql and .info fiiles.
Collapse
18: Re: JSON Array & objects (response to 17)
Posted by Iuri Sampaio on
Yes, I've noticed that!

In fact, that's why I'd rather use it.

JSON's format has become more and more popular, as for website and webservice's integration.
I realized we have very few ad_procs related to JSON. Unfortunately, differently from XML-RPC, JSON is very much encoding dependent, which makes it hard to maintain, even harder to build API standard procs, for community-wide contribution.

Best wishes,

Collapse
15: Re: JSON Array & objects (response to 13)
Posted by Iuri Sampaio on
Yes, it could be.