Forum OpenACS Development: Re: From database to JSON

Collapse
7: Re: From database to JSON (response to 5)
Posted by Brian Fenton on
Hi Neophytos

good to see you still around! I accept your point about the responsibility of the database, and the importance of types - very important points. However there will be occasions when there is a need to maybe apply a TCL API to the data before sending it to the client - this may not be achievable from the database. Flexibility is key here I think.

Brian

Collapse
8: Re: From database to JSON (response to 7)
Posted by Neophytos Demetriou on
tjson supports manipulation commands even though there is no create command (I will add it). Here is how your example looks like with tjson with the existing commands:


set sql "select user_id, username,  last_visit from users"
set json_arr [::tjson::parse {[]}]
db_foreach get_users $sql {
   ::tjson::add_item_to_array $json_arr \
        [list M \
                [list \
                        user_id [list N $user_id] \
                        username [list S $username] \
                        last_visit [list S $last_visit]]]
}
doc_return 200 application/json [::tjson::to_pretty_json $json_arr]
ad_script_abort

And here is what you get:


[
  {
    "user_id": 0,
    "username": "guest",
    "last_visit": ""
  },
  {
    "user_id": 759,
    "username": "test",
    "last_visit": "2023-07-26 09:27:37.590777-04"
  }
]

M means it is an object, N means it is a number, and S means it is a string. tjson also supports the triple notation that is used in mongodb driver (documentation for that feature is still pending).

Collapse
9: Re: From database to JSON (response to 8)
Posted by Brian Fenton on
This looks very good. I particularly like the type support, which is badly missing in util::json
Collapse
10: Re: From database to JSON (response to 9)
Posted by Neophytos Demetriou on
You can see more object or array manipulation commands in this example.

I will add a create command though. Thanks for that.

Collapse
11: Re: From database to JSON (response to 9)
Posted by Antonio Pisano on
tDOM also offers explicity JSON type support, see e.g. jsonType at [1]

One advantage of using tDOM is that OpenACS already depends on it, so there is no extra library to load. Plus, cross-platform compatibility is already sorted out.

[1] http://tdom.org/index.html/doc/trunk/doc/domNode.html

Collapse
Posted by Neophytos Demetriou on
I see your point but it does not support arrays, e.g. [1,2,3] or ["a","b","c"]. tdom does not parse it into anything. I tried it before I built tjson.
Collapse
Posted by Antonio Pisano on
I have just tried this in a development shell:

dom parse -json {[1,2,3]} test
$test asJSON

-->

[1,2,3]

so it seems that the parsing has worked. Can you provide an example where tDOM fails to parse a valid JSON?

Ciao

Collapse
Posted by Neophytos Demetriou on
I stand corrected.
Collapse
Posted by Neophytos Demetriou on
Another way is to write the result to a csv and then exec python to convert it to json. The following line should work:

set result [exec -ignorestderr python3.10 -c "import csv, json, sys; print(json.dumps(\[dict(r) for r in csv.DictReader(sys.stdin)\]))" < /path/to/data.csv]
Collapse
Posted by Neophytos Demetriou on
Can you provide an example where tDOM fails to parse a valid JSON?

Please download this file: https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json

and then run:


package require tdom
set fp [open "spec3.json"]
set data [read $fp]
close $fp
puts [time "dom parse -json $data test" 1

I get: error "JSON syntax error" at position 15

tjson runs fine with it:


package require tjson
set fp [open "spec3.json"]
set data [read $fp]
close $fp
puts [time "::tjson::parse $data" 1]

=> 18316 microseconds for tjson

It's probably something that I am doing wrong. Please let me know if you have better luck with it.

Collapse
Posted by Antonio Pisano on

Hi Neophytos,

I have tried the following on my test vanilla instance:

set fp [open "/tmp/spec3.json" r]
set data [read $fp]
close $fp
lappend results [time {dom parse -json $data test} 1] [$test asJSON]

And the result was:

{23639 microseconds per iteration} {{"components":{"schemas":... (truncated)

The problem was the double quotes in

"dom parse -json $data test"

the $data JSON is expanded into the command string and won't be valid anymore.

Ciao

Antonio