Forum OpenACS Development: Re: From database to JSON

Collapse
4: Re: From database to JSON (response to 3)
Posted by Brian Fenton on
I guess it would depend on how you plan to generate the JSON. For example, last time I looked, on Windows we were unable to use rl_json, so choosing that could be a problem for us. But rl_json's template approach is very appealing, so that would be a pity not to use. I see also Neophytos has his tjson library, which I haven't looked at. OpenACS has always had a great history of staying close to the database, so maybe it is worth taking a look at how Postgres returns JSON and maybe integrating with that?

I think it's tricky to make it more simple than my example, which still keeping it powerful enough to handle complex cases. If you look at something like Symfony's JsonResponse class, it sounds great, but really they're just moving the pain point somewhere else (to making sure your data is already structured properly before calling JsonResponse) https://symfony.com/doc/current/controller.html#returning-json-response

just some random (probably not very helpful) thoughts
Brian
PS we discussed some of this previously here https://openacs.org/forums/message-view?message_id=3854601 and Malte raised a very interesting discussion here https://openacs.org/forums/message-view?message_id=5452649

Collapse
5: Re: From database to JSON (response to 4)
Posted by Neophytos Demetriou on
Hi Brian,

Regarding tjson library: it does not support Windows at the moment. It should not be too much work to get it to run on Windows but I don't have the setup yet - I will try and let u know. tjson works on Linux and MacOS at the moment.

Coming to the question at hand. I think it is the responsibility of the database or its driver to produce JSON or an intermediate format (because there are no types in TCL). This is what the AWS DynamoDB and the MongoDB drivers do (not 100% sure about the latter). They return an intermediate format that tjson can use in both cases. The reason this needs to happen in the database or its driver is because they are aware of the types already.

... I have to run, I will follow up later.

Collapse
7: Re: From database to JSON (response to 5)
Posted by Brian Fenton on
Hi Neophytos

good to see you still around! I accept your point about the responsibility of the database, and the importance of types - very important points. However there will be occasions when there is a need to maybe apply a TCL API to the data before sending it to the client - this may not be achievable from the database. Flexibility is key here I think.

Brian

Collapse
8: Re: From database to JSON (response to 7)
Posted by Neophytos Demetriou on
tjson supports manipulation commands even though there is no create command (I will add it). Here is how your example looks like with tjson with the existing commands:


set sql "select user_id, username,  last_visit from users"
set json_arr [::tjson::parse {[]}]
db_foreach get_users $sql {
   ::tjson::add_item_to_array $json_arr \
        [list M \
                [list \
                        user_id [list N $user_id] \
                        username [list S $username] \
                        last_visit [list S $last_visit]]]
}
doc_return 200 application/json [::tjson::to_pretty_json $json_arr]
ad_script_abort

And here is what you get:


[
  {
    "user_id": 0,
    "username": "guest",
    "last_visit": ""
  },
  {
    "user_id": 759,
    "username": "test",
    "last_visit": "2023-07-26 09:27:37.590777-04"
  }
]

M means it is an object, N means it is a number, and S means it is a string. tjson also supports the triple notation that is used in mongodb driver (documentation for that feature is still pending).

Collapse
9: Re: From database to JSON (response to 8)
Posted by Brian Fenton on
This looks very good. I particularly like the type support, which is badly missing in util::json
Collapse
10: Re: From database to JSON (response to 9)
Posted by Neophytos Demetriou on
You can see more object or array manipulation commands in this example.

I will add a create command though. Thanks for that.

Collapse
11: Re: From database to JSON (response to 9)
Posted by Antonio Pisano on
tDOM also offers explicity JSON type support, see e.g. jsonType at [1]

One advantage of using tDOM is that OpenACS already depends on it, so there is no extra library to load. Plus, cross-platform compatibility is already sorted out.

[1] http://tdom.org/index.html/doc/trunk/doc/domNode.html

Collapse
Posted by Neophytos Demetriou on
I see your point but it does not support arrays, e.g. [1,2,3] or ["a","b","c"]. tdom does not parse it into anything. I tried it before I built tjson.
Collapse
Posted by Antonio Pisano on
I have just tried this in a development shell:

dom parse -json {[1,2,3]} test
$test asJSON

-->

[1,2,3]

so it seems that the parsing has worked. Can you provide an example where tDOM fails to parse a valid JSON?

Ciao

Collapse
Posted by Neophytos Demetriou on
I stand corrected.
Collapse
Posted by Neophytos Demetriou on
Another way is to write the result to a csv and then exec python to convert it to json. The following line should work:

set result [exec -ignorestderr python3.10 -c "import csv, json, sys; print(json.dumps(\[dict(r) for r in csv.DictReader(sys.stdin)\]))" < /path/to/data.csv]
Collapse
Posted by Neophytos Demetriou on
Can you provide an example where tDOM fails to parse a valid JSON?

Please download this file: https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json

and then run:


package require tdom
set fp [open "spec3.json"]
set data [read $fp]
close $fp
puts [time "dom parse -json $data test" 1

I get: error "JSON syntax error" at position 15

tjson runs fine with it:


package require tjson
set fp [open "spec3.json"]
set data [read $fp]
close $fp
puts [time "::tjson::parse $data" 1]

=> 18316 microseconds for tjson

It's probably something that I am doing wrong. Please let me know if you have better luck with it.

Collapse
Posted by Antonio Pisano on

Hi Neophytos,

I have tried the following on my test vanilla instance:

set fp [open "/tmp/spec3.json" r]
set data [read $fp]
close $fp
lappend results [time {dom parse -json $data test} 1] [$test asJSON]

And the result was:

{23639 microseconds per iteration} {{"components":{"schemas":... (truncated)

The problem was the double quotes in

"dom parse -json $data test"

the $data JSON is expanded into the command string and won't be valid anymore.

Ciao

Antonio

Collapse
6: Re: From database to JSON (response to 4)
Posted by Antonio Pisano on
I see I have also "contributed" to the conversation in the past! Funny how we did not get much further from there more than 10 years ago 😊