Forum OpenACS Development: Re: nsdbi, nsdbipg

4: Re: nsdbi, nsdbipg (response to 3)
Posted by Antonio Pisano on
I've read and agree with your considerations.

::xo::db::sql would be just the perfect place to add the new interface, as they would be transparent.

I personally wouldn't put xotcl api into the core, but rather "wrap the wrappable" in a cost-benefit fashion for the core packages and let those who want the best performances "do the leap" to the ::xo api.

5: Re: nsdbi, nsdbipg (response to 4)
Posted by Gustaf Neumann on
i'm not sore sure about the wrapping and its expected benefits on the longer term. The closet semantics between db_* and dbi_* have probably dbi_1row/dbi_0or1row and db_1row/db_0or1row. Without a deep look i see the following differences
  • dbi_* has no -dbn, but "-db" in dbi-* is probably wrappable
  • dbi_* has no -cache_key and -cache_pool
  • the option "-bind" is probably sufficient compatible
  • the db_* option "-column_array" is probably sufficient compatible with the differently named "-array" in dbi_*
  • the db_* option "-column_set" is not documented, but reading from the source, this is not implemented in dbi
  • the dbi_* option "-timeout" is not available for nd_db
  • the dbi_* option "-db" is not available for nd_db
  • the dbi_* interface has no .xql support
  • the developer support output is missing for dbi, so
    one needs here as wrapper or some sandwiching;
    wrapper require uplevel-ing for bindvars.
  • dbi* has no full sql statement (with variables substituted), since there is no such thing in dbi
  • Interactions with the db_* transcation management are not clear for me, for reading operations, that should not be an issue worse that the current situation

Many of the mentioned options are seldomly used, the developer support should be done anyhow, some more issues will popup probably.

If one overwrites the two db_* functions by a version based on dbi, this "wrapper" function has to support all features used in any obscure niche in OpenACS. Furthermore, one has to install in the long range both drivers in OpenACS, since the hope to make the code fully compatible is very little. Therefore if one "recommends" to install both drivers, the code becomes harder to maintain and to test.

By moving towards higher abstractions, one has more freedom in the implementation, one can mark lower-level access as deprecated and there is a clear development direction. Also, XOTcl's mixins could be used as an extension mechanism not requiring to overwrite existing functions physically.

Antonio, what are your concerns about using ::xo::db::sql in the core? The interface and implementation is rock-stable, these functions are used in the largest OpenACS sites, xotcl is a requirement of every OpenACS installation, all the contributors of the last two years are knowledgeable in xotcl.

all the best

6: Re: nsdbi, nsdbipg (response to 5)
Posted by Antonio Pisano on
I was forgetting some OpenAcs features like background delivery leverage xotcl, so I didn't think xotcl-core was a requirement for the minimal OpenACS installation, but just commonly present. This was one of the concerns.

I fully trust xotcl api, and think it gives advantages to developers, mainly related to flexibility and code reuse, but is also a different paradigm, requiring quite some cognitive load to people used to current codebase (here in Italy, most of the programming I've seen was procedural). Of course, this is not a big problem if changes happen "under the hood" in the api.

This said, it is right to move on when this gives improvement for performances and capabilities. xotcl-core is out there since a while now, so eventually a wider adoption into core codebase needs to be considered.

7: Re: nsdbi, nsdbipg (response to 5)
Posted by Gustaf Neumann on
i did some more tests and changed all db-reading operations of a warmed up xowiki instance start page to use partly new wrapper functions using the dbi interface, and using as well the xo::db::sql call for permissions as sketched above.

The total performance improvement as measured with a small instance by the developer support is more than 20%, which is quite good. To make the migration easier, i've extended the nsdbi module; not sure whether all my changes should go into the nsdbi module.

8: Re: nsdbi, nsdbipg (response to 7)
Posted by Jim Lynch on
heya Gustaf, curious... when you did the testing, did each iteration include preparing a statement? ordid you prepare a statement (say) at openacs instance startup time, and use that statement?
9: Re: nsdbi, nsdbipg (response to 8)
Posted by Gustaf Neumann on
the prepare statements are performed by dbi implicitly and their results are cached by dbi. For the xowiki test most prepares operations (maybe all) were performed after startup, but nevertheless most queries could already use a prepared statements (e.g. permissions). From this point of view, the results are realistic.
10: Re: nsdbi, nsdbipg (response to 7)
Posted by Gustaf Neumann on uses now an updated xotcl-core infrastructure, xowiki runs now fully on dbi_*