Forum OpenACS Development: Towards OpenACS 5.9
Below is my preliminary agenda for OpenACS 5.9. Some of the mentioned changes are already implemented and tested. The mentioned changes affect only non-functional properties of OpenACS (performance, storage requirements).
If some of you have more things in your sleeves which is already implemented or which you can implement in the near future, please add to this thread.
Slimming pg sql core:
The OpenACS data model for PostgreSQL has many places, where dependent tuples are deleted manually instead of using proper foreign key constraints. This happens in acs_object__delete() but as well on other types such as the content repository types.
My interpretation of this implementation is that ancient version of PostgreSQL (most likely a while before pg8) had a bug in the handling of referential integrities (this bug is sometimes referred to as the RI bug) which made manual deletions and extra triggers necessary. Since pg7 is not supported by OpenACS, it is now time for a cleanup in this respect. These changes will make the code simpler and faster.
tree_sortkeys vs. recursive queries:
OpenACS 5.8.0 (aug 2013) brought permission lookup based on recursive queries (part of PostgreSQL since 8.4, which had end-of-live in July 2014). Most of the permission code is based since that time on recursive queries and is significantly faster than before. There are still a few places uses the (huge) materialized table acs_object_context_index (on some of our system this table have 150 mio and 180 mio tuples) which is - for pg - today mostly useless. It is still used e.g. for reverse lookups (who has permissions on that objects) but it would be nice to clean this further up, since such huge tables reduce scalability.
OpenACS maintains as well a tree_sortkey on acs-objects (also a huge table, with over 40 mio entries on some of our systems) which reflects the context_id hierarchy. This hierarchy is the core of the permission system and as well reflected (in addition to the hierarchy of privileges) in the acs_object_context_index. Interestingly the tree_sortkey (and max_child_sortkey) of acs-objects is nowhere used in acs-core (and at least not in the 80 packages in oacs-5-8). Dropping this keys is a means to achieve required scalability. When one is e.g. changing the context_id of an acs_object, one update operation will take on our production system 2.5 minutes. The problem with this is that during this time acs_objects is locked, bringing effectively all of the system to full halt. For our production system this means that we can't do such updates during the semester - which is very bad. Without the tree_sortkey, the same operation takes 10 seconds (with updating the acs_object_context_index, which is as well already mostly obsolete). Since this tree_sortkey is useless, and pg-only it is safe to drop it. I have done so a while ago on two of my smaller systems without any bad effects.
There are other places, where recursive queries could be used instead of tree_sortkeys, which would bring the pg and oracle data models closer to each other. Some of these are simple to replace, others are more work (e.g. the tree-sortkey in the content repository, which is as well somewhat redundant with cr_child_rels). Not sure, it is feasible to address this in OpenACS 5.9.
One other plan for OpenACS 5.9 for the xo* packages is to require XOTcl 2.0 (presented at the Tcl conference in 2011).
This is partially due to my losing an argument with ArsDigita oh, what, 15 years ago? For whatever reason, the arsDigita folk were unwilling to use proper foreign key constraints in their Oracle data models, and at that time, for simplicity of tracking ongoing rapid changes by aD, we based our PG datamodel strictly on the aD Oracle data model (helped by automated conversion tools that did much of the original conversion of ACS 4 to PG).
Then it turned out that the PG RI implementation (gulp, I helped with the first version) had serious locking contention issues making it very slow and even susceptible to deadlock. Another blow against the full use of SQL.
Over the years the RI implementation got more robust, but resources to maintain Oracle datamodels were slim (they were named "Don Baccus", mostly). So an overhaul wasn't really practical.
If you want to make the move, do it, it's the right thing to do and has been ever since I first proposed it to aD back in the ACS 3 days (ignoring PG bugs).
And quit supporting Oracle, geez.
Getting rid of tree_sortkey is the right thing to do, even though I wrote most of the code and came up with the datamodel design and key representation. It was before recursive queries, of course, and we always used "connect by" in Oracle (except for some bizarre Ben Adida code).
Quit supporting Oracle, geez. Oh, I already said that :) Given that almost no one uses OpenACS any more, why support it? I'm guessing support is weak and much is broken anyway.
This could be the right for its eventual userbase to let their voices be heard, because dropping Oracle is very inviting and would free a big deal of resources for people working on the project.
Maybe we should open a poll?
Getting rid of tree_sortkey is the right thing to do, even though I wrote most of the code and came up with the datamodel design and key representation. It was before recursive queries, of course, and we always used "connect by" in Oracle.
I'm not using Oracle with OpenACS anymore and, for the record, I have used recursive queries in PostgreSQL already (in a different context) and found them to be quite handy. That said, I think the tree_sortkey implementation is interesting as a concept as it implies different architectural/design constraints on the system itself i.e. to some extent it does not rely on the underlying RDBMS. Either way, keeping it or not, it would not make much of a difference for me.
Concerning Oracle: we for our part have no intention to drop Oracle in total. We would rather try not to damage it, but we do not have the resources for actively testing it. There are several OpenACS installations (or tailored clones) based on Oracle we don't want to be left alone. We should try to get in touch with these to assess their situation. A possible road-map could be to keep the oracle "support" as it is for 5.9 and reconsider it for 6.0.
If you can't test oracle, by definition it no longer works.
The Oracle-based people have, for over a decade, screamed at the mention of dropping Oracle but have never offered significant resources to keep support current (though one company, when they learned I was doing so for core single-handedly on a volunteer basis, did give me a small gift as a token of appreciation. Not enough for even one decent dinner but I did appreciate it).
"We should try to get in touch with these to assess their situation" - post a thread stating "we're talking about dropping Oracle again!!!!" and see if anyone screams. If not, you're in the clear. If so, tell them to step up and maintain the damned thing.
I'm guessing that the various legacy users of oracle-based .lrn, for instance, haven't upgraded in a long time. I'm 99% certain that's true of UNED (Emmanuelle's no longer there, and in her last years there had nothing to do with the maintenance of the .LRN installation there - despite encouragement to upgrade, last I heard they were stuck on a version that's about five years old).
So if you investigate I think you'd find that any existing oracle usage is legacy usage, and already orphaned.
I could be wrong but it would simplify what's left of the project, and given the lack of resources now that most of us have moved on, would seem like a smart thing to do.
All recent development on OpenACS passed through the uneasy task of considering Oracle people still around, so it would be better if such struggle could end up useful.
Andrew, what would you guys ngals think of having to move to postgres soonish?Not going to happen, not anytime soon, and maybe never, for a variety of reasons.
Folks, keep in mind that we don't use our RDBMS just for a website, and I suspect this is pretty common for other small firms out there.
In our particular case, we use Oracle extensively for other non-OpenACS non-web data management. We take advantage of OpenACS to view, help maintain, etc. that data. Essentially, in addition to the other ways we use it OpenACS provides a UI to that data. It all lives in the same Oracle instance, in different table spaces.
When we were starting with this back in 2001, PostgreSQL wasn't quite ready for our needs and ACS/OpenACS 4.x didn't yet support PostgreSQL anyway. I'm told that PostgreSQL now supports the SQL:2003 OLAP (windowing, lead/lag, etc.) features, which are incredibly useful when dealing with financial and other time-series data. So if we were starting from scratch today, perhaps we'd go with PostgreSQL rather than Oracle. But not necessarily, it depends.
Many third parties - like data vendors - still support Oracle but not PostgreSQL in ways that can matter. E.g, in my experience most data vendors essentially just give you obnoxious, giant, completely de-normalized CSV files. The better ones (or perhaps the ones with the most complicated data) pre-specify a reasonable relational schema, and often provide their own loader tool to keep it updated. We use one which only supports Oracle and MS SQL Server. (Could they add support for PostgreSQL if they wanted to, or even SQLite? Probably, but so far they haven't.)
With a data vendor like that, do you have to use their loader? Often, no, but if you don't, you get to handle all the work of processing their internal-format transactional update files multiple times every day, etc. etc. None of that is insurmountable. But it is an example of why it is valuable to have a nice toolkit that just so happens to support the RDBMS that you also want to use for other reasons.
Should you care about this? About our specific needs, perhaps not. But I hope it least illustrates how the "legacy" feature of good support for both Oracle and PostgreSQL may be of more value than the typical PostgreSQL-only OpenACS user appreciates.
By the way, Oracle has ended up working well for us, but we didn't know that it would ahead of time, nor could we have, since we didn't even know many of the details of our own needs yet. (Heck, I didn't even know that lead, lag, and all the other SQL:1999 and SQL:2003 OLAP functions existed until 2001, and now we critically depend on them.)
In that situation, which I think is pretty common, the right approach is to bias your choice of tools towards flexibility. So which would sound like a better bet to you? "We only support one RDBMS, period." Or, "We support two of the most powerful and full-featured RDBMSs available, PostgreSQL and Oracle."
Back then, I was certainly thinking about the fact that if we ran into some unexpected major roadblock with Oracle, we might be able to get around it with PostgreSQL. The same feeling in the opposite direction might be more common.
None, is my guess.
Back when I was maintaining Oracle support, did you ever offer to help pay for my time? Or, when the subject was ever raised, did you just post as you have posted now: "we need oracle support, don't drop it!"
"Should you care about this? About our specific needs, perhaps not. But I hope it least illustrates how the "legacy" feature of good support for both Oracle and PostgreSQL may be of more value than the typical PostgreSQL-only OpenACS user appreciates."
Sure, valuable to you and other freeloaders. But the reality is that for over a decade, almost all of the financial and programmer resource support has come from those using PG.
Concerning Oracle: we for our part have no intention to drop Oracle in total. We would rather try not to damage it, but we do not have the resources for actively testing it.That sounds like the right approach to me.
Dear Andrew, we of course understand the reasons you raised for using Oracle, but be aware that this kind of support community has provided is fragile and has had almost no testing whatsoever, so if your company plans to enjoy new development on OpenACS you should seriously consider coding or other forms of contributions.
Something I think I can suggest to people using Oracle around is trying a fresh install of current OpenACS and give feedbacks, or even better go for an upgrade in a safe environment and report issues. Somebody could offer him/herself to address them.
Anyway, at risk of repeating myself, without concrete contribution from Oracle userbase all of you will eventually be left alone.
I cannot talk for the distribution in general.
When the standard distribution (from the repositories) is compiled for Windows using MS tool chain, nsoracle does not even compile.
I am continously checking naviserver code
(http://sonarsrv.spazioit.com/dashboard/index/42350) as support to Gustaf in his development/maintainace activities
and, as far as I can see, new versions are only compiled (and tested?) only on specific platforms (e.g. Linux and not Windows).
Hope it helps,
This is partially due to my losing an argument with ArsDigita oh, what, 15 years ago? For whatever reason, the arsDigita folk were unwilling to use proper foreign key constraints in their Oracle data modelsReally? That seems weird and suprising. I don't remember hearing about that back then. I wonder what their reasons were, and if they made any sense.
The argument was that with proper RI operators, someone might enter the DB, delete a class of objects, and wipe out the whole contents.
Where by using "foreign key" without "on delete", manual attempts of this sort would be aborted by the db.
Now, as to why one might delete such things manually, at that time. aD's code was really horrible at cleaning itself up.
1. I believe that supporting recursive queries everywhere and removing the tree_sortkey plus the acs_objects table is a long overdue architectural change that can't wait any longer.
2. I think that dropping Oracle support would be unwise. At least three organizations/companies I know of use OpenACS on Oracle. And in case OpenACS doesn't support Oracle anylonger they won't adopt PostgreSQL, but they will just stop using OpenACS. I hope this point gets through.
3. I believe that one of the things OpenACS needs to survive is the capability of supporting mobile website development. Since the 21st of April Google has changed its ranking algorithm giving priority to websites passing the "Mobile Friendly" test. I don't know if it is already possible to develop a mobile friendly website using only OpenACS (and not using some external JS based front-end framework + OpenACS / TCL as backend service to get the data... this would be cheating). The fact is that none of the sites listed in this page http://openacs.org/community/companies/ passes Google's test.
Gustaf and his team are the only people I know actively developing on the core of the platform, they have their own schedule and agenda and simply don't need to support Oracle. The fact that this is still "kinda being done" is an extra.
Companies moaning about drop of support should enter the field if they really care about it, at least trying to keep their systems up to date. Seems like this is far from being done.
About point 2, maybe there are platform built with mobile in mind from the start, but I really think this is just a matter of web design. Could you please prove me wrong by pointing me to a platform that achieves good mobile experience without "cheating" as you said? I mean without JS on the frontend and some kind of backend.
1. all my Windows-OpenACS distributions have a working nsoracle. I have done this work mostly for Quest.
The code is given back to the community as sources are part of my distribution. I am afraid that very few people do compile aolserver and/or naviserver code on Windows and in there nsoracle (as it is at the moment) does not even compile...
Mobile-Friendly Test Compatibility:
There's no need super duper things like Angular.js or Ember.js or Sencha or whatever... a plain WordPress site would pass the test (as you said it is just a matter of HTML and CSS). Anyhow it seems to me that, at least as far as I know, no OpenACS based website passes that test...
Regarding point 2, I think we can agree on the fact that losing support for something is bad, but as Don Baccus pointed in the first place: who will do the job? [...] Companies should enter the field if they really care about it, at least trying to keep their systems up to date. Seems like this is far from being done.
I haven't been around for quite some time as I was building my own stuff due to the particularities of my situation for many years (including lack of any funding), however, Antonio has raised an interesting point in the sense that without support from companies who are presumably better off than others to contribute resources there's not going to be much interest to continue supporting Oracle. The other side of the story has to be about making it possible for those companies to help.
Gustaf and his team are the only people I know actively developing on the core of the platform, they have their own schedule and agenda and simply don't need to support Oracle. The fact that this is still "kinda being done" is an extra.
To be fair, let me say in advance that Gustaf and his team have been consistent in their support of OpenACS and NaviServer over the years. The problem is that we find ourselves in a situation where other people also have their own schedule and agenda that does not seem to bode very well with the status quo (for many years now).
There's no doubt in my mind that the situation has to change. I must say that I have tried to discuss this with Gustaf when I was about to release my code in github and we have come up with some ideas but the problem seems to be much broader than that.
It is not enough to just want things to change. We should be ready to change as well, all of us. And, usually people do that by making concessions. Without concessions, we'll remain in this situation and, quite possibly, end up parting ways for good next time, in other words we all lose.
Regarding mobile website development, please post any criticism about b-responsive theme at http://openacs.org/forums/message-view?message_id=4267310
The site http://lehrbuch-wirtschaftsinformatik.org/ is based on OpenACS 5.8 + xowf + custom bootstrap theme and works very nice on various devices.
Is it possible to also add a goal of html/UI clean-up for openacs core?
At least one admin feature uses nested tables for example.
This also would help:
1. address accessibility issues allowing openacs to be utilized by a more diverse pool of admins
2. provide openacs coded solutions by example in openacs-core
3. provide a style base more adaptable to site-wide css instead of the current collection of unique and "artistic" html-based styles.
4. provide a style base that other packages could rely on for consistent defaults
I'd imagine an html validator could be used to crawl to detect major issues, such as nested TABLE at:
In general, much html should be as generic as possible to validate regardless of DTD and allow styling changes to be changed by css.
One could take this to an extreme by reducing block tags such as P and BLOCKQUOTE to DIV, and reducing in-line tags such as B and STRONG to SPAN with a reference to an existing (or new) site-master.css class equivalent. This approach at html reduction would eliminate needing re-write most html, because spans and divs can be nested.
Maybe it is practical to write a script to identify and make substitutions. Gustaf has made scripts to crawl openacs for tcl code standardization.
From an openacs packages dir, an awkwardly written grep shows circa 110+ TABLE tags that could have attributes removed, since they conflict with any css styles:
grep -R "<table" acs-* | grep -v resources | grep -v "www/doc/" | grep -v "www/xml/" | grep -v "<table>" | grep -v "acs-core-docs" | wc -l
Yet, there are apparently 2042 cases of P open tag and 1260 of P closed tag, using this awkward grep:
grep -R "</p" acs-* | grep -v resources | grep -v "www/doc/" | grep -v "www/xml/" | grep -v "acs-core-docs" | wc -l
The update script cares for all know occurences (which are just a few such as file-storage or downloads) but not cases, where site specific code uses views depending on this attribute. If your code includes acs_objects.* in a view, please recreate it after running the update script.
Note that as well i have corrected all easy reachable HTML pages of acs-core and common packages (see http://openacs.org/xowiki/oacs-5-9-html-validity). Furthermore, many of the hrefs in the code were vulnerable against injection attacks, many of the property references in adp files suffered from double quoting (see guidelines http://openacs.org/xowiki/ADP_Files). \@-variables in adp-files without modifier are HTML-quoted and internationalized, one can omit one of the operations with the modifier ";noquote" or ";noi18n" and both via the modifer ";literal".
With these changes OpenACS 5.9 will be faster and much more secure than previous releases.
In short, all these operations are faster in oacs-5-9
- acs_object creation: 16% faster
- acs_object deletion: 6% faster
- cr_item creation: 6% faster
- cr_item deletion: 9% faster
As mentioned above, on sites with millions acs-objects, the improvements can be mich larger. An update operation on the context_id of an acs_object dropped on our live system from 2.5 minutes to 10 seconds (under this context_id is a large tree of objects). On operations with leaf objects, the difference will be much less. The changes are especially good for large sites and improve the scalability of OpenACS.
=== OpenACS 5.8.1 acs_objects created 100000, total 85945ms, 0.8595ms / object acs_objects deleted 100000, total 152611ms, 1.5261ms / object cr_items created 100000, total 206543ms, 2.0654ms / object cr_items deleted 100000, total 432189ms, 4.3219ms / object === OpenACS 5.9.0 acs_objects created 100000, total 72457ms, 0.7246ms / object acs_objects deleted 100000, total 143823ms, 1.4382ms / object cr_items created 100000, total 193504ms, 1.9350ms / object cr_items deleted 100000, total 395428ms, 3.9543ms / object
there will be a beta2 release of OpenACS 5.9, which i hope to be able to package on the weekend. Among a few small upgrade glitches i've updated the timezone reference data. The old data was 7 years old, we have now 43 more timezones and many fixes and updates.
For example, just recently, the checker complains about minified .js files, which are not regarded as source code anymore. So we have to get rid of all minified files in the debian distro (and all references to it in the source). This affects mostly the rich-text editors. It looks to me as if the best solution would be to pull out the editors from acs-core and to make it own packages, which can be installed via "install from repository" in the future (... but that will require same refactoring).
Another problem are is the aolserver package in debian, which is in a very old state, but the the debian maintainers are quite unresponsive concerning bug reports. Hector has started to look into the details, how to getting a NaviServer package into debian (there is some work from the people at qcode.co.uk we can build upon). But in any case, the cycles are rather slow.
Short message: we are on it, we are getting closer to a release.
+1 for a stronger modularization with respect to external libraries.
As I have suggested earlier, I sympathize with the idea of having dedicated OpenACS packages for external JS libraries such as jQuery, YUI2, Bootstrap, etc...
These could follow a naming conventions similar to acs-* such as e.g.
But certainly, better approaches than ajaxhelper to handle complex js frameworks like bootstrap/yui/ext/... etc are as well needed, but that might be candidates for theming packages, since mixing theses is though...