Forum OpenACS Development: OACS 6 and beyond

Now that Oracle is officially being dropped, does it make sense to start using pgtcl in place of plpgsql?
I am not sure it does, but at least we will be lowering the cost of entry for new developers. Only haveing to know tcl and sql seems like a big win.

Also, what is the status of xotcl, I know it will be required, but is anything going to be redone, or just new stuff?

Hi, I don't understand because you say the following thing: Now that Oracle is officially being dropped. Can you help me please?. Thanks in advance.

Are you happy that there is a concensus among Openacs users that Oracle should be dropped? I, for one, think it's a backward step.

I can only tell for myself that I have a hard time providing changes to core packages due to the fact that I don't have the time to test the changes on Oracle. I think a couple of people feel the same.

To my knowledge though, an official TIP on that matter is still required, so maybe my wording was a little bit strong and premature. But this seems the direction in which we are steering.

Andrew, did you just volunteer yourself to be the "Oracle go-to guy" on OpenACS 6?

I mean, you are being paid by your company to develop on OpenACS/Oracle, is that correct?

Surely it is only in your best interest to contribute some time to the community to see this continue.

And for that matter, why not jump in right away to fix or port any/all the Postgres-only packages to work on Oracle?

I have not contributed code, but then again, I am not complaining about what is supported and what is not.

Yes, PostgreSQL users have been whining about of having to "support" Oracle for years. Fine, but also largely irrelevant.

It stops being irrelevant the day I stop saying "someday I will no longer support Oracle for you" and start saying "as of now I will no longer support Oracle for you". Pissing me off isn't going to do much to encourage me to support your platform for free in the future. Competent FOLLOWERS figure that out. Oh, financial support HAS been asked for, at the last .LRN conference, which is where the only Oracle users with money congegrate. One university has stepped forward to help out with testing and bug-fixing for .LRN 2.2 but it is not at all clear that THEY'LL be using Oracle a year from now, either...

Simon,

It has to do with enterprise users who contribute resources. The people who are currently contributing to develop, test, and release OpenACS are not using Oracle.

If there were resources to keep Oracle support up to date, there would not be an issue of dropping it, but rarely is there a large volunteer or commercially supported effort to do so.

Don Baccus has pretty much been doing all the Oracle testing on his personal server, and I'll let him explain more about how that affects the future of Oracle support if he wants to.

So, if Oracle support for OpenACS is important for your business, please let us know by showing your support. Right now we haven't seen many of the users who "require" Oracle support for OpenACS stepping up to actually maintain that support.

Someone suggested before that Oracle support could become a seperate branch of OpenACS, to be maintained and released on a seperate schedule from PostgreSQL. If someone volunteered to manage that, it seems like a possibility. That would allow the PostgreSQL support and release of OpenACS to move at the pace appropriate for the level of volunteer effort, while Oracle support could move at it's own pace, supported by the users of OpenACS that still require Oracle support.

So Simon, 5 hobbyists (which really isn't a good term, almost all the contributors are working either academically or commerically supporting OpenACS on PostgreSQL), that actually show up and contribute are worth many "enterprise" Oracle users who do not contribute. It would seem that the enterpise users would have even more resources to contribute than a few small companies and a handful of volunteers.

My personal opinion, speaking only for myself, is that focusing on Postgres will result in a net increase of all the good things about OpenACS.

The Oracle port is a distraction, and Postgres has improved enough over the years that Oracle is no longer needed for "high end" users.

More importantly, there are not enough Oracle users in the community that are putting back stuff to keep it current.

I have been observing OACS from a far lately. I think dropping Oracle is a step back, however the benefits outweigh it. It will be easier to support a single db, given that OACS data model is not trivial. Its one of the few web frameworks that actually use the db features. Also on a technical standpoint, there is nothing in Oracle that can't be done in postgresql in the web development problem space.

Dropping Oracle support will hopefully be done in a good manner to bring in the community going forward. Hopefully we avoid removing any community member and move them together with OACS 6 and beyond. I guess its also a community issue as well as a technical issue, so I hope the community aspect will get equal attention if not greater in dropping the support of Oracle.

I think a branch or tag to which OACS db changes will not synch with Oracle and PostgreSQL is needed. A wiki page explaining the rational and more importantly how to upgrade from Oracle to PostgreSQL, people/organizations who have done the port, etc.. Maybe a section or cvs module that has the tools contributed by different people used in porting from Oracle to PostgreSQL.

Anyway just an opinion from a community member observing from a far.

Hi,

I'm one of the quieter OACS users (My colleagues Ciaran, Michael and Brian are on here a bit). We're exclusivly Oracle users.

I would be interested to know how many other Oracle users there are. Perhaps some kind of poll or roll call is needed. If there are enough, and we can split the work between those that want it, it would be possible to keep the oracle support.

Also, I think the main interest is probably in the core packages. Possibly as part of a survey we could find out who uses what package, and what platforms they are wanted on?

Perhaps a specific thread for this is required?
Merv.

Ok Rocael, let me try and put this another way. Your comment has finally provoked me and after a day working with bloody moron hackers I'm in fighting mood :)

My area is XP. My opinion is that if you want to change some code you first provide a test to prove you aren't breaking anything, then you supply a test for the new functionality and then you add your code change.

You run you test and you check it passes. If it does your code stands, if it doesn't you either fix it or back it out.

What you *don't* do is say I've made a change, it works under these circumstances (PG) but doesn't under others (Oracle)so screw it.. I don't want to fix Oracle.

It is *not* the responsibility of the Oracle users to ensure that PG changes are worked into the oracle core. The OpenACS has always been dual database (does this come as a surprise??) and that should have meant any change from any quarter (to the core) satisfies both databases. Anything else is simply unworkable and frankly pretty stupid. I realise its gone on but that does not make it right.

Would you be happy if I made a change in Oracle but didn't reflect that in PS? I suspect not... shall I make some arbitrary core change now in Oracle and then say 'hey, dudes, I've shagged the PG core, so you better fix it?'

All this bullshit about Oracle people maintain Oracle, PG people maintain PG stinks and it always has.

Contributors should be OpenACS contributors. Full Stop. (I refer to those components that are considered central to the OpenACS as a product, individual packages are not the issue)

If it so happens that PG users make more submissions (for whatever reasons) they they have a responsibility for maintaining those changes in Oracle.

I should not be responsible for picking up work because some else couldn't be bothered to do it. I haven't changed the core, I haven't introduced any new functionality and I haven't broken anything. Why then do I have to do half of your work for you?

If you are contributing something to the code base it can be for one of only two reasons:

1) You need the change. Therefore it self-interest and you should apply it to both DBs.
2) You want to make the change i.e. you fancied doing.. same argument applies.

If you didn't need the change anyway (as is the case for me) then what are you bleating about.

The only reason I can see for not maintaining Oracle when you submit a PG change is because your too damn lazy (if it were a question of skills there have always been people on here who would help.)

AFAICT this whole issue is *really* about folks saying 'supporting two databases is more work'.. well of course it is! So is writing quality code, so is fixing bugs, so is documenting... One of ACS's prime features is that it can apply to an enterpise database and a cheap-as-chips database making it accessible to two different groups. That means MORE WORK. perhaps what should be happening is a bit more of the JFDI approach.

I have to work with this kind of lazy-assed, low quality, piss-poor attitude to development everyday. It *really* pisses me off.

It's probably because of those nob-ends that I never have time or energy for anything else... like contributing. You guys who all have soooo much time to contribute must have bloody idyllic careers.

If Don (who I respect a great deal) is having to do all the Oracle maintenance then that is a poor state of affairs. He is basically doing the work that should have been done by the so-called contributors who couldn't be arsed in the first place.

This idea that more contribution = more right to dictate the product structure is absolute shit. I don't contribute cos I simply don't need much change. As of about two years ago the base has done what I needed. I simply do not have time (as a small business man) to piss around with ACS for the sake of it. I'm sorry but that is just a fact of life. Over the last few years I have repeatedly tried to 'sell' ACS into various organisations, but with no success (in fairness its not even the core of what I do). Where they have used it its has been as a test tool, or a minor installation and therefore nothing useful by way of contribution has come of it? But, does that mean I haven't contributed? If I'd landed a big project that fed loads of stuff in, you'd change you're fucking tune then.

Can I also point out that just cos I haven't contributed recently doesn't mean I haven't contributed in the past.. I've been knocking around on and off with the ACS codebase since Ars Digita days..

All this drop 'the Oracle stuff' smells a lot to me like that 'lets re-write it all in Java' shit that finished off ArsDigita. Do I have to remind you that this community was the one that picked up the torch when aD went to the dogs?

I have organised boot camps and attempted to organise some kind of testing effort (which failed because our esteemed contributors were frankly not interested in testing). We even submitted a simple PG based testing package to allow contributors to submit tests to make it easier for others to contribute and retain quality... fat lot of use its been put to..

When I was in a previous (tiny) company, OpenMSG, we tried to pitch in. We had no money, few customers and hardly any time, but we tried to do what we could. I was fighting to pay my fucking rent and *still* trying to add some value. Don't lecture me about contributing or commiting. Frankly I can submit shit code if thats all it takes to be though a 'contributor'.

Do I sound annoyed? Well yes I am, and all this over a bit of software I use only occasionally.. wierd ain't it.

Well, I've been polite over the years, so just this once I consider myself entitled.

If you want to drop Oracle from the core fine. Do so. Why not drop everything that you don't fancy doing. Fuck it, so long as you're ok.

So yes, my vote is lets split the code base.

A smaller community, focussed around Oracle will be more agile, more fleet of foot and more able to make the (obviously infrequent) changes it needs to. To be honest, froma company POV a small community may have a lot of value. Lets face it does the name 'OpenACS' really generate vast incomes? does it hell. Its good tool, which could be preserved.

When we need the odd idea from the PG community we can simply take the flaky rubbish it usually produces and port it. Frankly almost every none core pg-package takes more work than re-building it.

And we can drop that shit name.

And drop that god-awful logo.

And focus on improving an enterpise-level solution.

I can volunteer something. I can volunteer to provide a server and responsibility for maintaining a new community around the oracle version. I will happily take a code cut. Drop that PG stuff and provide a nice clean start point. People can then contribute, or not as they wish. I will not mind either way.

We'll run it along XP principles and provide a better framework into which contribution can be made.

I posted up some time ago that I thought it was time OpenACS stopped being noddy website builder and focussed on serious, enterprise stuff.. well nows the chance. who knows I might make some bloody money from it ;)

And let the good ole PG community be consumed by RoR ;)

Ahhhh.. I feel better for that....

I thank you.

The only reason I can see for not maintaining Oracle when you submit a PG change is because your too damn lazy (if it were a question of skills there have always been people on here who would help.)

Many of our people no longer HAVE Oracle available to them. So it's not just a matter of "laziness", unless you think that going through Oracle installation hell should be a barrier everyone who wants to contribute to OpenACS should go through.

For the record, OACS Core 5.3 supports Oracle and the core will support oracle up until any announced drop date is reached.

And there's no reason you can't continue to use OACS 5.3 for your commercial sites. If it's working for you now, why won't it work for you tomorrow?

It's not trivial to maintain both environments and make sure the work you do for one works for the other. I've done it when I was at Sloan, and it did add a good deal more the the development time and it's frustrating....

I'd vote for a process where it was easy to contribute with one, but also that we are organized enough that others can use the other.

We do want OpenACS to be used by a wide audience... and to a large extent each of us only has so many hours in the day and so much energy. It's fun to be part of the community and in an interchange where we help each other out.. and some time people just have to get the bills payed. To some extent, this takes a little more effort from participants.. and that level of extra effort should be reasonable.

I don't believe anyone is suggesting that we'd do anything with OpenACS so that there was a large obstacle to using Oracle. Or that someone else couldn't jump onto the team and volunteer to do that work.

It's just that the current arrangement (meaning Don doing the vast majority when he doesn't need it at all) isn't working.

Oracle support is not optional.

We support Oracle for oacs core 5.2.3. We will support Oracle for oacs core 5.3. When people break Oracle, we've told them to fucking stop it.

WE ARE TALKING ABOUT A METADECISION TO DECIDE A POINT IN TIME WHEN WE DROP ORACLE SUPPORT.

A fucking YEAR from now and you insult me and the rest of us who make this project work with just about everything short of having a whore for mother and a Brit for a father.

Nice way to try to convince the community to continue Oracle support.

And, I repeat: fork and The Oracle effort will implode. You really think all these people posting up, screaming, will suddenly step forward and maintain, test, and release new features?

HA HA HA! British humour rocks.

"'Oracle support is not optional'.. well, at least we seem to agree on something."

When we talk about a metadecision to decide when we will decide that this will no longer be true, just what do you think this means?

People in this thread are reacting as though we propose to drop Oracle support NOW rather than talking about talking about possibly dropping it a YEAR FROM NOW.

Sheesh.

Now ... you really think that forking a project to support Oracle yourselves would be easier than supporting Oracle within the existing structure of the community?

That's a joke. It would be harder, not easier. Since no one is willing to do it today, why would they be willing to do it tomorrow?

Nick's absolutely right in this.

The 50% estimate is for reasonably stable code where little new development is done. Testing takes as much time for Oracle as it does for PG, you can't take shortcuts and assume that something that works under PG will also work under Oracle. Surely none of you are that naive.

Well, its a bit out of character for me to be honest. *blush*

Perhaps a split may not be such a bad thing, as I mentioned I don't think that the fact that something is 'OpenACS based' carries a lot of weight anyway. My hunch is that the reason people use it is because its a useful tool and it would continue to be so regardless of the size of the community. Its a tool I'd like to be able to keep in my kitbag (even if its a specialist tool)

The pg community seems annoyed at having to support oracle, and thats probably natural. If you're using PG you are unlikely to be solving any serious problems anyway so perhaps the gain from Oracle is seemingly small.

Anyway, lets face it, as you say, the current community hardly looks professional which has probably hampered OpenACS in the enterprise environment anyway.

Ok, here's my suggestion. Lets just make it a clean break. A new name, a new site, and a new focus (i.e. not just building wesites).

The existing oracle users will have an uncluttered environment in which to support their product and perhaps it will find a new lease of life.

If not, well, it didn't really cost much anyway ...

And besides I think OraACS has some opportunities around agility, testing, integration, web services and so forth.. without the contraints of PG we can actually START TO TAKE ADVANTAGE OF ORACLE. We'd get things like

XML parsing.
SOAP.
Java integration.
Distributed computing (with 10g)
Proper performance
Real marketing. Oracle has clout, PG has enthusiasts.
etc

all for FREE. We could actually massively increase what we can do by dropping PG support. I think the community has missed the point. All the concessions over the past few years have been the other way around. It PG that has compromised ACS not the other way around...

You can now develop with Oracle for free. Its your customers that pay for it.. not you.

and we could stop this nonsense of trying to move everything from the database into the scripts.

and stop worrying about that horrid .xql business, query dispatcher and so on..

we could *really* take advantage of being able to use Oracle specifics.

And top of my list we could support using Oracle result sets and Object Types.

And I could use a proper database that I can employ people to use, that companies understand and trust and that actually appeals to management...

after all, originally the Greenspun drive was to make use of the database, not hide from it.

The future's bright, the future's Oracle.

And top of my list we could support using Oracle result sets

When did Oracle start supporting result sets? Does 8i support them? If not, listen to people scream when we suggest dropping 8i support!

BTW PG has supported functions returning result sets for so long I don't even remember which version introduced it...

What I found interesting is that the Users of Oracle are not necessarily the ones contributing to the evolution of the code. So if a split is in order for them to

a) Feel save they wont be cluttered with new features they don't want
b) Make use of special Oraclisms and
c) act more professional

who am I to argue. I think we should formalize this though.

I do have some doubts about the ability of an Oracle stand alone product to appear more professional, on the other hand Project/Open has clearly shown the way, so I have some faith in them pulling it off (knowing that P/O will not drop Oracle support lightly, if at all).

put it another way

development without responsibilities is hacking.

Acually I want to hear some reasoning from these super-human, altruistic contributors who simply code to serve the community without consideration of self-interest.

Answer me these questions.

If you are contributing out of the goodness of your heart then why does that goodness not extend to oracle? Why do you care which database it is, you are surely doing it for higher reasons? The good of the community. The WHOLE community.

What is it so difficult to support oracle? What are you doing, writing the whole PG version and the saying 'fuck, this will be hard to convert to oracle?'. Surely you should simply be doing both at the same time. Here's my little PG query, here's my oracle equivalent. You do have unit tests to prove they both work? Is this really massively time consuming or is it just that you're going about it the wrong way?

Have you thought about trying to automate the conversion to some degree? Why was that so difficult? Could no-one here have advised you?

Is there something fundamentally harder about oracle than PG? Surely the oracle folks here, who may not have time to contribute code, are not adversed to offering advice or guidance? I don't exactly see many 'how do you do this in oracle' questions? This implies to me you aren't even trying in the first place. Is this true?

I'm far from a database expert, but I can do a bit of oracle without too much trouble. Have you considered that perhaps the PG code you came up with in the first place is just piss poor and therefore you then find it hard to re-express in another database? Is this Oracles fault or are you making it harder than it needs to be.

If you can write it in PG, but are stuck when it comes to Oracle what does that tell you about your own ability? Should you really be commiting code in the first place?

I'd like to hear the answers.

I don't know about Simon, but I am not claiming that an Oracle stand alone project would appear (or be) more professional; my only interest in that is to have the ability to continue serving current and future clients who require Oracle. What I do think is that it would make the remaining OpenACS project look more professional.

How many potential large users (since an installation using Oracle is more likely to be large than not) have silently left after doing a test install and finding Postgres syntax in sql or xql files clearly intended for Oracle? When someone is in evaluation mode that sort of thing can send them away screaming, and for ever after they will spread the word that this community is incompetent.

I believe that it has to go one way or the other - either Oracle is supported fully by everyone or it goes. Since the community has made it abundantly clear that not everyone is willing to support Oracle, and no-one in a leadership position is going to force the issue, then we seem to have only one option left.

This might just be highlighting a failure of our API.

You could be using a simple db API which would hide the database implementation from you. And then you could overload the db API to handle each database, and have a method of overriding it for Oracleisms or Postgres-isms.

Would be a couple hours work, much less time than that spent collectively on talking about all this.

Converting old code to the new API would be work, however. But again, the win is a larger community of developers, and an even greater ability to move to other databases.

And better code. You could rip out a one of the parts of OpenACS that are frustrating to code for, like the .xql files.

Not volunteering to do this. But I believe Tom Jackson even has some code (to do this at least part of this) that he's touted to the community many times, to little avail (probably because nobody realized how valuable this can be).

The nice thing about a good API like this is you can also build on it, with code generation, etc...

Jade

To put it another way -- sort of re-implement Rails in Tcl. That would be a good thing, btw. Also supporting "proper" databases like PG and Oracle would set OACS aside once again from a MySQL clout...

Don,

I understand where you're coming from, but (at the risk of repeating myself), I'm not sure I see the 'free' aspect in this.

That makes Oracle sound like an optional add-on. Or that by supporting Oracle a submitter is 'doing everyone a favour'.

My point is simply that the dual database functionality has been part of the OpenACS for years. Everyone who submits is aware of that. Its part of what OpenACS is. Therefore if you submit code that doesn't support Oracle you are only partially delivering OpenACS code. For me, not submitting the Oracle code is *exactly* the same as submitting code that has a bug in it. Expecting the oracle community to fix bugs that one has knowingly introduced is simply not practical. If the numbers of PG users outnumber the Oracle users 10-1, how on earth would they be expected to keep up anyway?

I accept it means more work than supporting one database, but I don't see how you can get away from that. (and support 2 databases)

To my mind it is a simple choice, you either accept that Oracle code is part of delivery, or you accept that it is not part of what OpenACS does. (and the decision appears to have already been made)

I have already outlined in previous postings how this should be addressed. Any submitter should have to provide both database versions. This would prevent the obviously unsustainable situation where one individual (i.e you) is having to do all the work. If they don't want to provide both then the core should not be polluted by the submission. To do otherwise is not in the spirit of community and thus should be managed separately.
(this is why I draw the distinction between packages and the core)

Do you not agree with this approach? Are you all convinced that this cannot work?

Also, if you are the primary maintainer i.e. it is being covered by one person, then surely if that effort were divided amongst all contributers as I have outlined, then it would not be such a burden to any one individual.

But I have no real axe to grind (especially not with you, I have benefited from your efforts over the years and am grateful for that).

I personally think dropping oracle is a big mistake. But if thats the concensus so be it. I've had (quite a big rant) and am happy to accept the decision if nothing I say has had any impact. (gotta try ain't I)

On a final note, can I repeat my disappointment that this community has become contributor-oriented. The idea that simple users have less of a say or are of less value does not sit well with me.

I use Linux, but I don't contribute.
I use Apache, but I don't contribute.
I use TCL, but I don't contribute.

I'm sure thats also true of contributors here. Should those communities not serve their needs? Without users and adopters then OpenACS is simply a personal project.

Thanks for listening, and I suspect its goodbye :(
Simon

Hi Simon,

Yes from an altruistic perspective, the generosity of coding for PG should extend to Oracle, regardless of Oracle contribitors input or other factors that lead developers to judge that Oracle developers do not deserve it.

Iirc, aD supported Oracle and gave away code as part of a business strategy. That strategy no longer exists in the realm of OpenACS. Similar strategies do exist from contributor/vendors that are not finding Oracle part of their market. And the ones that do? Where are they?

Maybe some collective triage process is at work here. Given the available resources, it is not practical to maintain a code base where there is little feedback or interest in sharing available resources. To be altruistic, one has to be responsible and take care of local issues before/while sharing resources that serve others.

A pg to Oracle query converter may be part of a workable solution. I'd like to see one that manages the queries in the xql files instead of generating them dynamically, so there is not yet another layer to process before returning a page.

Perhaps the Oracle community could designate a commercial entity to manage maintenance of the OpenACS core as OpenACS core *and* Oracle evolve, or host a test server with Oracle for testing.

It would then be up to the Oracle community members to find some equitable way to share the burden that fits well in the elite Oracle enterprise environment --call it a joint-venture perhaps, or just write it off as advertising and put your favorite company banners at the top of each page.

Simon, you write: "Expecting the oracle community to fix bugs that one has knowingly introduced is simply not practical." What? Is it that small and resource starved?

Why discourage code improvements of any kind? The manner of style is artful perhaps, but an iso9000 regimented methodology is just one way to continual improvement.

ps

aD's exposing the db to the server is just one technical strategy that OpenACS implements. As Jade and others point out, the tcl api provides a consistent environment for working with multiple databases among other things.

In this and previous discussions about this there is always a sense that the Oracle folks are somehow freeloading, that a lack of contribution from those who use Oracle is what is causing Don to put in all these unpaid hours. But I think that is misplaced. Nothing any of us who use Oracle have done is causing Don to have to do this. The cause is the lazy arsed programmers (Simon, did I get that right? :) who are committing broken code. I fully agree with Simon - as long as the project officially supports two databases than in my opinion a submission that only works for one database contains a bug, just like any other kind of bug. Which Don has been ending up dealing with, and I also fully agree with him that he should not have to do that. But I don't think it's any more fair to say that we who use Oracle should have to pay to have other people's messes cleaned up, that we had no involvement with at all.

If you think that monetary contributions are the solution to the problem, then each person who commits code that does not support both databases should have to pay for someone to complete the work for them. Sure, it would cut down on commits, but it would be fair. More fair than any of the other suggestions so far.

The basic fact is that the rules have not changed; the project supported two databases when most people here today started contributing to it, and it still does. The fact that fewer people use Oracle these days makes it tempting, and to some acceptable, to stop spending time on it. But IMHO that does not make it right.

I support the dropping of Oracle support because I think it's the only solution to the problem that has any chance of success. But I do object to being made out to be a bad guy in this.

Oh, next bug stomp is August 10-11, I believe.

Hope to see some of you Oracle enthusiasts there!

FWIW. I have been using a database abstraction layer for a couple of years now (written in XOTcl). It is not perfect by any means but it does enable you to write database neutral code.

• How do I define the data model of my application?

  
  DB_Class Blog_Item -lmap mk_attribute {
  
      {String body}
      {Timestamptz entry_date}
  
      {Boolean allow_comments_p -isNullable no -default 'f'}
      {Integer cnt_comments -isNullable no -default 0}
      {Timestamptz last_comment -isNullable yes}
  
  } -lmap mk_like {
  
      ::content::Object
      ::content::Title
      ::sharing::Flag_with_Start_Date
  
  } -lmap mk_index {
  
      {Index entry_date}
  
  }
  

• How do I store a new data object?

  
  set bi [Blog_Item new -mixin ::db::Object]
  
  ${bi} set id $id
  ${bi} set title $subject
  ${bi} set body $body
  ${bi} set shared_p $shared_p
  ${bi} set allow_comments_p $allow_comments_p
  
  ${bi} do self-insert
  
  

• How do I retrieve data?

  
  set blogdata [db::Set new -type ::Blog_Item -order "entry_date desc"]
  ${blogdata} load
  
  foreach blogItem [${blogdata} set result] {
      # do something with the data, for example:
      ns_write [$blogItem set title]
  }
  

• What else does it do?
  • Partitioning -- See http://www.oracle.com/technology/products/oracle9i/datasheets/partitioning.html
  • Scoping -- You write your code once but you can apply it to a different scope every time. That means, you no longer need to have the same database table for forum messages in your installation, i.e. each class/community/subsite/user can have their own tables which of course improves performance.
  • TCL-level triggers, onInsertSync, onDeleteSync, onUpdateSync -- For writing aggregators, for example, maintaining forum summaries is a form of aggregation. Aggregators in this abstraction layer are maintained/updated upon insert/update so that you no longer have to do a "select * from forum_messages order by creation_date limit 10"
  • You can join XOTcl defined data classes with database tables which have not been defined as data classes -- For example, you can join against the OpenACS Users table eventhough it has not been defined as an XOTcl class (see above). This is done easily by replacing "-type" with "-from"

There are much more that can be said here (how easy caching would be, etc). Eventhough it has served me well, there is a downside. The downside is that you would still need to invest time bringing it up to the needs of the corporate users in the community as well as writing the Oracle Translation/Interface Driver.

Being an infrastructure package, though, means that any investment that would go into it (either the package itself or related efforts to bring developers/packages up to speed) would only need to be done once.

The reason I have been silent about this is because I'm already pursuing full persistence (i.e. eliminate the need for foreign keys which they now exist in the definition of the data model classes and replace them with conceptual classes and relationships between them -- the persistence mechanism decides how to organize attributes/data into tables and it does it in the same way an expert developer would).

How does this database neutral persistence level deal with the difference between CONNECT BY tree queries and datamodels vs. tree_sortkey datamodels and their associated queries?

The two aren't comparable.

Now 10g's relaxation of restrictions on CONNECT BY queries actually helps a bit (because you can join tables against CONNECT BY subqueries) ... I can actually visualize being able to build similar queries for both schemes via subqueries and joins against them ... but they wouldn't scale well. Scaling tree-building queries using the two models requires different strategies.

We've talked about tcl-based table declarations of the sort Neophytos talks about in his xotcl stuff.

The problem is that we need to clean up the somewhat conflicting attribute systems in the content repository and that for base objects.

Lots of datamodel clean-up.

WITH LOTS OF UPGRADE SCRIPTS.

This grunt-level work would need doing for both databases, and of course would need to be EXTREMELY well-tested in both environments. With realistic data representative of real sites.

The need to provide an upgrade path is a (NECESSARY) millstone around the neck of those that would like to clean-up the datamodel as a first step towards making a much cleaner and easier development environment for programmers.

The need to provide upgrade paths for two databases is a double-weight millstone. Worse, actually, because as is pointed out, I've been the only person committed to keeping Oracle supported.

In short, the higher-level API path (exposing as much as possible through Tcl API as we've been increasingly doing the past couple years making it easy to write db-independent code, or an Xotcl approach such as Neophytos favors) helps the easy problem.

Unfortunately, if that were the only problem, Oracle wouldn't be a big deal.

But fundamental clean-up of the datamodel combined with upgrade scripts is a much harder problem, and eye-candy help such as higher level APIs for the easy part of the problem isn't all that beneficial.

Well, we support trees on top of two databases, that's not the issue. The issue is simply that this particular need is one which drives db query differences.

If you're suggesting that Rails does the nested tree support in the Ruby layer, I'd humbly suggest that this is likely to be terribly inefficient if sizable trees are involved.

I should also point out that the forums package's oracle version uses tree_sortkeys rather than CONNECT BY queries.

And that PG in the semi-near future should support SQL's recursive queries that would allow an approach similar to CONNECT BY.

Hi Andre,

1. Yes Oracle maybe dropped in the future if no one will step up in testing it.

2. Currently its only Don who does the testing, its difficult and not a sustainable model as Don is not really doing Oracle development

3. Its not a licensing issue, its a manpower issue.

I hope a member for the OCT can confirm my answers.

The database abstraction layer is ready (full-blown conceptual persistence is not, *yet*). I can commit it as soon as I get the ok. As I say I above it is by no means perfect. It was developed with scalability in mind (i.e. how would you build a site like yahoo.com) so it lacks some features that corporate users might need (or the developers doing contract work). It is, however, a starting point.

I don't want it in 5.3.

1. As Dave and I have discussed 5.3 (though we haven't run it by everyone on the OCT yet), the goal for 5.3 is to have an acs core that passes all automated tests. Getting to this state has been the focus of the last two bug bashes and, since there are still a bunch of failing tests (almost all in the "fails to meet coding standards", not "code doesn't work" category), will be the focus of the August bug bashes, too.

This is a very modest and very focused goal.

As a result, we've discussed branching the code in September, with a release date in October. By our standards, that's light-speed, no time for fancy schmancy stuff.

I'm committed to continuing the release manager role on (presumably, or rather necessarily given the realities of the community) a volunteer basis to get this out in that kind of time frame.

2. It is possible that Neophytos' work is the be-all and end-all of persistence layers that can be developed in Xotcl.

But there's no way I, at least, am open to deciding to settle on a new API that I haven't even seen yet!

Also, to be useful in the ACS model, it needs to deal with object attributes when defining datastructures. We should really be pushing for something along those lines. We have TCL-based code available that does it, adding such a feature to an Xotcl layer would be easy enough.

Though as I said in another post above, taking full advantage requires our finally biting the bullet to clean up the datamodels. Also we really need to be adding attributes to existing objects where folks don't do them if we really want to leverage stuff like being able to build forms and pages directly from the object model (code which exists in the CMS and elsewhere but which has been little used).

And, if we were to adopt a new API such as a persistence level, deploying it without rewriting acs core would just add another level of complexity to the code that newcomers would have to conquer in order to participate. The barrier's already too high with too many competing APIs and programming approaches. Adopting a new API means, I should hope, taking the time to rewrite core packages to actually use it.

Not something that's going to happen this August :)

No, it's more a Oacs 6 thing in my mind.

I do fully agree with Don, that no one should hold the breath until
we are able to provide a fully functioning higher level api to handle
all the stuff that current acs-core + dozens of packages require.
That's a task for 6.0, and not 5.3.

In the past, i have implemented various active record implementations
in xotcl (see
e.g. https://openacs.org/forums/message-view?message_id=185330), which
is highly flexible but has the problem of accessing the database
mostly by one tuple at a time, leading to bad scalability for some
tasks. The approach used by the for the oo-layer for the content
repository in xotcl-core is much better in this respect, but not a
full-blown generalization. i have seen and discussed the database
abstraction layer and persistence layer from Neophytos, which is in
many respects much more radical, but a very promising approach. But it
needs time. We have hired Neophytos for our EU-funded iCamp project,
so Neophytos will be able to spend some resources for completing and
polishing it.

The big question for me is how radical changes to OpenACS should/could
be in future versions. We have one of the largest installations of
dotlrn and have therefore as well interest in a stable core and dotlrn
packages, but at the same time i have often desires to change
fundamental pieces of OpenACS. We are in competition with other
environments catching up. To keep the lead, we must be able to make
significant changes. Just to use xotcl and keeping things as they are
does not help much in this respect. For example, as discussed earlier in
the forums, i have re-implemented ad_conn in xotcl, which is faster by
a factor of 10 compared to the current implementation. However, this
plug-compatible implementation tries to deal with the idiosyncrasies
existing in ad_conn. This is by no means a nice and well-designed
piece of oo software. I have no big interest of giving the software
out of my hands since i am rather afraid that someone would see the
code and ask me what strange kind of software i am writing. Why?
ad_conn is for example "always around" as a command, has its own logic
of being configured and initialized etc. Conceptually ns_conn and
ad_conn are objects keeping connection specific information. These
objects should be created when a connection thread becomes active and they
should be deleted when the connection thread has finished processing
an http query. There should be a connection class with the ability to subclass it. In
trying to be 100% compatible with the old version, it uses the strange
ways of being configured which are not needed in an oo language. I
made very similar experiences with templating. Although OpenACS is
object based in many respects, turning it into an true oo architecture
based e.g. on xotcl will change many things. If one would try to do
such a transformation piecewise on the existing acs-core, this would
require many TIPs, some of these would be rejected by developers with
many customers running legacy code.

From my point of view, the only approach for substantial redesign
within OpenACS is sideways development. The new code must be able to
co-exist with existing code, which is certainly possible with the
AOLserver. Another approach would be to start all over with a new
project - but this will take even longer until it will be ready for
primetime.

My goals are not to improve the speed of certain functions by an order
of magnitude, but to create the best and most cool web framework to
attract people joining our forces. To achieve this we must be more
attractive and better than RoR (which is not so hard). XOTcl is so
much more flexible than Ruby, so we have our chances. If we don't
succeed to create a vision, the fancy past of Alex and co will fade
over time.

In some respects, xotcl-core and xowiki is for me a project to test
our some oo ideas within OpenACS. Although i am not a friend of
announcements about things to be available in the future, i have to
say that we are pretty far with xosoap and xorb, which is a client and
server web-service environment for openacs based on xotcl. It will be
released in the not-to distant future. Neophytos uses my COMET
streaming technology (part of xowiki chat, see
https://openacs.org/xowiki/pages/en/chat) on phigita.net and he seems
very happy with it. We did already some work on templating, and with
Neophytos database and persistent layer, we are on a good track.

Concerning the Oracle thread: The main issue is maintenance. In the
best of all possible worlds, we are able to generate in the future
most of the application packages code. If the generated code has
interfaces for handcrafting, and is able to co-exist with a small
fraction of manually maintained code, maintenance should go
significantly down.

Moving to OO doesn't mean its going to be easier and productive. It may do it considering starting from 4.x the db schema is not traditional and OO-like but its not guranteed.

I was chatting with an ex-aD developer regarding this thread. Contemplating on history, it MAYBE all the power and complexity of ACS 4.x that scared new comers. That wasn't much an issue during the 3.x days, in fact its just like doing PHP only much more sane and easier. The 3.x codebase was easy to pickup and the output cycle was shorter (near instant gratification to a developer).

Anyway I don't want to muddle the thread any further since the decision to make ACS OO and make it powerful yet complex was decided years ago by aD.

I just had a discussion with an experienced PHP developer that I converted to OpenACS. He mentioned what is irritating is the fact that you have to deal with three files for one page and that variables are automatically set in each of them, without an explanation.

Example:

- If you call db_* you do not know which variables are returned in the tcl statement. So if you are searching for the variable you have not much of a clue, unless you look into the .xql file(s). So extending db_* calls with a documentation part that allows us to write which variables are expected to be returned could help

- This is especially true if you enter the variable in the .adp file without even calling it in .tcl.

But I agree with Rocael's assessment, we should not try to target the same group of people that PHP is targeting, for this OpenACS is just too complex as a framework.

One note though on XoTCL. Before we move forward in this direction, we definitely need training materials which describe common tasks as done before and how they are done afterwards, so we do not loose the experienced developers that OpenACS has at the moment. And obviously we should stay backwards compatible, so the old style of development still works.