Forum OpenACS Development: OACS 6 and beyond

Collapse
Posted by Jon Griffin on
Now that Oracle is officially being dropped, does it make sense to start using pgtcl in place of plpgsql?
I am not sure it does, but at least we will be lowering the cost of entry for new developers. Only haveing to know tcl and sql seems like a big win.

Also, what is the status of xotcl, I know it will be required, but is anything going to be redone, or just new stuff?

Collapse
2: Re: OACS 6 and beyond (response to 1)
Posted by Dave Bauer on
Jon,

I think we will more likely encapsulate the plpgsql, so that its not required to write new plpgsql code for normal operations on an object. That is, we'll provide an API close to what exists for the content repository.If you need to add addtional capabilities to an object you can add them in the Tcl layer.

Unless you have some serious performance issue that requires all your operations to be done in the database, coding your specific application requirements in tcl will be much easier.

Regarding XoTcl, we really need some sort of guidelines at to what XoTcl code will look like, and what the best practices to improve code organization with XoTcl are. We aren't there yet.

One problem is the most likely candidates to be "improved" with XoTcl, the pseudo-object apis like template::form etc. are quite stable, and so pervasive in the toolkit, there is a risk to trying to improve it. Maybe someone with more experience can explain the advantage to making these type of changes.

Collapse
3: Re: OACS 6 and beyond (response to 1)
Posted by Mario Aguado on

Hi, I don't understand because you say the following thing: Now that Oracle is officially being dropped. Can you help me please?. Thanks in advance.

Collapse
4: Re: Re: OACS 6 and beyond (response to 3)
Posted by Malte Sussdorff on
OpenACS 6, which is due next year, will not support Oracle any more. We will not delete the Oracle code, but the OpenACS core will most likely not work on Oracle out of the box as PostgreSQL changes have not made their way into the Oracle code.
Collapse
Posted by Ciaran De Buitlear on
Are you happy that there is a concensus among Openacs users that Oracle should be dropped? I, for one, think it's a backward step.
Collapse
Posted by Simon at TCB on
I share this concern. I've not been that in touch with the community recently, but this comes as a surprise.

What is the driver for this? Technical? Lack of support?

I agree. It would be a giant step backwards.

Looking through recent posts on this topic can I suggest a compromise?

The *core* should continue to support oracle. The packages (with the exception of perhaps a choice few) don't have to.

My thinking here is that future Oracle users are most likely to want to use the OpenACS as platform for building on i.e. most of the packages are pretty irrelevant (and frankly are rarely good enough for likely enterprise users)

I do understand that perhaps the Oracle community is less active here. But without Oracle ACS is pretty much useless to me. We use it for things like test management on oracle databases... there isn't much call for postgres in the enterprise world.

Perhaps the Oracle community is quiet because the parts they use (the core) are pretty stable. So, you have a good stable, enterprise core and you want to drop it? Is that not throwing out the baby with the bathwater?

I've always been of the opinion that its the core that is key to OpenAcs. The packages are just sugar on top, often more work than their worth.

Perhaps we shouldn't be measuring this in terms of numbers of users. I'm trying to put this politely, but are 5 hobbyists the equivalent of 1 enterprise user? Is sheer numbers alone a fair measure? Frankly if you're using it for something quite trivial its a bit of a hammer to crack a nut anyway...

Are there large numbers of bugs or oracle maintenance issues in the core? Is it *really* that much work to maintain oracle for the aspect?

Oh well, if its going I guess its going... nice while it lasted ;) Shame really, still my favourite bit of tech :)

Collapse
Posted by Malte Sussdorff on
I can only tell for myself that I have a hard time providing changes to core packages due to the fact that I don't have the time to test the changes on Oracle. I think a couple of people feel the same.

To my knowledge though, an official TIP on that matter is still required, so maybe my wording was a little bit strong and premature. But this seems the direction in which we are steering.

Collapse
12: Re: OACS 6 and beyond (response to 4)
Posted by Andrew Piskorski on
[sarcasm mode on]

And let me say that as an OpenACS on Oracle user, I so deeply appreciate the full and open discussion which led to this decision. How appropriate that the very first time I ever heard "OpenACS 6" mentioned at all in this, the public forum of record, occurred when Malte dropped this announcement out of the blue. It's so good to see transparent, accountable Open Source governance at work. Just gives me that warm fuzzy feeling all over.

[sarcasm mode off]

Collapse
Posted by Patrick Giagnocavo on
Andrew, did you just volunteer yourself to be the "Oracle go-to guy" on OpenACS 6?

I mean, you are being paid by your company to develop on OpenACS/Oracle, is that correct?

Surely it is only in your best interest to contribute some time to the community to see this continue.

And for that matter, why not jump in right away to fix or port any/all the Postgres-only packages to work on Oracle?

I have not contributed code, but then again, I am not complaining about what is supported and what is not.

Collapse
Posted by Andrew Piskorski on
Patrick, did you see me volunteer for anything? No you didn't, which you knew perfectly well - you're being disingenuous.

No, as I've stated previously elsewhere, I've hardly done any work using OpenACS in years. Web pages are perhaps 5% of my job at most. (Which is part of why the code I have contributed has been very modest, and it's been a long time since I did even that.)

But more importantly, that's all completely irrelevant.

The point is that the very existence of this thread is an illustration in slap-dash, half-assed, lackadaisical governance - not to mention leadership.

A competent Open Source software leader does not suddenly announce, as if after the fact, that the project is dropping all Oracle support, based on some sort of behind-closed-doors discussion and decision making that no one else in the community has ever heard of, and which is entirely absent from the public record.

Yes, PostgreSQL users have been whining about of having to "support" Oracle for years. Fine, but also largely irrelevant.

What a competent leader does, is come out of his private OCT meeting and say something along the lines of:

As many of you know, over the last several years we've had lots of complaints from PostgreSQL developers about the burden of porting to and testing their changes on Oracle, and we feel that these complaints are both justified and serious. Worse, the number of Oracle users stepping up to the plate to fund support and development just hasn't been enough to compensate for the added effort of supporting two RDBMSs.

Therefore, our draft plan - which will be posted to the TIP forum shortly, for further discussion and eventually a formal vote - is basically this:

If we're to keep Oracle support in the next major version of OpenACS, the following things must each happen within the next N months:

  1. X dollars of new funding specifically targetted to ongoing Oracle maintenance.

  2. Y additional developers demonstrating consistent active interest and support for Oracle.

  3. [Etc., whatever. I think you get the idea.]

If you want solid, robust support for OpenACS on Oracle, now is the time - step forward, and please put your money where your mouth is, because we need it.

Something like that, damn it. Not bullshit along the lines of, "Oh by the way, now that we've suddenly decided to completely drop Oracle support, after merely griping about it for years..."

In other words, if you feel an ultimatum is warranted, then issue a real ultimatum, don't just be lazy! You are the leaders of OpenACS - and for good reason - not me. So please act like leaders! Figure out what is needed for solid support of both Oracle and PostgreSQL, state those needs clearly, set a deadline, and state what the fallback plan and consequences are if those needs are not met.

Then at least no one can complain that you didn't try to keep Oracle support in the toolkit...

Collapse
Posted by Don Baccus on
Yes, PostgreSQL users have been whining about of having to "support" Oracle for years. Fine, but also largely irrelevant.
It stops being irrelevant the day I stop saying "someday I will no longer support Oracle for you" and start saying "as of now I will no longer support Oracle for you". Pissing me off isn't going to do much to encourage me to support your platform for free in the future. Competent FOLLOWERS figure that out. Oh, financial support HAS been asked for, at the last .LRN conference, which is where the only Oracle users with money congegrate. One university has stepped forward to help out with testing and bug-fixing for .LRN 2.2 but it is not at all clear that THEY'LL be using Oracle a year from now, either...
Collapse
Posted by Don Baccus on
[fuck you mode on]
We've been talking about it openly for about two years now.

If you were an active participant in the community you'd know that.

If more Oracle users were active participants in the community and in the support and development of the Oracle code base we wouldn't be having this discussion.

As it is there's not a single Oracle user active enough to get themselves elected on the OCT.
[fuck you mode off]

Collapse
8: Re: OACS 6 and beyond (response to 1)
Posted by Dave Bauer on
Simon,

It has to do with enterprise users who contribute resources. The people who are currently contributing to develop, test, and release OpenACS are not using Oracle.

If there were resources to keep Oracle support up to date, there would not be an issue of dropping it, but rarely is there a large volunteer or commercially supported effort to do so.

Don Baccus has pretty much been doing all the Oracle testing on his personal server, and I'll let him explain more about how that affects the future of Oracle support if he wants to.

So, if Oracle support for OpenACS is important for your business, please let us know by showing your support. Right now we haven't seen many of the users who "require" Oracle support for OpenACS stepping up to actually maintain that support.

Someone suggested before that Oracle support could become a seperate branch of OpenACS, to be maintained and released on a seperate schedule from PostgreSQL. If someone volunteered to manage that, it seems like a possibility. That would allow the PostgreSQL support and release of OpenACS to move at the pace appropriate for the level of volunteer effort, while Oracle support could move at it's own pace, supported by the users of OpenACS that still require Oracle support.

So Simon, 5 hobbyists (which really isn't a good term, almost all the contributors are working either academically or commerically supporting OpenACS on PostgreSQL), that actually show up and contribute are worth many "enterprise" Oracle users who do not contribute. It would seem that the enterpise users would have even more resources to contribute than a few small companies and a handful of volunteers.

Collapse
13: Re: Re: OACS 6 and beyond (response to 8)
Posted by Simon at TCB on
Hi Dave,

You're absolutely right. If its a question of no-one able to support the Oracle side, the I guess I have to (reluctantly) agree. The laws of natural selection quite rightly have to apply.

Once Oracle is not maintained actively in the core however that is effectively consigning it to 'code heaven'. You would be better off removing it entirely.

(by the way, I meant no disrespect by the term hobbyist, but they must make up some portion of the community and naturally oracle support isn't going to be a major factor for them. Not sure pure democracy is the right way to make these decisions).

Its an unfortunate foible of commerce I'm afraid that work expended needs to be justified by recognisable need. OpenACS is not my hobby. I have little enough spare time as it is.

We only really use OpenACS the core and are rarely in a position where we need to extend/change it. Therefore the natural opportunities for contribution are limited. We aren't a large company by any means, but the people we work for are. We simply can't afford to allocate paid-for time to contributing to stuff we don't need (i.e. packages etc).

The only PG thing we use OACS for is as a test tool/harness (you may recall we created the automated tesing package).. so limited opportunities there.

Our clients are Oracle based, Postgres simply isn't a commerical reality for us. And, it looks like, neither will OpenACS be.

I guess we'll just take a cut and do with it what we will.

Collapse
9: Re: OACS 6 and beyond (response to 1)
Posted by Patrick Giagnocavo on
My personal opinion, speaking only for myself, is that focusing on Postgres will result in a net increase of all the good things about OpenACS.

The Oracle port is a distraction, and Postgres has improved enough over the years that Oracle is no longer needed for "high end" users.

More importantly, there are not enough Oracle users in the community that are putting back stuff to keep it current.

Collapse
10: Re: OACS 6 and beyond (response to 1)
Posted by Tracy Adams on
Yes, it does come down to resources.

It's not a "good or bad" decision at all.

The platform is driven forward by either volunteers or organizations interested in using OpenACS for their needs.

And it looks like the core contributors are saying "hey, I don't really have a use for the Oracle side, I'm going to concentrate my time in energy into PostGres which will benefit me directly"...

Now, what can happen next? Should individuals or organizations want and value the Oracle part, they figure out a way to make that to happen because they it's for their own benefit.

Collapse
Posted by Jun Yamog on
I have been observing OACS from a far lately. I think dropping Oracle is a step back, however the benefits outweigh it. It will be easier to support a single db, given that OACS data model is not trivial. Its one of the few web frameworks that actually use the db features. Also on a technical standpoint, there is nothing in Oracle that can't be done in postgresql in the web development problem space.

Dropping Oracle support will hopefully be done in a good manner to bring in the community going forward. Hopefully we avoid removing any community member and move them together with OACS 6 and beyond. I guess its also a community issue as well as a technical issue, so I hope the community aspect will get equal attention if not greater in dropping the support of Oracle.

I think a branch or tag to which OACS db changes will not synch with Oracle and PostgreSQL is needed. A wiki page explaining the rational and more importantly how to upgrade from Oracle to PostgreSQL, people/organizations who have done the port, etc.. Maybe a section or cvs module that has the tools contributed by different people used in porting from Oracle to PostgreSQL.

Anyway just an opinion from a community member observing from a far.

Collapse
15: Re: OACS 6 and beyond (response to 1)
Posted by Malte Sussdorff on
I think sarcasm in either direction does not help us. What I would suggest is that we focus on the following:

How can the OpenACS community accomodate the remaining Oracle users and help them to keep a viable Oracle port.

I think we still can do a couple of things:

- Create new Ticket types:
-- Test on Oracle needed
-- Port for Oracle needed

So whenever a commit to OpenACS core is done, where we changed the DB file and are unsure if this works on Oracle, we can ask for a test (if we provided a separate -oracle.xql file and upgrade file), or alternatively ask for someone to port it.

In this ticket we write what we changed and if possible the link to the CVS commit (as per xarg.net or fisheye).

This would allow the Oracle users of the core to maintain it easily and I concurr with Simon that chances are high that the amount of specific Oracle work will be fairly small.

As for the packages... This only makes sense if someone commits himself to work on said package for Oracle. Then you could ask people to file a report for this package whenever they do a release / commit.

Hope this brings us a little bit further. I don't think we want to "pull the carpet under your feet" (if you have a saying like that in English...), but we have to face the realities and the reality is that Don is on his own with testing and fixing OpenACS core Oracle commits and probably wants to spend his time more efficiently.

Collapse
16: Re: OACS 6 and beyond (response to 1)
Posted by merv colton on
Hi,

I'm one of the quieter OACS users (My colleagues Ciaran, Michael and Brian are on here a bit). We're exclusivly Oracle users.

I would be interested to know how many other Oracle users there are. Perhaps some kind of poll or roll call is needed. If there are enough, and we can split the work between those that want it, it would be possible to keep the oracle support.

Also, I think the main interest is probably in the core packages. Possibly as part of a survey we could find out who uses what package, and what platforms they are wanted on?

Perhaps a specific thread for this is required?
Merv.

Collapse
Posted by Rocael Hernández Rizzardini on
People with interest in oracle, can say it here. Then, organize themselves and say that they can commit to suppport oracle (at least acs-core).
Collapse
Posted by Simon at TCB on
Ok Rocael, let me try and put this another way. Your comment has finally provoked me and after a day working with bloody moron hackers I'm in fighting mood :)

My area is XP. My opinion is that if you want to change some code you first provide a test to prove you aren't breaking anything, then you supply a test for the new functionality and then you add your code change.

You run you test and you check it passes. If it does your code stands, if it doesn't you either fix it or back it out.

What you *don't* do is say I've made a change, it works under these circumstances (PG) but doesn't under others (Oracle)so screw it.. I don't want to fix Oracle.

It is *not* the responsibility of the Oracle users to ensure that PG changes are worked into the oracle core. The OpenACS has always been dual database (does this come as a surprise??) and that should have meant any change from any quarter (to the core) satisfies both databases. Anything else is simply unworkable and frankly pretty stupid. I realise its gone on but that does not make it right.

Would you be happy if I made a change in Oracle but didn't reflect that in PS? I suspect not... shall I make some arbitrary core change now in Oracle and then say 'hey, dudes, I've shagged the PG core, so you better fix it?'

All this bullshit about Oracle people maintain Oracle, PG people maintain PG stinks and it always has.

Contributors should be OpenACS contributors. Full Stop. (I refer to those components that are considered central to the OpenACS as a product, individual packages are not the issue)

If it so happens that PG users make more submissions (for whatever reasons) they they have a responsibility for maintaining those changes in Oracle.

I should not be responsible for picking up work because some else couldn't be bothered to do it. I haven't changed the core, I haven't introduced any new functionality and I haven't broken anything. Why then do I have to do half of your work for you?

If you are contributing something to the code base it can be for one of only two reasons:

1) You need the change. Therefore it self-interest and you should apply it to both DBs.
2) You want to make the change i.e. you fancied doing.. same argument applies.

If you didn't need the change anyway (as is the case for me) then what are you bleating about.

The only reason I can see for not maintaining Oracle when you submit a PG change is because your too damn lazy (if it were a question of skills there have always been people on here who would help.)

AFAICT this whole issue is *really* about folks saying 'supporting two databases is more work'.. well of course it is! So is writing quality code, so is fixing bugs, so is documenting... One of ACS's prime features is that it can apply to an enterpise database and a cheap-as-chips database making it accessible to two different groups. That means MORE WORK. perhaps what should be happening is a bit more of the JFDI approach.

I have to work with this kind of lazy-assed, low quality, piss-poor attitude to development everyday. It *really* pisses me off.

It's probably because of those nob-ends that I never have time or energy for anything else... like contributing. You guys who all have soooo much time to contribute must have bloody idyllic careers.

If Don (who I respect a great deal) is having to do all the Oracle maintenance then that is a poor state of affairs. He is basically doing the work that should have been done by the so-called contributors who couldn't be arsed in the first place.

This idea that more contribution = more right to dictate the product structure is absolute shit. I don't contribute cos I simply don't need much change. As of about two years ago the base has done what I needed. I simply do not have time (as a small business man) to piss around with ACS for the sake of it. I'm sorry but that is just a fact of life. Over the last few years I have repeatedly tried to 'sell' ACS into various organisations, but with no success (in fairness its not even the core of what I do). Where they have used it its has been as a test tool, or a minor installation and therefore nothing useful by way of contribution has come of it? But, does that mean I haven't contributed? If I'd landed a big project that fed loads of stuff in, you'd change you're fucking tune then.

Can I also point out that just cos I haven't contributed recently doesn't mean I haven't contributed in the past.. I've been knocking around on and off with the ACS codebase since Ars Digita days..

All this drop 'the Oracle stuff' smells a lot to me like that 'lets re-write it all in Java' shit that finished off ArsDigita. Do I have to remind you that this community was the one that picked up the torch when aD went to the dogs?

I have organised boot camps and attempted to organise some kind of testing effort (which failed because our esteemed contributors were frankly not interested in testing). We even submitted a simple PG based testing package to allow contributors to submit tests to make it easier for others to contribute and retain quality... fat lot of use its been put to..

When I was in a previous (tiny) company, OpenMSG, we tried to pitch in. We had no money, few customers and hardly any time, but we tried to do what we could. I was fighting to pay my fucking rent and *still* trying to add some value. Don't lecture me about contributing or commiting. Frankly I can submit shit code if thats all it takes to be though a 'contributor'.

Do I sound annoyed? Well yes I am, and all this over a bit of software I use only occasionally.. wierd ain't it.

Well, I've been polite over the years, so just this once I consider myself entitled.

If you want to drop Oracle from the core fine. Do so. Why not drop everything that you don't fancy doing. Fuck it, so long as you're ok.

So yes, my vote is lets split the code base.

A smaller community, focussed around Oracle will be more agile, more fleet of foot and more able to make the (obviously infrequent) changes it needs to. To be honest, froma company POV a small community may have a lot of value. Lets face it does the name 'OpenACS' really generate vast incomes? does it hell. Its good tool, which could be preserved.

When we need the odd idea from the PG community we can simply take the flaky rubbish it usually produces and port it. Frankly almost every none core pg-package takes more work than re-building it.

And we can drop that shit name.

And drop that god-awful logo.

And focus on improving an enterpise-level solution.

I can volunteer something. I can volunteer to provide a server and responsibility for maintaining a new community around the oracle version. I will happily take a code cut. Drop that PG stuff and provide a nice clean start point. People can then contribute, or not as they wish. I will not mind either way.

We'll run it along XP principles and provide a better framework into which contribution can be made.

I posted up some time ago that I thought it was time OpenACS stopped being noddy website builder and focussed on serious, enterprise stuff.. well nows the chance. who knows I might make some bloody money from it ;)

And let the good ole PG community be consumed by RoR ;)

Ahhhh.. I feel better for that....

I thank you.

Collapse
Posted by Simon at TCB on
Ahhh. the days of ArsDigita...

A code base so full of holes you could call it a golf course where when you asked a question or tried to submit a fix they just carried on and ignored you anyway...

You've never had it so good.

Collapse
Posted by Don Baccus on
The only reason I can see for not maintaining Oracle when you submit a PG change is because your too damn lazy (if it were a question of skills there have always been people on here who would help.)
Many of our people no longer HAVE Oracle available to them. So it's not just a matter of "laziness", unless you think that going through Oracle installation hell should be a barrier everyone who wants to contribute to OpenACS should go through.

For the record, OACS Core 5.3 supports Oracle and the core will support oracle up until any announced drop date is reached.

And there's no reason you can't continue to use OACS 5.3 for your commercial sites. If it's working for you now, why won't it work for you tomorrow?

Collapse
Posted by Simon at TCB on
You seem to have posted several in quick succession and I can't keep up ;)

I think my comment appears a little out of context. It is made in light of my opinion that doing the oracle bit was never supposed to be 'optional'.

But, I do understand the practical point you raise.

However, the 'pain' you describe is about a days work. Thats on fedora on an IBM laptop, and I am in no way an oracle expert. How often does this have to be done? Once a year maybe? You only need to get the thing starting to be able to prove some basic SQL code. Its not as if you have to spend a lot of time in maintenance or tuning.

Is that really such a barrier to entry?

It can't be financial either because Oracle is free for development purposes.

If that amount of work is really a barrier to willing contributors then I think you are right. It must surely have to be dropped.

And yes I agree. What we have will continue to work.. it'll just be a case of setting up a smaller, separate community to maintain it I guess.

Collapse
Posted by Tracy Adams on
It's not trivial to maintain both environments and make sure the work you do for one works for the other. I've done it when I was at Sloan, and it did add a good deal more the the development time and it's frustrating....

I'd vote for a process where it was easy to contribute with one, but also that we are organized enough that others can use the other.

We do want OpenACS to be used by a wide audience... and to a large extent each of us only has so many hours in the day and so much energy. It's fun to be part of the community and in an interchange where we help each other out.. and some time people just have to get the bills payed. To some extent, this takes a little more effort from participants.. and that level of extra effort should be reasonable.

I don't believe anyone is suggesting that we'd do anything with OpenACS so that there was a large obstacle to using Oracle. Or that someone else couldn't jump onto the team and volunteer to do that work.

It's just that the current arrangement (meaning Don doing the vast majority when he doesn't need it at all) isn't working.

Collapse
Posted by Simon at TCB on
Hi Tracy,

'..and to a large extent each of us only has so many hours in the day and so much energy'

And herein lies my dilemma also. And always has. We're a small consultancy, not a software house. If I had time to spare there are many other things I have to put higher up the list. Whilst we make use of OpenACS from time to time (and would like to do so more) it isn't the core of what,as XP consultants, we do. :(

So, yes, I accept that I can rant all I like, but, I can't contribute in the way I'd like, so I can't expect anything...

I just want to see this fully debated (and vigorously) before the decision can't be undone.

Collapse
Posted by Don Baccus on
Oracle support is not optional.

We support Oracle for oacs core 5.2.3. We will support Oracle for oacs core 5.3. When people break Oracle, we've told them to fucking stop it.

WE ARE TALKING ABOUT A METADECISION TO DECIDE A POINT IN TIME WHEN WE DROP ORACLE SUPPORT.

A fucking YEAR from now and you insult me and the rest of us who make this project work with just about everything short of having a whore for mother and a Brit for a father.

Nice way to try to convince the community to continue Oracle support.

And, I repeat: fork and The Oracle effort will implode. You really think all these people posting up, screaming, will suddenly step forward and maintain, test, and release new features?

HA HA HA! British humour rocks.

Collapse
Posted by Simon at TCB on
'Oracle support is not optional'.. well, at least we seem to agree on something.

Why you wish to take this so personally I don't know. You are not the one ingoring oracle, and I'm sure thats true of others. But *somebody* must be for there to be an issue and for you to be doing the majority of the work.

My frustration was directed at submitters who have not supported oracle sufficiently and who seem to be suggesting that is was someone elses responsibility all along.

Oh and just in case no one picked up on it I meant the core, just the core. I have in no way suggested at ANY point that this applies to application packages. Just the CORE. I did not think this was unreasonable.

But, if you estimate 50% effort for that (and I confess I did not realise the figure was so high) then I can accept that it is simply not practical.

I accept, you are right and I am wrong. I should have stayed out of it, and therefore I hope everyone will have the good sense to dis-regard my inappropriate comments.

Thank you all for you efforts in the past, and my best wishes for the future.

Simon

Collapse
Posted by Don Baccus on
"'Oracle support is not optional'.. well, at least we seem to agree on something."

When we talk about a metadecision to decide when we will decide that this will no longer be true, just what do you think this means?

People in this thread are reacting as though we propose to drop Oracle support NOW rather than talking about talking about possibly dropping it a YEAR FROM NOW.

Sheesh.

Now ... you really think that forking a project to support Oracle yourselves would be easier than supporting Oracle within the existing structure of the community?

That's a joke. It would be harder, not easier. Since no one is willing to do it today, why would they be willing to do it tomorrow?

Nick's absolutely right in this.

The 50% estimate is for reasonably stable code where little new development is done. Testing takes as much time for Oracle as it does for PG, you can't take shortcuts and assume that something that works under PG will also work under Oracle. Surely none of you are that naive.

Collapse
20: Re: OACS 6 and beyond (response to 1)
Posted by Janine Ohmer on
Simon, tell us how you *really* feel! :) Thanks for that, actually, I found myself nodding and laughing as I read it. Not many of us would put our feelings so bluntly, but I can sure understand where you are coming from.

FWIW I completely agree with you that it's unacceptable for programmers to be breaking the database they don't care about. And I would include the entire codebase, not just the core. IMHO as long as the project officially supports both databases it damn well ought to support them both. This "oh, if you want to use that package you have to clean up someone else's mess first" stuff is just not right.

However, having said that, I also recognize that dual-database support is not happening and is not going to happen. The only way it will ever happen is if we have an iron-fisted gatekeeper who can require it, and we're never going to have that. So, I would prefer to see Oracle support dropped entirely than to have the "we sorta do and we sorta don't" situation we have today. I think it makes the project seem sloppy and unprofessional.

furfly has one major client, the Army Corps of Engineers, who requires Oracle, so this is going to be painful for us and we might be interested in participating in some sort of split. It would probably make sense to use a codebase that's maintained by and for people using Oracle and choose to port over changes that we really want, rather than have to fix the entire thing each time we want to set up a new site or upgrade an old one.

Collapse
Posted by Simon at TCB on
Well, its a bit out of character for me to be honest. *blush*

Perhaps a split may not be such a bad thing, as I mentioned I don't think that the fact that something is 'OpenACS based' carries a lot of weight anyway. My hunch is that the reason people use it is because its a useful tool and it would continue to be so regardless of the size of the community. Its a tool I'd like to be able to keep in my kitbag (even if its a specialist tool)

The pg community seems annoyed at having to support oracle, and thats probably natural. If you're using PG you are unlikely to be solving any serious problems anyway so perhaps the gain from Oracle is seemingly small.

Anyway, lets face it, as you say, the current community hardly looks professional which has probably hampered OpenACS in the enterprise environment anyway.

Ok, here's my suggestion. Lets just make it a clean break. A new name, a new site, and a new focus (i.e. not just building wesites).

The existing oracle users will have an uncluttered environment in which to support their product and perhaps it will find a new lease of life.

If not, well, it didn't really cost much anyway ...

And besides I think OraACS has some opportunities around agility, testing, integration, web services and so forth.. without the contraints of PG we can actually START TO TAKE ADVANTAGE OF ORACLE. We'd get things like

XML parsing.
SOAP.
Java integration.
Distributed computing (with 10g)
Proper performance
Real marketing. Oracle has clout, PG has enthusiasts.
etc

all for FREE. We could actually massively increase what we can do by dropping PG support. I think the community has missed the point. All the concessions over the past few years have been the other way around. It PG that has compromised ACS not the other way around...

You can now develop with Oracle for free. Its your customers that pay for it.. not you.

and we could stop this nonsense of trying to move everything from the database into the scripts.

and stop worrying about that horrid .xql business, query dispatcher and so on..

we could *really* take advantage of being able to use Oracle specifics.

And top of my list we could support using Oracle result sets and Object Types.

And I could use a proper database that I can employ people to use, that companies understand and trust and that actually appeals to management...

after all, originally the Greenspun drive was to make use of the database, not hide from it.

The future's bright, the future's Oracle.

Collapse
Posted by Don Baccus on
Go for it, Simon. The Oracle OpenACS community would implode within months. How do I know that? If it hadn't already imploded we wouldn't be having this discussion. Think about it.
I think the community has missed the point. All the concessions over the past few years have been the other way around. It PG that has compromised ACS not the other way around...
Umm ... if it weren't for PG the OpenACS project would've never started and you and I would've never met. Nice to see what you think of us "hobbyists". Yeah, I have an idyllic career. That's why I'm so broke I'm deeply in debt and can barely meet expenses. It's because I've been dumb enough to spend significant chunks of my life working on code for free so that you can: 1. make money off it 2. insult me while doing it Nice, Simon. Nice.
Collapse
Posted by Don Baccus on
And top of my list we could support using Oracle result sets
When did Oracle start supporting result sets? Does 8i support them? If not, listen to people scream when we suggest dropping 8i support!

BTW PG has supported functions returning result sets for so long I don't even remember which version introduced it...

Collapse
23: Re: OACS 6 and beyond (response to 1)
Posted by Patrick Giagnocavo on
I apologize if my earlier comments appeared to be hostile or flaming.

Andrew, my apologies if I gave offense - but if you are not committing code, why should anyone else? You must have a wealth of Oracle knowledge that OpenACS could benefit from - but according to the stats I see, you haven't committed anything in the last 800 days.

I am sure everyone would welcome, any sort of documentation / unit tests / new code / whatever, for Oracle or Postgres or even some third database.

My belief is, that in the time frame between now and the first release of OpenACS 6, whenever that may be ... very little Oracle code will be committed in proportion to the amount of bboard postings decrying Oracle being dropped.

Collapse
24: Re: OACS 6 and beyond (response to 1)
Posted by Malte Sussdorff on
What I found interesting is that the Users of Oracle are not necessarily the ones contributing to the evolution of the code. So if a split is in order for them to

a) Feel save they wont be cluttered with new features they don't want
b) Make use of special Oraclisms and
c) act more professional

who am I to argue. I think we should formalize this though.

I do have some doubts about the ability of an Oracle stand alone product to appear more professional, on the other hand Project/Open has clearly shown the way, so I have some faith in them pulling it off (knowing that P/O will not drop Oracle support lightly, if at all).

Collapse
Posted by Simon at TCB on
Well, its not as if we're being given much of a choice is it. With Oracle being essentially 'deprecated' there isn't any future for it here.

I think I addressed contribution. If Oracle users are not introducing changes or bugs then why do they have to develop the oracle equivalent for those who do so for PG?

I think its the same situation we had a few years ago around testing. Developers who make changes should also submit sufficient tests. Why is it someone elses resposibility simply because the developer 'doesn't want to do testing'?

Same for documentation. Doc quality is low because people intoduce new features, but expect someone else to communicate it for them.

Even if there were more oracle contributors doing 'other peoples jobs' then unless you have a pretty much even number of both sides then one will always be playing catch up. the rate of submission will always be controlled by the larger community. So effectively might (or size) makes right.

The problem is not Oracle, its the attitude towards what constitutes acceptable development standards.

The rules shoudl have always been
- no code without test.
- no query without both databases.
- no functions without user documents/guides.

This community could learn a lot from XP.

Collapse
Posted by Simon at TCB on
put it another way

development without responsibilities is hacking.

Collapse
Posted by Simon at TCB on
how about something more positive? (sorry I keep hitting POST before I've finished thinking)

Perhaps there should have been a stream of activity in producing a 'converstion' utility to map PG to Oracle (and vice versa).

Ok it will never be perfect and will probably not produce 'optimal' code, but in good XP fashion 'leave optimisation till last.'

Optimisation would properly belong in the domain of the oracle experts who would optimise if they required it.

There's nothing all that clever about most of the DB code in ACS so this should not be too hard. If you adopted the XP style of code developement the smaller, simpler functions would lend themselves to conversion.

Or even simpler. If the PG people were to produce reasonably exhaustive unit/acceptance tests then at least there'd be something for the oracle folks to aim at in terms of porting (rather than just having to figure out what the hell was intended)

I'm sorry but this dropping oracle stuff just doesn't stand up. There are many other avenues that could and should have been explored before such draconian action.

One of the things that I think any open source governance has to do is to accept that they are there to protect the essence of the project, not to dictate it.

OACS is dual-database. Thats one of its foundation features. Once its gone its lost for good.

Next it'll be, 'this TCL stuff, but we're mainly PERL/Java/Ruby devleopers.. we don't want to support it'

You are not evolving the product, you are fundamentally altering in a way the constitutes something new. That is not the function of an OS community in my book.

Actually, perhaps another question ought to be raised?

Why are there are significantly more PG users?

With the greatest respect to PG it simply ain't Oracle. its not even in the same solar system. Ok its free, but its no Linux is it. And frankly there is a reason oracle costs money.

Oracle is now free to developers (and has been for some time).

I can only surmise that the majority of applications of OpenACS must there be in trivial, low performance requirements where the power of oracle is surplus to requirements.

So then, why are you using OpenACS in the first place? Have you looked at it recently, the size of it, the depth of it. Originally it was designed to be powerful, capable of running highly demanding, database-backed websites. It uses AOLServer for that very reason (ask Dossy).

If you remove oracle you are demoting OpenACS. It becomes a second class application. Can't offer serious performance, but in turn remaining over-complicated for the tasks at hand. You'd be better off on Rails.

Collapse
Posted by Simon at TCB on
Acually I want to hear some reasoning from these super-human, altruistic contributors who simply code to serve the community without consideration of self-interest.

Answer me these questions.

If you are contributing out of the goodness of your heart then why does that goodness not extend to oracle? Why do you care which database it is, you are surely doing it for higher reasons? The good of the community. The WHOLE community.

What is it so difficult to support oracle? What are you doing, writing the whole PG version and the saying 'fuck, this will be hard to convert to oracle?'. Surely you should simply be doing both at the same time. Here's my little PG query, here's my oracle equivalent. You do have unit tests to prove they both work? Is this really massively time consuming or is it just that you're going about it the wrong way?

Have you thought about trying to automate the conversion to some degree? Why was that so difficult? Could no-one here have advised you?

Is there something fundamentally harder about oracle than PG? Surely the oracle folks here, who may not have time to contribute code, are not adversed to offering advice or guidance? I don't exactly see many 'how do you do this in oracle' questions? This implies to me you aren't even trying in the first place. Is this true?

I'm far from a database expert, but I can do a bit of oracle without too much trouble. Have you considered that perhaps the PG code you came up with in the first place is just piss poor and therefore you then find it hard to re-express in another database? Is this Oracles fault or are you making it harder than it needs to be.

If you can write it in PG, but are stuck when it comes to Oracle what does that tell you about your own ability? Should you really be commiting code in the first place?

I'd like to hear the answers.

Collapse
Posted by Don Baccus on
My estimate is that Oracle support adds about 50% overhead to the ongoing maintenance of a package. Less for new development.

We used an automated convertor when first porting ACS to the OpenACS dual-RDBMS support environment. For ongoing support, it's not as big a win as you might think because the tricky things can't be automatically translated. Anything that has to do with the tree structure of objects, in particular.

Now, at the same time we've been talking about dropping Oracle support, we've been moving more and more of the API into Tcl. Use of Tcl API makes support of Oracle easy during development.

BUT, as you well know, adequate testing takes time. Automated tests only take you so far.

There's also the administrative overhead of managing releases and the like.

Now, the current .LRN 2.2 release has gone well for both Oracle and PG. That's because we have ORACLE CONTRIBUTORS WILLING TO DIRTY THEIR HANDS ALONG WITH US INCOMPETENT "HOBBYISTS".

You guys step to the plate and help out, then Oracle support needn't end.

The dialogue over the past two years has taken the following form:

IF Oracle users don't step forward and contribute to Oracle support THEN we will probably drop it.

No one steps forward.

Repeat: IF Oracle users ...

No one steps forward.

OK, we'll drop Oracle support in, oh, a year or so, since no one cares enough to help support the platform.

SCREAMS OF PAIN AND ANGUISH! Oh Lord, we've stabbed you in the back!

Collapse
Posted by Janine Ohmer on
I don't know about Simon, but I am not claiming that an Oracle stand alone project would appear (or be) more professional; my only interest in that is to have the ability to continue serving current and future clients who require Oracle. What I do think is that it would make the remaining OpenACS project look more professional.

How many potential large users (since an installation using Oracle is more likely to be large than not) have silently left after doing a test install and finding Postgres syntax in sql or xql files clearly intended for Oracle? When someone is in evaluation mode that sort of thing can send them away screaming, and for ever after they will spread the word that this community is incompetent.

I believe that it has to go one way or the other - either Oracle is supported fully by everyone or it goes. Since the community has made it abundantly clear that not everyone is willing to support Oracle, and no-one in a leadership position is going to force the issue, then we seem to have only one option left.

Collapse
Posted by Janine Ohmer on
After re-reading that I realized I worded it badly. It's not that I think having an Oracle stand-alone version would make OpenACS look more professional, it's that I think dropping Oracle support would make OpenACS look more professional. Supporting it fully would have the same effect, and I would argue it's the better choice, but it seems clear that is not going to happen so I'm not wasting much effort trying to evangelize for it.
Collapse
31: Re: OACS 6 and beyond (response to 1)
Posted by Jade Rubick on
This might just be highlighting a failure of our API.

You could be using a simple db API which would hide the database implementation from you. And then you could overload the db API to handle each database, and have a method of overriding it for Oracleisms or Postgres-isms.

Would be a couple hours work, much less time than that spent collectively on talking about all this.

Converting old code to the new API would be work, however. But again, the win is a larger community of developers, and an even greater ability to move to other databases.

And better code. You could rip out a one of the parts of OpenACS that are frustrating to code for, like the .xql files.

Not volunteering to do this. But I believe Tom Jackson even has some code (to do this at least part of this) that he's touted to the community many times, to little avail (probably because nobody realized how valuable this can be).

The nice thing about a good API like this is you can also build on it, with code generation, etc...

Jade

Collapse
32: Re: OACS 6 and beyond (response to 1)
Posted by Patrick Giagnocavo on
I think Jade has a good point.

Possibly using awk, m4, or even TCL as a macro language, perhaps code for both Oracle and Postgres could be developed at the same time from the same set of higher-level macro definitions. Adding support for a third database (DB2? SapDB a/k/a MySql MAX-5?) would then be a case of porting the table definitions and the macro language.

This would reduce simple errors like mis-spellings and possibly lead to a series of tests that could be auto-generated as well.

Collapse
Posted by Andrei Popov on
To put it another way -- sort of re-implement Rails in Tcl. That would be a good thing, btw. Also supporting "proper" databases like PG and Oracle would set OACS aside once again from a MySQL clout...
Collapse
33: Re: OACS 6 and beyond (response to 1)
Posted by Don Baccus on
Well, since Dave asked this question:

"Don Baccus has pretty much been doing all the Oracle testing on his personal server, and I'll let him explain more about how that affects the future of Oracle support if he wants to."

Last spring in Madrid, when asked to give a talk, I essentially said "if the Oracle folks don't provide support for Oracle, Oracle will at some point be dropped".

I also said "I'm not going to be the primary maintainer for the Oracle version of OpenACS core for free forever".

Everybody screams as to how much pain it will cause them.

Nobody offers funding ...

Oh, Simon, it's not the "hobbyists" who want to drop Oracle support. It's those of us making our living doing OpenACS work. We're the ones either supporting it for free (me) or ignoring it altogether (Malte).

Now if we DO continue to support Oracle, I would agree that core and important packages must continue to support Oracle. Malte made a more-or-less unilateral decision to stop doing so. How do we stop him from doing so? How do you force him to support Oracle FOR FREE?

Collapse
Posted by Simon at TCB on
Don,

I understand where you're coming from, but (at the risk of repeating myself), I'm not sure I see the 'free' aspect in this.

That makes Oracle sound like an optional add-on. Or that by supporting Oracle a submitter is 'doing everyone a favour'.

My point is simply that the dual database functionality has been part of the OpenACS for years. Everyone who submits is aware of that. Its part of what OpenACS is. Therefore if you submit code that doesn't support Oracle you are only partially delivering OpenACS code. For me, not submitting the Oracle code is *exactly* the same as submitting code that has a bug in it. Expecting the oracle community to fix bugs that one has knowingly introduced is simply not practical. If the numbers of PG users outnumber the Oracle users 10-1, how on earth would they be expected to keep up anyway?

I accept it means more work than supporting one database, but I don't see how you can get away from that. (and support 2 databases)

To my mind it is a simple choice, you either accept that Oracle code is part of delivery, or you accept that it is not part of what OpenACS does. (and the decision appears to have already been made)

I have already outlined in previous postings how this should be addressed. Any submitter should have to provide both database versions. This would prevent the obviously unsustainable situation where one individual (i.e you) is having to do all the work. If they don't want to provide both then the core should not be polluted by the submission. To do otherwise is not in the spirit of community and thus should be managed separately.
(this is why I draw the distinction between packages and the core)

Do you not agree with this approach? Are you all convinced that this cannot work?

Also, if you are the primary maintainer i.e. it is being covered by one person, then surely if that effort were divided amongst all contributers as I have outlined, then it would not be such a burden to any one individual.

But I have no real axe to grind (especially not with you, I have benefited from your efforts over the years and am grateful for that).

I personally think dropping oracle is a big mistake. But if thats the concensus so be it. I've had (quite a big rant) and am happy to accept the decision if nothing I say has had any impact. (gotta try ain't I)

On a final note, can I repeat my disappointment that this community has become contributor-oriented. The idea that simple users have less of a say or are of less value does not sit well with me.

I use Linux, but I don't contribute.
I use Apache, but I don't contribute.
I use TCL, but I don't contribute.

I'm sure thats also true of contributors here. Should those communities not serve their needs? Without users and adopters then OpenACS is simply a personal project.

Thanks for listening, and I suspect its goodbye :(
Simon

Collapse
40: Re: OACS 6 and beyond (response to 1)
Posted by Tracy Adams on
I think it's a lot expect people that contribute to always have done the work on both PostGres and Oracle. It is significantly more work to maintain both environments and test each one as well. It makes the barrier to contribute much much higher.

I'd favor a process where things were clearly labeled "not ye t tested postgres" or "not yet tested Oracle" and then other people can grab that work and complete. It's actually a really great project for newbies to grab something that works in one database and make sure it works in the other.

Collapse
38: Re: OACS 6 and beyond (response to 1)
Posted by Torben Brosten on
Hi Simon,

Yes from an altruistic perspective, the generosity of coding for PG should extend to Oracle, regardless of Oracle contribitors input or other factors that lead developers to judge that Oracle developers do not deserve it.

Iirc, aD supported Oracle and gave away code as part of a business strategy. That strategy no longer exists in the realm of OpenACS. Similar strategies do exist from contributor/vendors that are not finding Oracle part of their market. And the ones that do? Where are they?

Maybe some collective triage process is at work here. Given the available resources, it is not practical to maintain a code base where there is little feedback or interest in sharing available resources. To be altruistic, one has to be responsible and take care of local issues before/while sharing resources that serve others.

A pg to Oracle query converter may be part of a workable solution. I'd like to see one that manages the queries in the xql files instead of generating them dynamically, so there is not yet another layer to process before returning a page.

Perhaps the Oracle community could designate a commercial entity to manage maintenance of the OpenACS core as OpenACS core *and* Oracle evolve, or host a test server with Oracle for testing.

It would then be up to the Oracle community members to find some equitable way to share the burden that fits well in the elite Oracle enterprise environment --call it a joint-venture perhaps, or just write it off as advertising and put your favorite company banners at the top of each page.

Simon, you write: "Expecting the oracle community to fix bugs that one has knowingly introduced is simply not practical." What? Is it that small and resource starved?

Why discourage code improvements of any kind? The manner of style is artful perhaps, but an iso9000 regimented methodology is just one way to continual improvement.

ps

aD's exposing the db to the server is just one technical strategy that OpenACS implements. As Jade and others point out, the tcl api provides a consistent environment for working with multiple databases among other things.

Collapse
44: Re: OACS 6 and beyond (response to 1)
Posted by Janine Ohmer on
In this and previous discussions about this there is always a sense that the Oracle folks are somehow freeloading, that a lack of contribution from those who use Oracle is what is causing Don to put in all these unpaid hours. But I think that is misplaced. Nothing any of us who use Oracle have done is causing Don to have to do this. The cause is the lazy arsed programmers (Simon, did I get that right? :) who are committing broken code. I fully agree with Simon - as long as the project officially supports two databases than in my opinion a submission that only works for one database contains a bug, just like any other kind of bug. Which Don has been ending up dealing with, and I also fully agree with him that he should not have to do that. But I don't think it's any more fair to say that we who use Oracle should have to pay to have other people's messes cleaned up, that we had no involvement with at all.

If you think that monetary contributions are the solution to the problem, then each person who commits code that does not support both databases should have to pay for someone to complete the work for them. Sure, it would cut down on commits, but it would be fair. More fair than any of the other suggestions so far.

The basic fact is that the rules have not changed; the project supported two databases when most people here today started contributing to it, and it still does. The fact that fewer people use Oracle these days makes it tempting, and to some acceptable, to stop spending time on it. But IMHO that does not make it right.

I support the dropping of Oracle support because I think it's the only solution to the problem that has any chance of success. But I do object to being made out to be a bad guy in this.

Collapse
Posted by Don Baccus on
How many people who want oracle support participate in bug bashes (we've had two this month)?

How many step forward to agree to implement new features agreed upon via the TIP process for Oracle?

How many step forward to to provide test servers running various versions of Oracle they'd like to see supported?

How many agree to test beta releases on Oracle?

In absolute terms, it's not the number of hours I spend, it's the principle, the fact that I'm the only person in the community who's made the commitment to make sure oacs core releases work for at least one version of Oracle (9i).

You are essentially saying "people should not work on core unless they pledge to install, test and support Oracle as well as PG".

Despite the fact that they never use Oracle themselves.

Come to think of it, of those of you using Oracle, how many work on the core at all? Yes, many of you did in the past, but now? How many?

(AFAIK the question to all of the above is "zero")

No one is saying y'all are "bad people", we're just saying y'all need to step up to the plate, get involved, and help support the platform.

The expectation by some seems to be that no one in the Oracle community need be involved in development, maintenance and testing of the platform.

That's wrong.

Collapse
51: Re: OACS 6 and beyond (response to 1)
Posted by Don Baccus on
Oh, next bug stomp is August 10-11, I believe.

Hope to see some of you Oracle enthusiasts there!

Collapse
52: Times are changing (response to 1)
Posted by Nick Carroll on
It is interesting to reflect back on the second posting made when the openacs forums were first opened. The topic of discussion then was where does ACS/pg fit in? People were concerned back then of how PG will keep up-to-date with Oracle.

Today, it is a different story. We are now concerned with keeping Oracle up-to-date with PG!

When I came across the OpenACS community 5 years ago, I found a project that wanted to commit to a full open source stack (web server, language, db, os). The main reason (I think) was to distinguish OpenACS from ACS. I think it was always the goal for OpenACS to support PostgreSQL. This was probably why scripts were created to port Oracle queries to PG queries back then, so as to make the transition from Oracle to PG.

Things changed when Arsdigita got swallowed up. Those that were able to snatch up some of Arsdigita's clients ended up continuing with ACS support, as well as transitioning some businesses to OpenACS, whilst keeping data in Oracle. Since a clean break from Oracle could not have been established in OpenACS, Oracle support was continued.

Now we have reached another interesting point in the history of OpenACS. Do we continue with Oracle or not? I personally would like to see OpenACS become a completely open source stack by severing ties with Oracle. However, there are those that rely on Oracle, as they believe it to be a marketable feature. If this helps them get business and spread the OpenACS gospel, then so be it, but they should also commit resources to supporting their preferred DB, instead of just leeching off others.

Within the last 2 years there has been a lot of discontent with supporting oracle. This occurs mostly in irc, but also appears in the forums from time to time, so it isn't something that "just happened". During this time nobody has stepped up to defend Oracle. So most of the people that have been contributing code were under the impression that nobody uses oracle, except for a few legacy ACS systems. I know I was thinking that. I didn't know people were still wanting Oracle support until this thread.

For me, supporting oracle is a problem. It means having to write more code than what a client is willing to pay for. I have not come across a client that is willing to pay for extra time spent on supporting a database that they will not use. In fact I have not come across a client that has demanded Oracle, so I welcome the drop in support for Oracle.

I find this outrage about dropping oracle support to be quite amusing. If nobody has been supporting oracle for the last two years, with the exception of Don, then what makes people think that creating a new openacs/oracle project will suddenly muster community support? I would like to see this happen, I really do, but I'm afraid it is just hot air, just like this thread. It would be easier to just maintain the 5.3 branch.

Moving forward, I would like to see the Active Record pattern employed in openacs, as is done in RoR and Turbo Gears. Perhaps we can look towards using xotcl for this. That way we won't have to bother about strictly supporting a specific type of database. We might get there, but it would probably be a few years away if we ever decide to go down that path. Hopefully it will happen in time for OpenACS 10, aka OpenACS X! ;)

FWIW. I have been using a database abstraction layer for a couple of years now (written in XOTcl). It is not perfect by any means but it does enable you to write database neutral code.
  • How do I define the data model of my application?
    
    DB_Class Blog_Item -lmap mk_attribute {
    
        {String body}
        {Timestamptz entry_date}
    
        {Boolean allow_comments_p -isNullable no -default 'f'}
        {Integer cnt_comments -isNullable no -default 0}
        {Timestamptz last_comment -isNullable yes}
    
    } -lmap mk_like {
    
        ::content::Object
        ::content::Title
        ::sharing::Flag_with_Start_Date
    
    } -lmap mk_index {
    
        {Index entry_date}
    
    }
    
  • How do I store a new data object?
    
    set bi [Blog_Item new -mixin ::db::Object]
    
    ${bi} set id $id
    ${bi} set title $subject
    ${bi} set body $body
    ${bi} set shared_p $shared_p
    ${bi} set allow_comments_p $allow_comments_p
    
    ${bi} do self-insert
    
    
  • How do I retrieve data?
    
    set blogdata [db::Set new -type ::Blog_Item -order "entry_date desc"]
    ${blogdata} load
    
    foreach blogItem [${blogdata} set result] {
        # do something with the data, for example:
        ns_write [$blogItem set title]
    }
    
  • What else does it do?
    • Partitioning -- See http://www.oracle.com/technology/products/oracle9i/datasheets/partitioning.html
    • Scoping -- You write your code once but you can apply it to a different scope every time. That means, you no longer need to have the same database table for forum messages in your installation, i.e. each class/community/subsite/user can have their own tables which of course improves performance.
    • TCL-level triggers, onInsertSync, onDeleteSync, onUpdateSync -- For writing aggregators, for example, maintaining forum summaries is a form of aggregation. Aggregators in this abstraction layer are maintained/updated upon insert/update so that you no longer have to do a "select * from forum_messages order by creation_date limit 10"
    • You can join XOTcl defined data classes with database tables which have not been defined as data classes -- For example, you can join against the OpenACS Users table eventhough it has not been defined as an XOTcl class (see above). This is done easily by replacing "-type" with "-from"
There are much more that can be said here (how easy caching would be, etc). Eventhough it has served me well, there is a downside. The downside is that you would still need to invest time bringing it up to the needs of the corporate users in the community as well as writing the Oracle Translation/Interface Driver.

Being an infrastructure package, though, means that any investment that would go into it (either the package itself or related efforts to bring developers/packages up to speed) would only need to be done once.

The reason I have been silent about this is because I'm already pursuing full persistence (i.e. eliminate the need for foreign keys which they now exist in the definition of the data model classes and replace them with conceptual classes and relationships between them -- the persistence mechanism decides how to organize attributes/data into tables and it does it in the same way an expert developer would).

Collapse
Posted by Nick Carroll on
Neophytos,

You have been doing some interesting work there! When do you think you will be finished with this? Is it something that will be added to the openacs repository? If so, then it would be great if we can coordinate this with the planned 6.0 release.

Cheers,
Nick.

Collapse
Posted by Don Baccus on
How does this database neutral persistence level deal with the difference between CONNECT BY tree queries and datamodels vs. tree_sortkey datamodels and their associated queries?

The two aren't comparable.

Now 10g's relaxation of restrictions on CONNECT BY queries actually helps a bit (because you can join tables against CONNECT BY subqueries) ... I can actually visualize being able to build similar queries for both schemes via subqueries and joins against them ... but they wouldn't scale well. Scaling tree-building queries using the two models requires different strategies.

We've talked about tcl-based table declarations of the sort Neophytos talks about in his xotcl stuff.

The problem is that we need to clean up the somewhat conflicting attribute systems in the content repository and that for base objects.

Lots of datamodel clean-up.

WITH LOTS OF UPGRADE SCRIPTS.

This grunt-level work would need doing for both databases, and of course would need to be EXTREMELY well-tested in both environments. With realistic data representative of real sites.

The need to provide an upgrade path is a (NECESSARY) millstone around the neck of those that would like to clean-up the datamodel as a first step towards making a much cleaner and easier development environment for programmers.

The need to provide upgrade paths for two databases is a double-weight millstone. Worse, actually, because as is pointed out, I've been the only person committed to keeping Oracle supported.

In short, the higher-level API path (exposing as much as possible through Tcl API as we've been increasingly doing the past couple years making it easy to write db-independent code, or an Xotcl approach such as Neophytos favors) helps the easy problem.

Unfortunately, if that were the only problem, Oracle wouldn't be a big deal.

But fundamental clean-up of the datamodel combined with upgrade scripts is a much harder problem, and eye-candy help such as higher level APIs for the easy part of the problem isn't all that beneficial.

Don,

On CONNECT BY: Rails implements nested sets and trees on top of either of the three databases -- Oracle, PG, MySQL. Maybe there are a few lessons that could be learnt from it.

There does seem to be some performance impact in doing it the way Rails does, but having Tcl inside of AOLServer may just as well neutralize it, don't you think?

Well, we support trees on top of two databases, that's not the issue. The issue is simply that this particular need is one which drives db query differences.

If you're suggesting that Rails does the nested tree support in the Ruby layer, I'd humbly suggest that this is likely to be terribly inefficient if sizable trees are involved.

I should also point out that the forums package's oracle version uses tree_sortkeys rather than CONNECT BY queries.

And that PG in the semi-near future should support SQL's recursive queries that would allow an approach similar to CONNECT BY.

Hi Guys,

I am not an active participant nor a frequent user of openACS. But after going thorught this long forum postings, I have a question.

Can someone tellme what exactly is the reasons of dropping the oracle support? It would be good if we know exactly the work/effort required then we can collective divide the work and start supporting openACS with oracle.

1. Is it because of lack of Oracle users in the community who will test oracle code and give green signal to it?

2. Since Don is the only person doing this and he is not interested in doing and no one is there to take that responsiblity. So we are dropping oracle support?

3. Or is it because of licensing issues of oracle?

What exactly are the reasons of dropping oracle support? We know that upto now openACS was suppporting both databases and most of the websites are running on openACS having backend both oracle and postgres. Can someone (Malte, Don or Dave) please actually list down the things that is required to support oracle with openACS 6

I see most of them had shown anger/disrespct to other users but again its not directly from the heart but its more about the frustration of not having oracle support.

Why dont we actually know the things that is required in order to continuing support oracle?

I will be waiting for the answers then..

Thanks,
Andre

Hi Andre,

1. Yes Oracle maybe dropped in the future if no one will step up in testing it.

2. Currently its only Don who does the testing, its difficult and not a sustainable model as Don is not really doing Oracle development

3. Its not a licensing issue, its a manpower issue.

I hope a member for the OCT can confirm my answers.

Well, since this discussion has started, people have stepped forward offering server support and the like. So perhaps some of the resource issues will be resolved. We shall see.
Collapse
54: Re: OACS 6 and beyond (response to 1)
Posted by Neophytos Demetriou on
The database abstraction layer is ready (full-blown conceptual persistence is not, *yet*). I can commit it as soon as I get the ok. As I say I above it is by no means perfect. It was developed with scalability in mind (i.e. how would you build a site like yahoo.com) so it lacks some features that corporate users might need (or the developers doing contract work). It is, however, a starting point.
Collapse
Posted by Malte Sussdorff on
Please do us a favour and release the code somewhere. Furthermore, we might want to have a TIP to officially include this in 5.3. Maybe the acs-object TCL API could already make use of this abstraction layer?
Collapse
Posted by Don Baccus on
I don't want it in 5.3.

1. As Dave and I have discussed 5.3 (though we haven't run it by everyone on the OCT yet), the goal for 5.3 is to have an acs core that passes all automated tests. Getting to this state has been the focus of the last two bug bashes and, since there are still a bunch of failing tests (almost all in the "fails to meet coding standards", not "code doesn't work" category), will be the focus of the August bug bashes, too.

This is a very modest and very focused goal.

As a result, we've discussed branching the code in September, with a release date in October. By our standards, that's light-speed, no time for fancy schmancy stuff.

I'm committed to continuing the release manager role on (presumably, or rather necessarily given the realities of the community) a volunteer basis to get this out in that kind of time frame.

2. It is possible that Neophytos' work is the be-all and end-all of persistence layers that can be developed in Xotcl.

But there's no way I, at least, am open to deciding to settle on a new API that I haven't even seen yet!

Also, to be useful in the ACS model, it needs to deal with object attributes when defining datastructures. We should really be pushing for something along those lines. We have TCL-based code available that does it, adding such a feature to an Xotcl layer would be easy enough.

Though as I said in another post above, taking full advantage requires our finally biting the bullet to clean up the datamodels. Also we really need to be adding attributes to existing objects where folks don't do them if we really want to leverage stuff like being able to build forms and pages directly from the object model (code which exists in the CMS and elsewhere but which has been little used).

And, if we were to adopt a new API such as a persistence level, deploying it without rewriting acs core would just add another level of complexity to the code that newcomers would have to conquer in order to participate. The barrier's already too high with too many competing APIs and programming approaches. Adopting a new API means, I should hope, taking the time to rewrite core packages to actually use it.

Not something that's going to happen this August :)

No, it's more a Oacs 6 thing in my mind.

Collapse
56: Re: OACS 6 and beyond (response to 1)
Posted by Malte Sussdorff on
"Malte made a more-or-less unilateral decision to stop doing so. How do we stop him from doing so? How do you force him to support Oracle FOR FREE?"

Force wont be helpful at all, probably even counter productive (take into account that at the moment we write -oracle.xql files and /oracle/upgrade/whatever.sql if we are aware that the code we change wont work on Oracle. Its not that we lost our AD training in Oracle altogether). But we are not testing it anymore as we do not need Oracle for the customers we are serving.

What would be helpful though is a server with an Oracle installation of OpenACS, where we could get ssh access to and test the upgrade scripts and other oracle.xql scripts. Let alone run automated test cases. It should be easy (read: "sh reinstall-db.sh") to reinstall the database in case my upgrade script does not work. And as Simon was willing to contribute a server to the split Oracle project "I can volunteer to provide a server and responsibility for maintaining a new community around the oracle version.", maybe he would also volunteer for the provision of such a testing server.

The goal should be to make it as easy as possible for contributors to submit and share their code. And if you ask for the core to remain with Oracle, but are not willing to port things yourself and say that it is in the responsibility of the contributor to commit for Oracle as well, make it as pleasant for him to do so.

Don, please correct me if I am utterly mistaken, but I think the number of database specific commits on core has decreased considerably with the extension of our API. Furthermore, with Neophytos database abstraction layer, the whole point might become mute (just as an example: when writing the invoice package we only had to 7 out of 39 .xql files postgresql specific.)

When I said we are going to drop Oracle support in version 6 it is not entirely true. Depending on your point of view we dropped Oracle support the moment I was forced to ask Don "Hey Don, I made a change in core, committed the Oracle part, could you test it please". In Version 6 we will most likely not bother to write the -oracle.xql files anymore, unless we have an easy way of testing them. We are getting new employees on board, and having them learn Oracle syntax despite the fact that they are not going to use it in their daily work is a little bit mute. So I would have to do the testing for Oracle for all our commits and trust me, I have better things to do as well.

And note, we are only talking about core packages here. With other packages we have releases of packages, maturity level and the sudden drop of support for one or the other database. Take project manager as an example. In 5.2 the project manager version works with Oracle, thanks to the efforts of Janine who needed it for one of their clients. Before she went down the road we started enhancing the HEAD version of project-manager without writing or testing any Oracle code, because the code base we started from did not work on oracle. To make things easy though, we usually commit PostgreSQL specific stuff in seperate .xql files and do not store SQL code in .tcl files (and yes, I said usually for a reason, we are not perfect in this).

I am sorry that my comment started this whole discussion as I think the resources could be put into use much better e.g.

  • Writing a new developer guide on how to make sure the quality of the commits adheres to the levels we want. I would help in writing one, though not sure if I'm the best suited person given my personal history in this area 😊.
  • Training manual on how to write tclwebtests. I still have not much of a clue here either (which probably puts me clearly in the "hobbist" group Simon talked about).
  • setting up a test server for Oracle and postgres with public access so people could test their commits immediately on a neutral environment (and I added postgres, because if someone writes in Oracle he should not be forced to maintain a postgres installation either).
  • Come up with a clear standard on commits, releases and maturity levels. There has been discussion on this topic, where I voiced my point that you should commit early and often, but be more strict (with regards to quality) once you release and give a maturity level. But a decision and consensus on how we want to achieve the goal (prevent code duplication, allow preview of what people worked on, make it easy for them to commit their work, instead of having it remain on their computers as they don't have time to clean it up, test and release it) never was reached (or I missed it...).
  • Clean up the code of the packages. Mark them with clear maturity levels and release them once tested. I would like to reduce packages that people want to install because they think they work, though they are actually only meant for developers.
Collapse
61: Re: OACS 6 and beyond (response to 1)
Posted by Gustaf Neumann on
I do fully agree with Don, that no one should hold the breath until
we are able to provide a fully functioning higher level api to handle
all the stuff that current acs-core + dozens of packages require.
That's a task for 6.0, and not 5.3.

In the past, i have implemented various active record implementations
in xotcl (see
e.g. https://openacs.org/forums/message-view?message_id=185330), which
is highly flexible but has the problem of accessing the database
mostly by one tuple at a time, leading to bad scalability for some
tasks. The approach used by the for the oo-layer for the content
repository in xotcl-core is much better in this respect, but not a
full-blown generalization. i have seen and discussed the database
abstraction layer and persistence layer from Neophytos, which is in
many respects much more radical, but a very promising approach. But it
needs time. We have hired Neophytos for our EU-funded iCamp project,
so Neophytos will be able to spend some resources for completing and
polishing it.

The big question for me is how radical changes to OpenACS should/could
be in future versions. We have one of the largest installations of
dotlrn and have therefore as well interest in a stable core and dotlrn
packages, but at the same time i have often desires to change
fundamental pieces of OpenACS. We are in competition with other
environments catching up. To keep the lead, we must be able to make
significant changes. Just to use xotcl and keeping things as they are
does not help much in this respect. For example, as discussed earlier in
the forums, i have re-implemented ad_conn in xotcl, which is faster by
a factor of 10 compared to the current implementation. However, this
plug-compatible implementation tries to deal with the idiosyncrasies
existing in ad_conn. This is by no means a nice and well-designed
piece of oo software. I have no big interest of giving the software
out of my hands since i am rather afraid that someone would see the
code and ask me what strange kind of software i am writing. Why?
ad_conn is for example "always around" as a command, has its own logic
of being configured and initialized etc. Conceptually ns_conn and
ad_conn are objects keeping connection specific information. These
objects should be created when a connection thread becomes active and they
should be deleted when the connection thread has finished processing
an http query. There should be a connection class with the ability to subclass it. In
trying to be 100% compatible with the old version, it uses the strange
ways of being configured which are not needed in an oo language. I
made very similar experiences with templating. Although OpenACS is
object based in many respects, turning it into an true oo architecture
based e.g. on xotcl will change many things. If one would try to do
such a transformation piecewise on the existing acs-core, this would
require many TIPs, some of these would be rejected by developers with
many customers running legacy code.

From my point of view, the only approach for substantial redesign
within OpenACS is sideways development. The new code must be able to
co-exist with existing code, which is certainly possible with the
AOLserver. Another approach would be to start all over with a new
project - but this will take even longer until it will be ready for
primetime.

My goals are not to improve the speed of certain functions by an order
of magnitude, but to create the best and most cool web framework to
attract people joining our forces. To achieve this we must be more
attractive and better than RoR (which is not so hard). XOTcl is so
much more flexible than Ruby, so we have our chances. If we don't
succeed to create a vision, the fancy past of Alex and co will fade
over time.

In some respects, xotcl-core and xowiki is for me a project to test
our some oo ideas within OpenACS. Although i am not a friend of
announcements about things to be available in the future, i have to
say that we are pretty far with xosoap and xorb, which is a client and
server web-service environment for openacs based on xotcl. It will be
released in the not-to distant future. Neophytos uses my COMET
streaming technology (part of xowiki chat, see
https://openacs.org/xowiki/pages/en/chat) on phigita.net and he seems
very happy with it. We did already some work on templating, and with
Neophytos database and persistent layer, we are on a good track.

Concerning the Oracle thread: The main issue is maintenance. In the
best of all possible worlds, we are able to generate in the future
most of the application packages code. If the generated code has
interfaces for handcrafting, and is able to co-exist with a small
fraction of manually maintained code, maintenance should go
significantly down.

Collapse
Posted by Rocael Hernández Rizzardini on
Gustaf, moving oacs to oo might be the next path, specially to grow and distinguish, maybe you can set a wiki to put together your ideas, and the interested parties can comment on that.

Besides technology, making easy for new commers will be important factor if we are going to put an effort, this means easy ways to learn and be productive.

Collapse
Posted by Jun Yamog on
Moving to OO doesn't mean its going to be easier and productive. It may do it considering starting from 4.x the db schema is not traditional and OO-like but its not guranteed.

I was chatting with an ex-aD developer regarding this thread. Contemplating on history, it MAYBE all the power and complexity of ACS 4.x that scared new comers. That wasn't much an issue during the 3.x days, in fact its just like doing PHP only much more sane and easier. The 3.x codebase was easy to pickup and the output cycle was shorter (near instant gratification to a developer).

Anyway I don't want to muddle the thread any further since the decision to make ACS OO and make it powerful yet complex was decided years ago by aD.

Collapse
Posted by Rocael Hernández Rizzardini on
Easy to learn sometimes is just at least: good tutorials & wikis, and ways to keep them up-to-date (probably making them part of the process for new pieces of code that goes into the core), these is something that we don't have even with the actual openacs.

Probably the target audience is not newbie programmers such as PHP seems to have. We might like to target professional - competent programmers that OO, DB and webservers is not something new, probably already mastered, those that care about programming, then those ones can try, realize and see the power of openacs as a web framework.

Collapse
Posted by Malte Sussdorff on
I just had a discussion with an experienced PHP developer that I converted to OpenACS. He mentioned what is irritating is the fact that you have to deal with three files for one page and that variables are automatically set in each of them, without an explanation.

Example:

- If you call db_* you do not know which variables are returned in the tcl statement. So if you are searching for the variable you have not much of a clue, unless you look into the .xql file(s). So extending db_* calls with a documentation part that allows us to write which variables are expected to be returned could help

- This is especially true if you enter the variable in the .adp file without even calling it in .tcl.

But I agree with Rocael's assessment, we should not try to target the same group of people that PHP is targeting, for this OpenACS is just too complex as a framework.

One note though on XoTCL. Before we move forward in this direction, we definitely need training materials which describe common tasks as done before and how they are done afterwards, so we do not loose the experienced developers that OpenACS has at the moment. And obviously we should stay backwards compatible, so the old style of development still works.

Collapse
62: Re: OACS 6 and beyond (response to 1)
Posted by Don Baccus on
We have hired Neophytos for our EU-funded iCamp project,
so Neophytos will be able to spend some resources for completing and
polishing it.
That's excellent, Gustaf.

I see nothing I disagree with in principle in your post.

Maintenance and development for application code is already much easier due to having implemented more and more stuff in the Tcl API, making it less necessary to write queries.

The big issue I see is that if we do decide to rationalize the datamodel doing so for both Oracle and PG will be a very large task because of upgrade scripts, mostly.

They're painful to write, painful to test, and testing's useless without meaningful real-world data to test with.

Coming up with a better application code environment is important, yes, but rationalizing the datamodel is equally important.