Forum OpenACS Development: OpenACS 4 Search Integration, what should it look like?
integration strategies. Since I haven't found a site indexing engine
that I like, I am of the belief that the OpenACS should support many
different types of site indexers and search engines.
It's no revelation that many sites already use other search engines
(htDig, SWISH, mnogo, even Microsoft Context Server (is that the
name?) and it would be beneficial (and even synergistic) to be able
to integrate the OpenACS into those sites, letting them keep their
same search engine technology. Ideally, I would like the OpenACS to
come, out of the box, with a search engine that works for any OpenACS
supported platform: *nix or Windows, Oracle or Postgres or ...? Then
a developer would have a working solution but would also have the
ability to change out/swap/tune that solution with other appropriate
technologies.
Here's a partial list of search engines that have been proposed for
OpenACS sites.
a) Intermedia
b) OpenFTS
c) Lucene
d) htDig
e) SWISH<fork>
f) mnoGoSearch
g) ???
What a panoply of technology! We have C, C++, Perl, and Java. We
have GPL and not. There are engines based on site crawling, engines
based on db crawling, and those that would do both. Some of these
engines are specific to a given database (Intermedia/Oracle, and
OpenFTS/Postgres).
My proposal is to see a "search engine driver" based approach to
search engine integration. This would consist of two sets of
interfaces that each site indexer would have to support, with
the "promise" that once the support is present, the site indexer
would plug right into the ACS.
On the frontend, a search engine would support an aolserver module
(specifically an aolserver db driver like interface) that exposes
methods to open and close an index, to specify a search, to retrieve
results, and to allow a limited amount of introspection (is this type
of document handled, are excerpts returned, etc.) This would
let /search pages handle all sorts of different search engines
generically in the same manner that ns_db or db_ lets us deal with
many different types of RDBMS.
On the backend, a module would support some sort of "search me"
callback interface similar perhaps to the "ACS new stuff" interface,
or similar to the ACS Intermedia interface. As glue then, support of
a site indexer would consist of code that runs through the "search
me" callbacks or scans the ACS Intermedia table returning the various
documents to the site indexer, and perhaps ACS-wide parsers that
might help an indexer understand ACS objects when encountering them
as embedded links within a document (users, bboards and bboard
topics).
With a bit of experience, I would prefer a callback scheme to an ACS
classic/Intermedia table based scheme. There is a conflict within
the ACS today: put keep all the content in the db vs. allow some of
the content to reside in the file system. A callback scheme lets the
module determine where the content lies. A callback scheme would let
the site indexer index remote content (via a callback that supports
XML-RPC or SOAP, or any sort of webservice). A table based scheme
may cause the data to be replicated one more time: it's somewhere in
db or file space in it's primary format, it's replicated in indexes
within the site indexer, and we then ask it to be stored one more
time within the table to be scanned.
What do you folks think? Is this an OpenACS requirement,
distraction, or bloat? What should these interfaces look like?
Callbacks or table driven?
distraction. I'm fairly confident that OpenFTS will provide a number
of good solutions to the problems posed above, but you're right that
it may not be enough.
In theory, I like the design you're pushing. I am worried about the
efficiency of the call-back mechanism you're talking about. We have to
worry about updating indexes when content changes, either live or
batch mode, and we have to have this be efficient... I'm not saying a
callback interface isn't possible, I'm saying the efficiency issues
are going to be non-negligible.
Jerry, do you have time to lead investigation on this front? I'd love
to have you handle this, in cooperation with the OpenFTS folks.
It's called the Content Repository and the reason I keep hammering home the point that all content should be stuffed there is precisely so any indexing facility will have only have to talk to the ACS in one
place.
Thus the supposed conflict you mention no longer exists. Making it disappear is the *point* of rewriting the CR to allow content to optionally be stored in the filesystem.
It may be that the CR API isn't quite up to the task of handling content stored in arbitrary locations within a server's file system, but if that's true we'll just need to modify the CR API to make it so.
Seamlessly dealing with remote content is more difficult, but I'll personally be very, very happy to see the lesser problem of providing decent search functionality for PG-based OpenACS sites solved before I worry too much about this additional issue.
And of course we'll need to keep the CR as lightweight as possible when you're not dealing with content in the db. Since any scheme to track filesystem content will want to be hooked into ACS permissions, which implies that at some level the scheme will be built on top of ACS Objects, the reality is that the CR is about as lightweight as any
such scheme will be.
Unless we decide we want a parallel and separate package to deal exclusively with indexing issues, which I'll lobby long and hard against.
If someone wants to sit down and write alternative indexers that operate on the CR, that could be substituted for InterMedia or the OpenFTS solution, sure, they can go for it.
At the moment, though, I consider this to be more distraction than useful. I don't look at the search problem as being a critical path item for the current incarnation of the OpenACS 4.x project. We have an Oracle solution and we have what appears to be a very workable Postgres solution. Yes, we hope to support other RDBMS systems in the future and I'd love to see additional search/indexing options available in the future, too.
But ... there's a bunch of "right now" issues that are more important,
IMO. If you have time on your hands, Jerry, why not pick up one of the unclaimed OpenACS 4.x packages and port it?
I would love OpenFTS to be wonderfully terrific, but what if it's not? I know it's open source, but I have enough time dealing with OpenACS issues....
I guess my concerns are several:
One. With little information and little user experience out there regarding OpenFTS or its performance, reliability, or stability, I am concerned that relying on OpenFTS for OpenACS Postgres search is risky for OpenACS users. Will it hold up under system load? Is it actually faster/smaller/better than any of the other solutions? Will the initial releases be robust and will releases make it out in a timely fasion? How does anyone know this? Is OpenFTS really far enough along that you're ready to narrow the design space?
Two: I want to make sure there is a cheap 80% solution (that is, the solution that costs 20% and solves 80% of the problems):
Right now, I have seen htDig and SWISH<fork> work with OpenACS and explored their use and implementation. I know a good 80% (or 90%) solution can be made of them. I can discuss their performance requirements and their reliability. I can reference many sites around the net that are using these solutions and talk to the folks that have implemented them to discover how these things work in reality. I can't do that with OpenFTS, and it doesn't look as though I will be able to do so except in the long run. I can't tell if OpenFTS is a 20/80 solution, but it doesn't appear to be.
I am concerned that emphasis on two db crawlers (Intermedia and OpenFTS) will focus/narrow/bias the OpenACS interfaces in such away as to make it difficult to implement the cheap 80% solutions.
Three: I would like to find a solution for OpenACS 4 that can be used now with OpenACS 3.2.5. I am not sure how migration from 3.2.5 to 4 will work, and so I suspect there will be 3.2.5 sites out there for a long time. I would prefer to master one set of search engine dependencies if possible.
Four: I already have dependencies on AOLserver and OpenACS, two organizations working very hard that still don't have enough resources. I am concerned about adding another dependency on a fledgling search engine without much track record and without many developers regardless of its promise, and regardless of how well it may be performing on one or two demonstrator sites.
Five: when you write, "If someone wants to sit down and write alternative indexers that operate on the CR, that could be substituted for InterMedia or the OpenFTS solution, sure, they can go for it" I worry that emphasizing OpenFTS/Intermedia/db crawling now will limit us as to what we can actually substitute in later on.
Think of today's big limitation of AOLserver -- Tcl. The implementers of AOLserver knew/wanted db neutrality and extensability and lots of db support, so they created a simple db driver interface that lots of things could implement: solid, sybase, oracle, and even postgres. They wanted communications neutrality and extensability so they created a simple communications driver interface and we have nssock, nsopenssl, nsssl, nsvhr, nsunix. But, they targeted Tcl and Tcl alone for scripting, and so today we don't have wonderfully integrated python, java, perl, javascript or ruby solutions with AOLserver. Merely Apache and Tomcat. Once again, I worry that by examining/emphasizing db crawlers today (Intermedia and OpenFTS) we will overly restrict the search engine/ACS interfaces.
Six: OpenACS integration into existing infrastructures. By focusing on OpenFTS now, we make it that much harder to sell OpenACS into an organization already using brand foo. "Well, you need two search engines, and your existing indexer will never really work as well with your OpenACS content as it does with your existing content. What we really ought to do, is get rid of your other search engine and replace it with our OpenFTS search engine." This may well be the best thing to do technically, but it is a harder sell and may not have been required to produce an requirements satisfying site.
So what am I missing? I've visited several OpenFTS sites, including the one in Russia and the one's mentioned by Google. Why are you so confident that OpenFTS is the way of the future and will be reliable and timely for use in OpenACS?
The code base has also been used to index the 160,000 posts in the various postgres mailing lists, and it seems to be working well there, too.
Neophytos has posted this information in the past, and the current status.html file makes mention of it and includes and invitation to e-mail him directly if you have further questions.
Since you do, why don't you?
As far as the rest of your rant goes, if you're tired of Tcl, like Apache and Tomcat, and are primarily concerned with being able to sell
OpenACS into existing organizations who are more interested in buzzword compliance than in capabilities, well, then I think ACS 5 will better fit your needs than OpenACS 4.x.
Besides which I'm unconvinced that either swish or htdig have the buzzword compliance sheen necessary to make them an easy sell into these organizations.
If you really want a swish solution to be available in OpenACS - do it! Build it on top of the CR and it can be offered as an alternative
search strategy. As I said in my previous note, I'd love to see alternatives available.
But you're not really suggesting an alternative. You're arguing that
we not pursue the workable solution we already have, and that ain't going to happen.
Mostly you're arguing that we do things your way, though, aren't you?
That argument would carry a lot more weight if you were actually contributing to the project. The OpenFTS solution has floated to the
top of the list because Neophytos (with some urging and advice from me) picked it up and ran with it, including helping the Russian company that paid for the implementation figure out licensing issues.
Swish was on my original status sheet as an option, but no one picked it up and ran with it. So I took it off. If you want to see it implemented ... implement it!
Jerry, I'm not sure what you mean by AOLserver being the bottleneck. Is it because there aren't enough ways to plug other languages in? Petru Paler is working on a FastCGI implementation so that OpenACS can work more seamlessly with Apache. And there is the PyWX project that I haven't heard much from lately. But even if these lead to nothing, I still don't think we lose as much as we gain by using AOLserver, other than the aforementioned buzzword compliance.
talli
Ignoring the straw men you set up, you're claiming that Jerry is arguing that we not pursue the workable solution we already have.
What he's arguing is instead of locking ourselves into one solution (like aD did with Oracle... sound familiar?) we do it right the first time around and have a search abstraction layer so (a) it's much easier to use other search engines, and (b) we can easily allow an rpc interface. Nobody is saying OpenFTS shouldn't be a search backend, but we shouldn't shortsightedly keep ourselves from using others, either.
Speaking of strawmen, the only person I hear raising that issue is Jerry.
Months ago I made it clear that I thought it would be good to have a non-db-based as well as db-based solution. My first response in this thread repeated that point. As did my last.
There's only a single solution in the works at the moment because no one offered to pick up Swish or htdig integration as a work item back when I had it listed as a possible solution in need of a champion.
That's the one and the only reason. If someone had picked that up and
if no one had volunteered to investigate the "Russian solution" then I'd undoubtably be accused of locking us into Swish or htdig.
Right?
Do I need to do anything more to knock down that strawman? Do I need to drench it in gasoline and light fire to it, as well?
As far as abstracting out an interface to indexing tools, the place to make the hook is the CR. As long as we follow that paradigm then adding additional indexing tools in the future shouldn't be a big deal. Adding them now shouldn't be, for that matter.
Yes, Jerry has worked on the xml stuff. When I said he's not been contributing I was referring to the OpenACS 4.x project - if OpenNSD were actually viable and active, the xml/soap stuff would more properly be hosted and championed there because it is in no way an OpenACS-specific/only tool. Jerry has done other good things in the AOLserver space.
I will repeat that I don't see this as anything close to a critical path item for project completion. My goal is to get a releasable product done in reasonable time. I'm convinced that OpenFTS will provide an adequate search solution. I've used it on the postgres site and it works quite well. Others report being happy with it as well.
I personally believe that those of us who are looking to complete OpenACS 4.x soon should be focusing our efforts on other things at this point in time.
Those who don't hold that goal are certainly welcome to do whatever they please, but I'm not going to let go of my focus and I'm going to resist efforts to dilute project resources in ways that work against achieving the goal I'm focused on.
Actually I didn't set up any straw man at all, which m-w tells me is "a weak or imaginary opposition (as an argument or adversary) set up only to be easily confuted."
What I did is often more properly labeled as "stating my concerns." I didn't answer them, and I asked others with more experience and vision to answer them and clarify the situation.
I believe you may have set up a straw man when you misstated my post and my intent as asking for us to not pursue a workable solution and then flamed me a bit more for not offering any alternatives. (And then you got a bit personal and suggested I mostly just wanted folks to do things my way. I honestly don't know how that could be, since as I started off this thread by saying, I haven't yet met a search engine I like, and gosh, I'd really like to talk about that. You also suggested that OpenACS wasn't for me (and presumably I wasn't for it and maybe I should move along.) Yeah, that kind of hurt, and so I thank everyone who came along to my defense.
So just to repeat: I never said we shouldn't pursue OpenFTS, in fact I said I would probably be a user of OpenFTS. I said I was concerned with OpenFTS's maturity at this stage of the game and I said I was concerned that an OpenFTS focus now might lead to an API that was db crawler focused. What I didn't make clear was that by having "n" modules implement that API would then lead to enough inertia in the system as to preclude a web crawler based solution later. I offered the alternative that I started this thread with: a discussion on what search engine requirements are and what search engine interfaces should look like. And then I explicitly asked where my assumptions and concerns were misplaced.
I didn't expect the Spanish Inquisition (does anyone?) (that by the way is homage to Monty Python AND Philip K. Dick.)
Now perhaps you haven't been following the forums lately, but in fact, I have been working on SWISH<fork> and htDig integration with OpenACS 3.2.5 to solve some client problems and to learn enough to redo it in the OpenACS 4 framework. I apologize if my timeline wasn't your timeline, but the fact is, I have been trying to run with it.
It appears your response is: because of the content repository's central role, you don't believe that a solution developed first for OpenFTS and Intermedia will preclude other technologies; that your personal experience with OpenFTS to date suggests to you that it is stable and capable of the job; and that the OpenACS 4 release is near enough that you believe now isn't the time to reopen this issue.
Is that reasonably accurate? Anyway, if so, that's a justifiable response for the project lead to make, I hope you're right, and I'm behind you. But you sure pissed all over this thread, and I for one, don't feel particularly welcome here or at all eager to figure out just what I might have to do in the future just to get you to state that I have contributed to the project.
And you would've known that there's interest in supporting non-db indexers and that the decision to use OpenFTS wasn't meant to preempt the adding of other search technology in the future.
If you're interested in helping, you could, for instance, offer to help Neophytos and Dan implement a reasonable abstraction layer to tie together the site-wide search and content repository package. They might take you up on your offer, or they may not. They own these tasks and it is up to them. They may not be interested in addressing anything at the moment beyond making the handling of intermedia and OpenFTS transparent a the client level in our first release. If that's true, I'll support them, because we need to get this release rolled out in a matter of weeks, not months, if at all possible.
If you do want to help, you'd better hurry - Neophytos is planning to dig into the site-wide search package this weekend. He works fast.
Well, if you'd done just a little bit of research you would've known that the CR was rewritten to allow both filesystem and db storage of content, so there's no need to propose a solution to this already-solved problem.Well the research I did before posting was to
- participate in a lengthy discussion evaluating various OpenACS search strategies from htDig to SWISH, to something I now recognize as OpenFTS: here's your confident support of it just two months ago:
possibly synergistic efforts include an in-database search engine being developed by folks who run a Russian portal. If someone's interested in tracking this down and evaluating it both for usability and completeness (i.e. are they done yet?) e-mail me. I know that to make it work very well on their portal site they had to make a change to the PG core's optimization and evaluation of limit that Tom Lane didn't like all that well.
- implement htDig and SWISH on OpenACS 3.2.5
- sets up an htDig based search for OpenACS.ORG and OpenNSD.org that lets users use boolean, wildcard, fuzzy, misspelling correction, stemming, date ranges, while also letting the user specify which portions of the site to search. This demo lets folks index their word, excel, and pdf files. (This seemed like a damn fine search to start with, better than offered at /search, but I now understand how unsophisticated this search actually is.)
- searched OpenACS's /bboards, /doc, and /wp for the word OpenFTS
- read BOTH threads that contain the word OpenFTS in them (prior to this thread)
- tried to discover (but failed) where the OpenFTS discussion came from (the main OpenFTS thread starts off by saying this is an update)
- had a brief email discussion with Dan in which he seemed to agree that OpenFTS was PG centric, that Oracle SE doesn't offer Intermedia (we think this is right), and in which he agreed there was room for both approaches: db and web crawling, and in which he asked for my opinion of SWISH applicability to OpenACS 4, which I offered back in this thread.
- Found a link to a status report from one month ago status report suggesting that Swish++ (and htDig ) based approaches would be a good thing
And the content repository should be the hook for searching dynamic content regardless of where it's stored, DB or file system. There's no reason why static content couldn't be mapped via the content repository, too, now that it knows about stuff stored in the file system ...
I think I did my preliminary bit of research. No one suggested the problem may not be real. No one suggested the OpenFTS/PG weirdness has been fixed. No one mentions that OpenFTS has been found reliable and faster than the alternatives. And no one mentions just what search features OpenFTS actually supports.
I stand corrected now, and I understand that before asking any questions, or stating any concerns, or prior to any effort to participate with you, I must first read every thread and every document available in the system and be prepared to defend my understanding of this material. I must make private inquiries to all the developers and ask what their status is and what their thoughts are.
So in ten weeks of Baccusland, OpenFTS moves from being unknown code that munges PG in not pleasant ways to being savior of OpenACS 4 search. And I'm not supposed to voice any concerns.
Got it, I think I understand the process now.
Thank you for the hammering reset.
This is hardly a magical outcome. As you point out, I posted that the Russian search engine project needed to be evaluated and that I couldn't have any confidence in it until after such an evalutation was completed. If the person doing the evaluation had come back and said "it sucks", this potential solution would've been tossed into the toilet faster than you can say "Gulag Archipelego".
The major basis for my griping at you is the fact that you've made no effort to be part of the OpenACS 4.x team thus far. You're not asking
"what does the OpenACS 4.x project need? How can I help? What's on the critical path? Where can I be useful? Would helping out with the
searching problem be useful?"
Instead you're ... cowboying.
Check with our team members. I think you'll find that in each case I've asked folks "is there anything in particular you'd like to work on?" and have worked hard to assign folks tasks which fit their interests.
If you were to state that you'd like to be part of the OpenACS 4.x team, and that you'd like to help with search indexing solutions, hey, you might just get a positive response.
But I'm not hearing that you want to be part of the team. I think you see yourself as playing a somewhat different role. Perhaps I'm wrong, if I am tell me so.
The only role that's open, though - just so there is no misunderstanding - is that of "team player".
Anything different would be unfair to the folks who have been working hard in that role.
I'll be honest, I resent the energy that is being squandered here. Searching is no longer on our critical path. I'd really like to see our project remain focused on the primary goal: getting our first release out by the end of August. That's about seven weeks, I believe. That's not much time.
Do you really want to help the project where we most need help, or are
you just interested in working on pet projects that happen to strike your fancy at the moment?
If you really want to help the project where we most need help, our most critical need at the moment lies in testing. We have no infrastructure for testing and we need one. We need someone to champion testing, to grab as much as we can from aD (who's been working considerably harder on testing infrastructure issues in the last few months than they ever have before), to organize those volunteers who've offered to help, etc.
Now ... there's a chunk of work that would take a load off my mind. I've been planning to start tackling the issue next week, polling folks looking for someone who'll step up to the plate and take it on.
But since you seem eager to make a positive contribution to the OpenACS 4.x project, I might as well mention this actual need now rather than continue to butt heads with you over your inaccurate perception of what I need.
If you were to state that you'd like to be part of the OpenACS 4.x team, and that you'd like to help with search indexing solutions, hey, you might just get a positive responseDon,
I've discussed search strategies and technologies in these forums, have developed working demonstrations, and am working towards getting that code back to the community. I've asked the community if what I've done seems interesting. I've made several proposals regarding APIs and discussed the basis of their requirements and I've asked others to please comment. I've done more than offer to pitch in, I've been busy digging, and all in the open. I've emailed various developers and gotten lots of encouragement in return.
Folks can see my progress towards search, they can use my code to search this very system, and soon they can download it (I need to resolve a few AOLserver patch issues and get some time). Wish I could do the same with OpenFTS, eh?
I know several other folks here got the idea I was interested in working with the search issues. But I didn't email you directly and ask permission to work on some portion of the system? Jeez, is there some other way you could have told me that was a requirement than with this flaming?
Search isn't on the critical path? That's definitely not the impression I've gotten from listening to others in these forums. That's definitely not my impression from implementing the system nor from listening to my customers.
What you denigrate as cowboying, might be what others consider scratching an itch. I saw a problem, it was not at all clear that others were working on this problem, or that their efforts were going to converge in a timely fashion, I had some resources but not a whole lot, I saw how one solution towards it could satisfy a client's needs, and I've posted my comments and progress about it. That's what many of us call win-win. Makes the client happy, makes the family happy, and adds progress to an opensource project.
Isn't that the way opensource projects work? Keep your family happy first? Post patches when you can. Add modules when you can? Scratch the itch?
A few weeks ago the local community got together and built a new playground for the kids. Over the period of two months, about three hundred of us would show up when we could and we would literally start shoveling, cutting, hammering, sanding, or painting. Where we could, when we could, how we could. Yes, there was some professionals around at times to handle the backhoe, but the process wasn't "come by and I'll give you work", it was "grab a shovel or a hammer or a saw and attack what's interesting to you, or see where other people need help and join in."
What has my crime been? When I started my work in early June, I didn't know that the OpenFTS initiative existed and I didn't ask them to join. I suspect you disagree, but the OpenFTS project and OpenACS strategy is just not that well known. In ten weeks, it is explained in one thread, referenced once in two others, and mentioned with vague promises in the status report. I've posted about my own efforts many many times since then. I even wrote to Dan without knowing he was a participant in the OpenFTS project. Well, at no time did anyone from the OpenFTS project and certainly not you write me to let me know of their existence, much less invite me to pitch in with that effort.
And basically all your bluster aside, you're still just ranting at me, and your stupid claims that I want to be any other than be a team player just besmirches my reputation here. And all that just makes it very difficult for me to want to join any efforts involving you. No olive branch, not even a grunt, just more dick waving. I've been a cowboy? I haven't contributed? Screw you, you've been a shitty host and a shitty leader.
Anyone reading the status report regularly would be aware of the above facts. If the status report isn't clear to you, you are more than welcome to e-mail me directly for more details.
As far as your work with Swish and htdig, if you'd made any effort to coordinate your work with me, gee, maybe I wouldn't say you're working outside the project framework?
Do you need my permission to participate in the OpenACS 4.x project? Well ... you need my (or Ben's) permission to commit to our tree. 'Nuff said? I don't believe in anarchy, sorry. You can do whatever you want on your own, even start your own JerryACS.org project and take all our users and implementors away from openacs.org, but if you're going to be part of this project then, yes, you're going to have to recognize that this project has leaders who do make decisions.
Read your last post carefully. It is a perfect description of someone
working solo on issues that interest them without making any effort to
coordinate with either the OpenACS 3.2.x project or 4.x project managers. This is "cowboying", without doubt. Yes, you make your stuff available to others, you connect with individuals, some who are involved in the projects, some who aren't, but fundamentally you are working on your own and at best making an effort to interface with the community and with our code.
This is not the same as being a member of the project(s) team(s). You're not helping us pull our cart, you're pulling your own. That's fine, but, please, be straight about it.
I'm curious about the playground project. Was there a plan, or did you all just grab shovels and start shoveling randomly without any overall direction? I believe in plans, myself, and frankly imagine that this playground project had plans, coordinators, and provided direction to volunteers.
If your interest was in cutting all the handles off the project's shovels and burning the lumber used to build playground fixtures, would
the project leadership have suggested you do something different, or would they hand you a pile of shovels, a can of gasoline, a box of matches and encourage you to follow your muse, while holding you up as an example of a good team member?
Naw...I humbly suggest that they required that you volunteers work within the project framework.
And I ask for the same level of cooperation here.
Interestingly, a couple of times now in this thread I've suggested that there's needed work going begging and have asked if you might be interested in actually pitching in on stuff that is on the critical path. No response on your part. We do have volunteers who are part of this project who first and foremost want to see OpenACS 4.x roll out the door in a few weeks.
I like those volunteers a lot.
If people can't work with me, then I should step down as project manager. Actually ... if folks insist on an anarchist approach where the project manager is to be ignored, where people do what they please with no overall project planing, etc, well, I *will* step down and to be honest step away from the project entirely.
I'm just not interested in participating in a project where people hammer, saw, paint, and dig at random with no framework or direction.
That doesn't mean I need to be the person providing that direction, but by God this project needs direction if I'm going to be part of it.
From this discussion and from a previous discussion in which the issue
of a steering committee was raised, I get the impression that you, Jerry, would like more control over the project.
Or at least want the freedom to do whatever you please while claiming to be part of the project, I'm not entirely clear as to what you want.
I'm guessing you want either personal control or total anarchy, but I have no real notion as to which.
So, why not just tell us what you want, Jerry? Do you want to take over? Do you want us to adopt a free-for-all approach with no rules?
Is it just personal, is it just Ben and Don that need to go away? Just Don?
Exactly what do you want?
I have limited resources. I have to satisfy many masters. I am sorry that I cannot devote the resources you require in the direction that would suit you best. This is not because I don't want to help the project, but because I am a frail human and have to satisfy many masters. I regret you don't see it that way, and I regret you cannot find a way to appreciate the efforts devoted on your projects by any who come this way.
There's no response to your request that I volunteer because I am a human being, and you bad mouthed me in one sentence and then badgered me to volunteer in the next, and went further by suggesting that I work on something completely different, which is another way once more of your stating that my honest, honorable, respectful, cooperative, efforts to date are actually worthless. In such circumstances I find it very hard to turn around and ask of you, gosh Don, where is it that I can best help? I think that's too much to ask of most people, and I wonder if you realize that and are expecting no response from me so you can flog me once more, or, well, you need another course in the powers of persuasion. I am trying to improve in my ability to deal in such situations, but Master Po, I am still just grasshopper. What might you have done in your postings to make it easier for me?
When this was pointed out to you by others, instead of doing the honorable thing, and showing your respect for the community and me, and apologizing for your flames, and for your obstinacy, you simply presented your trump, and suggest I no longer play with your marbles.
I mentioned several posts ago that I respect your decision regarding OpenFTS and that I think your decision regarding OpenFTS is reasonable. I believe that our disagreement is on your staffing policies, your communication style, and your inability to bring others along, by letting them ask questions or make mistakes. It's one thing to say, thank you, but let's work in this direction, or let's not do this. It's another to say, why don't you apologize for your bad efforts and then ask us where you can use your help.
No one's calling for you or Ben or anyone to step down (why do people keep bringing up that strawman?) It might be nice to see a steering committee form, because one way you could increase the OpenACS reputation as a professional solution would be to don the cloaks of professionalism: formal structure, process, and respect for others.
I'm not sure what you're getting at with the playground building experience, but regretfully I note that once more you're suggesting my efforts are akin to burning the ACS down. Of course there were plans. And of course it was very well planned out and coordinated. A lot of energy obviously went to logistics and scheduling. But at the nuts and bolts level, it was self assembling by the community. It was point the community to the general problems and the offered solutions and trust in them to get the job down, and to take initiative, and to bring others up to speed, and to raise their concerns when they had them. And concerns were raised, and issues were discussed, and we were building a playground. We weren't building the pyramids. Wow, I just realized that was a whole year ago. And there weren't a couple of hundred of us, the estimates were that it was a couple of thousand of us, from kids to seniors, and to think, with no flaming either. Just how did those leaders pull that off? How did they know what required organization, what required invitation, what required a gentle push, and what required trust?
I believe that our disagreement is on your staffing policies, your communication style, and your inability to bring others along, by letting them ask questions or make mistakes.Good, you've actually made an honest statement as to what lies at the heart of this discussion. Why didn't you say so earlier? It would've saved all of us a whole bunch of time. I would've been a heck of a lot more polite if you'd cut to the chase right off.
I was starting to think I'd never be able to flush you out into the open.
If it is true that my staffing policy sucks, that my communication style sucks, that I am unable to bring folks along by letting them ask questions or by letting then fail, then I am, of course, absolutely unqualified to be running this project.
No one's calling for you or Ben or anyone to step down (why do people keep bringing up that strawman?)Because it's not a strawman. If your description of my job performance is accurate, I should be replaced. I'm only raising it because I don't feel like taking the time to lead you step-by-step to the logical conclusion that follows from your observation that I'm incompetent.
If the project team agrees with you, I'll be more than glad to step down. Leading this project is something I took on because I saw a void that needed filling. I look at it as a responsibility, not something I'm doing for fun and certainly not for profit.
So ... if your description of the job I'm doing is accurate, you'll have no problem getting the OpenACS 4.x project team to ask me to step down. I will do so willingly.
I'm ready! It's summer! I can spend it photographing birds!
You expect me to be unhappy or something?
Incidently, Jerry's posted a lot more info about what he's been doing vis a vis searching than N.P. or anyone else. I realize you were on vacation but I don't think there are grounds for criticizing Jerry for not reading status reports or trying to play with the team in that aspect. This is one of the topics I'm most interested in and to go by what's been publicly posted there was precious little 'official' work being done on this up to when Jerry started working on it. It's one thing to say, "Here's what N.P. has been doing," but it comes off wrong to flame Jerry for stepping on toes he had no way of knowing were there.
For what it's worth, I've learned a lot from both of you and hope to continue doing so.
And if the OpenACS 4.x project team (not observers outside the team) agree with Jerry's assessment, I will.
On the other hand, if they disagree, maybe Jerry will step back, think things through, and offer to jump in and help out in ways that will make my life easier, rather than harder. I would welcome that.
The silence of your supporters is deafening.
The problem with your analogy is that it's not my ball. The ball belongs to the project team, not to me. The proper analogy runs more like this - "I'll step down so the project team can play with their ball using whatever rules they want".
In your analogy, if I take my ball home, the game comes to an end. Certainly that's not the case here. If I step down, the game will be played, with the same ball, just with different rules, that's all.
Now, as the saying goes, the ball is in Jerry's court...
No one is asking you to step down. You are the only one that has mentioned this.But if Jerry's description is accurate, certainly we can find someone who will do a better job, right? I know that no one's asked me to step down.
I already know I'm the wrong guy to manage the release process, which is why I'm so glad to see Roberto doing it for 3.2.*, he'll be well-practiced at this art when I get on my hands and knees later and beg him to do it for 4.x, as well (assuming I'm still in the project manager role).
I don't think of myself as being a particularly good project manager. I don't think Jerry's assessment is entirely fair, but I don't find it hard to imagine someone doing a better job. On the other hand, I find it easy to imagine someone doing a worse job, I guess.
If someone else were managing it, I'd be freed up to do more coding, to finish cleaning up the APM, etc. Again, following the ballgame analogy, I'd just move to outfield rather than call balls and strikes behind the plate.
I guess the bottom line is that if I'm doing a bad job, I don't want the job, OK?
As far as being an ass to people who are "just trying to help", I do think Jerry wants to help, but I don't think that's a 100% accurate description of his position nor do I think this has much to do with this thread, frankly. I think he'd like to have a larger voice in the direction openacs takes, and I think he wants to help but according to his rules, rather than adapt to (say) the OpenACS 4.x framework others seemingly have been at least somewhat content to live with.
If I'm wrong, well ... then I'm wrong. It may be that it's just *me* that Jerry feels like he can't work with. He wouldn't be the first, and won't be the last. That's certainly one liability of having me in this role.
The beauty of having all this source code is that anyone who is complaining can take it, modify it, start their own project, keep it for themselves or almost anything they like.
If they don't like things the way they are, they can
- Host YOUR own server
- do YOUR own admin
- proselytize for YOUR own project
Leaders have to take strong positions sometimes. It's part of the job, and it's not an esay job. C'mon guys, put yourself in our shoes. It's hard to manage all this stuff. There are a lot of things going on and many more come out every day. Just keeping track with all the new things, ideas, propositions, that come up in a day is a daunting task.
What Don is trying to do is make us stay focused, so we can get 4.x out the door. Once the core is stable and release, we'll have a great base to create exciting new packages, refine the core, improve performance, add searching capabilites. IF we get a core released. If we take 1 year to release, we'll lose momentum, and that's no good.
Don (and me, and Ben, and Dan) are also trying to cultivate a team spirit here, and avoid fragmentation. It's not easy. Suggestion and constructive criticism is good, but we also should support our leaders if they are doing the best they can and showing work. I can't say Don's not doing all this.
I also see what Jerry is trying to do, and I like it. Searching is great, and important. But given everything else that's going on, I hope he (and others in the community) will understand the direction that we're heading here. I hope everyone can see that it's a good direction for everyone, including Jerry, and everyone else that wants to support their families through OpenACS work.
So why don't we put our differences apart, roll-up our sleeves, and focus on the core? Of course, those wanting to work on other things (e.g. searching) are free to do so, but please help us to get a good, stable, core out. Help us get more people to work on the core.
Another very important thing that I've been wanting to get jump-started is a documentation project for 4.x. It'd give others who are not familiar with the code, something to pitch in.
Let's keep it together folks, and make OpenACS kick some serious butt.
talli
First off, though, let me chine in and say that I appreciate the work and research Jerry has done and shared with the community (which, in my mind, is larger than just "OpenACS" but also extends to ACS, WTR, and AOLserver/OpenNSD), and I'd hate to see him ostracized--though my guess is that since his tussle with Kriston didn't cause him to leave, this won't either. ;^) I also appreciate Don's role in getting the OpenACS project off the ground with Ben and Roberto and Dan way back when, and for coordinating the porting effort now and I'm sure no one, Jerry included, wants him to step down.
The topic of this thread is "OpenACS 4 Search Integration, what should it look like?." If we change the title to "Future Search Integration, what should it look like?" (so as not to imply or leave it open for interpretation any attempt to expand the scope of the current 4.0 porting effort or sidetrack it), then perhaps we could have the type of conversation that Jerry was trying to start (this being a discussion forum, after all, where historically lots of people have shared ideas, even those who aren't actively involved in the porting of actual code--even people who don't program themselves). Basically pick up the thread from Ben's reply and continue on...
- Jerry writes Don, saying "I'm interested in providing htdig and swish search support in OpenACS 4.x. What do I need to do to take this on? I've already got it mostly working for 3.2.x so it is mostly a matter of integrating it with the 4.x site-wide search package"
- My response would've been simple: "I dropped that option because no one took it up, meanwhile the OpenFTS solution looks very viable. I've delegated OpenFTS integration to Neophytos and Dan, why don't you e-mail them to see if the three of you can come up with an approach that will let you work towards integrating swish and htdig, without delaying their efforts to integrate OpenFTS?"
- After putting their heads together, Neophytos, Dan and Jerry get back with a plan that will work within our strict timeframe, or come back and say "we can take some steps to ease future integration but there's no way we can support multiple options by the end of August"
- This serves as a basis for a discussion within the group at large, explaining the decision and explaining why the decision was made.
- If the decision turned out to be to go ahead and shoot for integration of external indexing tools as well as OpenFTS, then I would expect Jerry to accept responsibility for delivering on his end within our timeframe. Our timeframe's pretty loose, with no firm deadline (seeing as we're all volunteers) but, still, I'd only be willing to add additional tasks needing completion before first release if the person asking to include the task were willing to accept responsibility for following through.
In essence, this is all I mean when I ask that folks work within the project framework. I'd like folks to 1) work through me so I can make sure the appropriate team members are involved 2) work with those who already may own responsibility for the task or an overlapping task (in this case, responsibility for delivering a search solution has already been assigned and it seems reasonable to expect anyone proposing additional approaches to work with those responsible) and 3) be willing to accept responsibility for completion and delivery for whichever pieces of work they agree to take on.
That's just about it. This is a loose and relatively informal framework, one that others haven't found overly restrictive.
If Jerry's willing to work within this framework, I would *love* to see him jump in and work with Neophytos and Dan on ideas as to how best integrate various search technologies into the site-wide search package, if they're willing to take the time ot do so. I would leave the final decision up to them, though - if they felt it would be too distracting to work towards that goal during today's push to finish OpenFTS integration by our first release, I'd ask that Jerry live with that decision and I'd hope that he'd be willing to revisit the issue once integration was complete.
As far as whether or not I should be project manager, I mostly want to make it clear that I took on the role because the job needed doing, and no other candidate was popping up asking to take it on. Ben's too busy, Dan's never expressed an interest in taking it on (if he had, I would've done my best to stick him with it from the beginning!), etc.
I'm not doing it because I love power, want to enhance my resume, etc. If someone else were able to do as good or, more likely, a better job well ... that would free me up to spend more time digging into architectural issues, doing a more complete rewrite of the APM and cleaner multi-db support, etc etc etc.
So ... I'm not asking to be kicked out. Nor do I think I'm doing as bad a job as Jerry's comments would imply. But ... I'm not devoted to maintaining myself in the role as project manager, either.
What I am devoted to is doing what I can to get this project out the door in roughly seven weeks, if at all possible.
I apologize for losing my temper with Jerry, and take no offense as a result of him having lost his temper with me. Jerry's made significant contributions to AOLserver and I know he wants to see OpenACS be a great success. I think he knows I want OpenACS to be a great success, too. I think we have slightly different visions as to what that means, and perhaps as to how the OpenACS world ought to be structured, but I can live with that.
LaGuardia, where I caught the 3:30pm Delta shuttle to Boston. The
turbulence in flight and 45-minute runway delay should have alerted me
to the pending flamefest in OpenACS-land, but I didn't take notice. I
spent time with friends, and at 11pm, Dr. Wolfman himself interrupted
my brilliant game of pool: "So what's up with the community and this
'sex on friday' thing?" I chose to temporarily ignore my email and get
some sleep.
So, now that I am rested and have read through the volumes of
contributions, I have a few important, harshly honest points to make:
- Don has been doing an incredible job leading OpenACS 4.x. Much
better than I did on 3.x, and much better than many "professional"
project managers I've worked with. Doubts as to my support of Don's
approach in resolving OpenACS 4.x issues are wholly unfounded.
- Don's style is direct, but honest. I appreciate that, because I know
that if I slip up, I might get a nastygram from Don, and that keeps
me on my toes. In fact, it helps bypass the politics and get to the
meat of the issue. If the meat of the issue involves significantly
readjusting expectations, then we don't beat around the bush.
- Most successful open-source projects are led by a small set of
individuals with strong opinions who aren't afraid to put their foot
down. The "bazaar" model is fun marketing, but it's not exactly
correct (oh no! The OSI black helicopters are on their way...). This
means that sometimes people in the community will not be
satisfied. No matter what model you choose, that's going to be the
case. In this model, at least the project has a direction and vision
offered by people who have time and resources to actively
contribute.
- However, in order to perform efficiently, one must attempt to
decentralize the effort to a certain degree. Mostly, this is done by
pushing the overhead of communication and management as far away
from the center as possible. When I managed OpenACS 3.x, I
repeatedly ignored the first email I received from new potential
contributors. I would wait until the second reminder email, and then
add them to SourceForge and respond with instructions. At first, it
was because I was overwhelmed. Eventually, I realized that this was a
good process in general (gasp! I did it on purpose...) because it
filtered out those unwilling to put in the extra effort to retry,
adapt, and generally be pro-active about lowering the communication
overhead.
- Jerry, I responded positively to your initial posting because I
think you could be an excellent contributor to OpenACS
core. I like a lot of your ideas, and many here agree that the
quality of what you produce is quite high. However, I agree with Don
that you've not helped us on the communication overhead issue. Your
contributions are currently standalone, following your own
priorities, and any attempt you've made at integration with the team
involves a suggestion that the management process change. Sorry, too
much overhead. We'd love to get your contributions, but we can't
seem to find a way to do it without taking on huge amounts of
coordination work on our end. This is why it doesn't feel that
you're a team member, because it seems you're working on your own,
following your own track, and leaving us to figure out how the heck
to make use of what you've produced. If you can help us by
taking some of this work on, by finding ways to contribute without
requesting too much time from others, I think we'd begin to see the
light at the end of this flamefest tunnel.
- Please refrain from making statements of the "go have sex"
kind. I'm not about to censor anyone, but I would like to encourage
people to channel their frustration in more productive
ways.
- The reason people bring up the issue of "stepping down," frankly, is
that talk is cheap, and we are all volunteers with other interests,
too. I met Don through his bboard code. Don met me through some
code/posting interaction. The mutual respect we have is not based on
talk, it's based on actions of collaboration and true attempts to
work things out. I don't care how often I'm outvoted by Don, as long
as we discuss the issues at hand. The moment it seems that someone
is more interested in *their* point of view than in contributing to
the community, I tend to stop listening.
Thus, follow Roberto's recommendation. Take all that energy, that time
you have to post to the discussion forum, and find a way in which *you*
can contribute more or less within the current structure. Use the
gatekeepers as coordinators, but try not to impose tons of additional
management work because of some brilliant idea you have, even if
you're Einstein.
Okay. I'm taking another shuttle back to NYC in a few hours. It's
Friday. I hope everyone takes some time to enjoy themselves in
whatever manner they choose to.
Well, I was interested, until I heard how much the job paid :).
If I were a rich man ...
We need your help in integrating the OpenFTS in our website. we install all the supported things instead of it we didn’t able to get the expected basic steps to run the integration tool. can u guide me to fetch the exact result?
Please reply asap.
Thanks and regards,
magesh.
