It's no revelation that many sites already use other search engines
(htDig, SWISH, mnogo, even Microsoft Context Server (is that the
name?) and it would be beneficial (and even synergistic) to be able
to integrate the OpenACS into those sites, letting them keep their
same search engine technology.  Ideally, I would like the OpenACS to
come, out of the box, with a search engine that works for any OpenACS
supported platform: *nix or Windows, Oracle or Postgres or ...?  Then
a developer would have a working solution but would also have the
ability to change out/swap/tune that solution with other appropriate
technologies.

Here's a partial list of search engines that have been proposed for
OpenACS sites.

a) Intermedia
b) OpenFTS
c) Lucene
d) htDig
e) SWISH<fork>
f) mnoGoSearch
g) ???

What a panoply of technology!  We have C, C++, Perl, and Java.  We
have GPL and not.  There are engines based on site crawling, engines
based on db crawling, and those that would do both.  Some of these
engines are specific to a given database (Intermedia/Oracle, and
OpenFTS/Postgres).

My proposal is to see a "search engine driver" based approach to
search engine integration.  This would consist of two sets of
interfaces that each site indexer would have to support, with
the "promise" that once the support is present, the site indexer
would plug right into the ACS.

On the frontend, a search engine would support an aolserver module
(specifically an aolserver db driver like interface) that exposes
methods to open and close an index, to specify a search, to retrieve
results, and to allow a limited amount of introspection (is this type
of document handled, are excerpts returned, etc.)  This would
let /search pages handle all sorts of different search engines
generically in the same manner that ns_db or db_ lets us deal with
many different types of RDBMS.

On the backend, a module would support some sort of "search me"
callback interface similar perhaps to the "ACS new stuff" interface,
or similar to the ACS Intermedia interface.  As glue then, support of
a site indexer would consist of code that runs through the "search
me" callbacks or scans the ACS Intermedia table returning the various
documents to the site indexer, and perhaps ACS-wide parsers that
might help an indexer understand ACS objects when encountering them
as embedded links within a document (users, bboards and bboard
topics).

With a bit of experience, I would prefer a callback scheme to an ACS
classic/Intermedia table based scheme.  There is a conflict within
the ACS today: put keep all the content in the db vs. allow some of
the content to reside in the file system.  A callback scheme lets the
module determine where the content lies.  A callback scheme would let
the site indexer index remote content (via a callback that supports
XML-RPC or SOAP, or any sort of webservice).  A table based scheme
may cause the data to be replicated one more time: it's somewhere in
db or file space in it's primary format, it's replicated in indexes
within the site indexer, and we then ask it to be stored one more
time within the table to be scanned.

What do you folks think?  Is this an OpenACS requirement,
distraction, or bloat?  What should these interfaces look like?
Callbacks or table driven?

It's called the Content Repository and the reason I keep hammering home the point that all content should be stuffed there is precisely so any indexing facility will have only have to talk to the ACS in one
place.

Thus the supposed conflict you mention no longer exists.  Making it disappear is the *point* of rewriting the CR to allow content to optionally be stored in the filesystem.

It may be that the CR API isn't quite up to the task of handling content stored in arbitrary locations within a server's file system, but if that's true we'll just need to modify the CR API to make it so.

Seamlessly dealing with remote content is more difficult, but I'll personally be very, very happy to see the lesser problem of providing decent search functionality for PG-based OpenACS sites solved before I worry too much about this additional issue.

And of course we'll need to keep the CR as lightweight as possible when you're not dealing with content in the db.  Since any scheme to track filesystem content will want to be hooked into ACS permissions, which implies that at some level the scheme will be built on top of ACS Objects, the reality is that the CR is about as lightweight as any
such scheme will be.

Unless we decide we want a parallel and separate package to deal exclusively with indexing issues, which I'll lobby long and hard against.

If someone wants to sit down and write alternative indexers that operate on the CR, that could be substituted for InterMedia or the OpenFTS solution, sure, they can go for it.

At the moment, though, I consider this to be more distraction than useful.  I don't look at the search problem as being a critical path item for the current incarnation of the OpenACS 4.x project.  We have an Oracle solution and we have what appears to be a very workable Postgres solution.  Yes, we hope to support other RDBMS systems in the future and I'd love to see additional search/indexing options available in the future, too.

But ... there's a bunch of "right now" issues that are more important,
IMO.  If you have time on your hands, Jerry, why not pick up one of the unclaimed OpenACS 4.x packages and port it?

The code base has also been used to index the 160,000 posts in the various postgres mailing lists, and it seems to be working well there, too.

Neophytos has posted this information in the past, and the current status.html file makes mention of it and includes and invitation to e-mail him directly if you have further questions.

Since you do, why don't you?

As far as the rest of your rant goes, if you're tired of Tcl, like Apache and Tomcat, and are primarily concerned with being able to sell
OpenACS into existing organizations who are more interested in buzzword compliance than in capabilities, well, then I think ACS 5 will better fit your needs than OpenACS 4.x.

Besides which I'm unconvinced that either swish or htdig have the buzzword compliance sheen necessary to make them an easy sell into these organizations.

If you really want a swish solution to be available in OpenACS - do it!  Build it on top of the CR and it can be offered as an alternative
search strategy.  As I said in my previous note, I'd love to see alternatives available.

But you're not really suggesting an alternative.  You're arguing that
we not pursue the workable solution we already have, and that ain't going to happen.

Mostly you're arguing that we do things your way, though, aren't you?

That argument would carry a lot more weight if you were actually contributing to the project.  The OpenFTS solution has floated to the
top of the list because Neophytos (with some urging and advice from me)  picked it up and ran with it, including helping the Russian company that paid for the implementation figure out licensing issues.

Swish was on my original status sheet as an option, but no one picked it up and ran with it.  So I took it off.  If you want to see it implemented ... implement it!

Actually I didn't set up any straw man at all, which m-w tells me is "a weak or imaginary opposition (as an argument or adversary) set up only to be easily confuted."

What I did is often more properly labeled as "stating my concerns."  I didn't answer them, and I asked others with more experience and vision to answer them and clarify the situation.

I believe you may have set up a straw man when you misstated my post and my intent as asking for us to not pursue a workable solution and then flamed me a bit more for not offering any alternatives.  (And then you got a bit personal and suggested I mostly just wanted folks to do things my way.  I honestly don't know how that could be, since as I started off this thread by saying, I haven't yet met a search engine I like, and gosh, I'd really like to talk about that.  You also suggested that OpenACS wasn't for me (and presumably I wasn't for it and maybe I should move along.)  Yeah, that kind of hurt, and so I thank everyone who came along to my defense.

So just to repeat: I never said we shouldn't pursue OpenFTS, in fact I said I would probably be a user of OpenFTS.  I said I was concerned with OpenFTS's maturity at this stage of the game and I said I was concerned that an OpenFTS focus now might lead to an API that was db crawler focused.  What I didn't make clear was that by having "n" modules implement that API would then lead to enough inertia in the system as to preclude a web crawler based solution later.  I offered the alternative that I started this thread with: a discussion on what search engine requirements are and what search engine interfaces should look like.  And then I explicitly asked where my assumptions and concerns were misplaced.

I didn't expect the Spanish Inquisition (does anyone?) (that by the way is homage to Monty Python AND Philip K. Dick.)

Now perhaps you haven't been following the forums lately, but in fact, I have been working on SWISH<fork> and htDig integration with OpenACS 3.2.5 to solve some client problems and to learn enough to redo it in the OpenACS 4 framework.  I apologize if my timeline wasn't your timeline, but the fact is, I have been trying to run with it.

It appears your response is: because of the content repository's central role, you don't believe that a solution developed first for OpenFTS and Intermedia will preclude other technologies; that your personal experience with OpenFTS to date suggests to you that it is stable and capable of the job; and that the OpenACS 4 release is near enough that you believe now isn't the time to reopen this issue.

Is that reasonably accurate?  Anyway, if so, that's a justifiable response for the project lead to make, I hope you're right, and I'm behind you.  But you sure pissed all over this thread, and I for one, don't feel particularly welcome here or at all eager to figure out just what I might have to do in the future just to get you to state that I have contributed to the project.

Well, if you'd done just a little bit of research you would've known that the CR was rewritten to allow both filesystem and db storage of content, so there's no need to propose a solution to this already-solved problem.

• participate in a lengthy discussion evaluating various openacs search strategies from htDig to SWISH, to something I now recognize as OpenFTS: here's your confident support of it just two months ago:

  possibly synergistic efforts include an in-database search engine being developed by folks who run a Russian portal. If someone's interested in tracking this down and evaluating it both for usability and completeness (i.e. are they done yet?) e-mail me. I know that to make it work very well on their portal site they had to make a change to the PG core's optimization and evaluation of limit that Tom Lane didn't like all that well.

• implement htDig and SWISH on OpenACS 3.2.5
• sets up an htDig based search for OpenACS.ORG and OpenNSD.org that lets users use boolean, wildcard, fuzzy, mispeling correction, stemming, date ranges, while also letting the user specify which portions of the site to search. This demo lets folks index their word, excel, and pdf files. (This seemed like a damn fine search to start with, better than offered at /search, but I now understand how unsophisticated this search actually is.)
• searched OpenACS's /bboards, /doc, and /wp for the word OpenFTS
• read BOTH threads that contain the word OpenFTS in them (prior to this thread)
• tried to discover (but failed) where the OpenFTS discussion came from (the main OpenFTS thread starts off by saying this is an update)
• had a brief email discussion with Dan in which he seemed to agree that OpenFTS was PG centric, that Oracle SE doesn't offer Intermedia (we think this is right), and in which he agreed there was room for both approaches: db and web crawling, and in which he asked for my opinion of SWISH applicability to OpenACS 4, which I offered back in this thread.
• Found a link to a status report from one month ago status report suggesting that Swish++ (and htDig ) based approaches would be a good thing
In none of these discussions nor the status reports, nor in the private email with Dan was there any indication that the CR should be the sole target of the search nor that that would solve the problem away for both db and web based crawlers. And I apologize for being too obtuse to figure all that out from this one paragraph of yours: Full Text Search

And the content repository should be the hook for searching dynamic content regardless of where it's stored, DB or filesystem. There's no reason why static content couldn't be mapped via the content repository, too, now that it knows about stuff stored in the filesystem ...

I think I did my preliminary bit of research. No one suggested the problem may not be real. No one suggested the OpenFTS/PG weirdness has been fixed. No one mentions that OpenFTS has been found reliable and faster than the alternatives. And no one mentions just what search features OpenFTS actually supports.

I stand corrected now, and I understand that before asking any questions, or stating any concerns, or prior to any effort to participate with you, I must first read every thread and every document available in the system and be prepared to defend my understanding of this material. I must make private inquiries to all the developers and ask what their status is and what their thoughts are.

So in ten weeks of Baccusland, OpenFTS moves from being unknown code that munges PG in not pleasant ways to being savior of OpenACS 4 search. And I'm not supposed to voice any concerns.

Got it, I think I understand the process now.

Thank you for the hammering reset.

If you were to state that you'd like to be part of the OpenACS 4.x team, and that you'd like to help with search indexing solutions, hey, you might just get a positive response

I've discussed search strategies and technologies in these forums, have developed working demonstrations, and am working towards getting that code back to the community. I've asked the community if what I've done seems interesting. I've made several proposals regarding APIs and discussed the basis of their requirements and I've asked others to please comment. I've done more than offer to pitch in, I've been busy digging, and all in the open. I've emailed various developers and gotten lots of encouragement in return.

Folks can see my progress towards search, they can use my code to search this very system, and soon they can download it (I need to resolve a few AOLserver patch issues and get some time). Wish I could do the same with OpenFTS, eh?

I know several other folks here got the idea I was interested in working with the search issues. But I didn't email you directly and ask permission to work on some portion of the system? Jeez, is there some other way you could have told me that was a requirement than with this flaming?

Search isn't on the critical path? That's definitely not the impression I've gotten from listening to others in these forums. That's definitely not my impression from implementing the system nor from listening to my customers.

What you denigrate as cowboying, might be what others consider scratching an itch. I saw a problem, it was not at all clear that others were working on this problem, or that their efforts were going to converge in a timely fashion, I had some resources but not a whole lot, I saw how one solution towards it could satisfy a client's needs, and I've posted my comments and progress about it. That's what many of us call win-win. Makes the client happy, makes the family happy, and adds progress to an opensource project.

Isn't that the way opensource projects work? Keep your family happy first? Post patches when you can. Add modules when you can? Scratch the itch?

A few weeks ago the local community got together and built a new playground for the kids. Over the period of two months, about three hundred of us would show up when we could and we would literally start shoveling, cutting, hammering, sanding, or painting. Where we could, when we could, how we could. Yes, there was some professionals around at times to handle the backhoe, but the process wasn't "come by and I'll give you work", it was "grab a shovel or a hammer or a saw and attack what's interesting to you, or see where other people need help and join in."

What has my crime been? When I started my work in early June, I didn't know that the OpenFTS initiative existed and I didn't ask them to join. I suspect you disagree, but the OpenFTS project and OpenACS strategy is just not that well known. In ten weeks, it is explained in one thread, referenced once in two others, and mentioned with vague promises in the status report. I've posted about my own efforts many many times since then. I even wrote to Dan without knowing he was a participant in the OpenFTS project. Well, at no time did anyone from the OpenFTS project and certainly not you write me to let me know of their existence, much less invite me to pitch in with that effort.

And basically all your bluster aside, you're still just ranting at me, and your stupid claims that I want to be any other than be a team player just besmirches my reputation here. And all that just makes it very difficult for me to want to join any efforts involving you. No olive branch, not even a grunt, just more dick waving. I've been a cowboy? I haven't contributed? Screw you, you've been a shitty host and a shitty leader.

I have limited resources.  I have to satisfy many masters.  I am sorry that I cannot devote the resources you require in the direction that would suit you best.  This is not because I don't want to help the project, but because I am a frail human and have to satisfy many masters.  I regret you don't see it that way, and I regret you cannot find a way to appreciate the efforts devoted on your projects by any who come this way.

There's no response to your request that I volunteer because I am a human being, and you bad mouthed me in one sentence and then badgered me to volunteer in the next, and went further by suggesting that I work on something completely different, which is another way once more of your stating that my honest, honorable, respectful, cooperative, efforts to date are actually worthless.  In such circumstances I find it very hard to turn around and ask of you, gosh Don, where is it that I can best help?  I think that's too much to ask of most people, and I wonder if you realize that and are expecting no response from me so you can flog me once more, or, well, you need another course in the powers of persuasion.  I am trying to improve in my ability to deal in such situations, but Master Po, I am still just grasshopper.  What might you have done in your postings to make it easier for me?

When this was pointed out to you by others, instead of doing the honorable thing, and showing your respect for the community and me, and apologizing for your flames, and for your obstinacy, you simply presented your trump, and suggest I no longer play with your marbles.

I mentioned several posts ago that I respect your decision regarding OpenFTS and that I think your decision regarding OpenFTS is reasonable.  I believe that our disagreement is on your staffing policies, your communication style, and your inability to bring others along, by letting them ask questions or make mistakes.  It's one thing to say, thank you, but let's work in this direction, or let's not do this.  It's another to say, why don't you apologize for your bad efforts and then ask us where you can use your help.

No one's calling for you or Ben or anyone to step down (why do people keep bringing up that strawman?)  It might be nice to see a steering committee form, because one way you could increase the OpenACS reputation as a professional solution would be to don the cloaks of professionalism: formal structure, process, and respect for others.

I'm not sure what you're getting at with the playground building experience, but regretfully I note that once more you're suggesting my efforts are akin to burning the ACS down.  Of course there were plans.  And of course it was very well planned out and coordinated.  A lot of energy obviously went to logistics and scheduling.  But at the nuts and bolts level, it was self assembling by the community.  It was point the community to the general problems and the offered solutions and trust in them to get the job down, and to take initiative, and to bring others up to speed, and to raise their concerns when they had them.  And concerns were raised, and issues were discussed, and we were building a playground.  We weren't building the pyramids.  Wow, I just realized that was a whole year ago.  And there weren't a couple of hundred of us, the estimates were that it was a couple of thousand of us, from kids to seniors, and to think, with no flaming either. Just how did those leaders pull that off?  How did they know what required organization, what required invitation, what required a gentle push, and what required trust?

Incidently, Jerry's posted a lot more info about what he's been doing vis a vis searching than N.P. or anyone else.  I realize you were on vacation but I don't think there are grounds for criticizing Jerry for not reading status reports or trying to play with the team in that aspect.  This is one of the topics I'm most interested in and to go by what's been publicly posted there was precious little 'official' work being done on this up to when Jerry started working on it.  It's one thing to say, "Here's what N.P. has been doing," but it comes off wrong to flame Jerry for stepping on toes he had no way of knowing were there.

For what it's worth, I've learned a lot from both of you and hope to continue doing so.

The silence of your supporters is deafening.

The beauty of having all this source code is that anyone who is complaining can take it, modify it, start their own project, keep it for themselves or almost anything they like.

If they don't like things the way they are, they can

• Host YOUR own server
• do YOUR own admin
• proselytize for YOUR own project
With all due respect to all involved, are you guys nuts? The team is fine.

talli

• Jerry writes Don, saying "I'm interested in providing htdig and swish search support in OpenACS 4.x. What do I need to do to take this on? I've already got it mostly working for 3.2.x so it is mostly a matter of integrating it with the 4.x site-wide search package"
• My response would've been simple: "I dropped that option because no one took it up, meanwhile the OpenFTS solution looks very viable. I've delegated OpenFTS integration to Neophytos and Dan, why don't you e-mail them to see if the three of you can come up with an approach that will let you work towards integrating swish and htdig, without delaying their efforts to integrate OpenFTS?"
• After putting their heads together, Neophytos, Dan and Jerry get back with a plan that will work within our strict timeframe, or come back and say "we can take some steps to ease future integration but there's no way we can support multiple options by the end of August"
• This serves as a basis for a discussion within the group at large, explaining the decision and explaining why the decision was made.
• If the decision turned out to be to go ahead and shoot for integration of external indexing tools as well as OpenFTS, then I would expect Jerry to accept responsibility for delivering on his end within our timeframe. Our timeframe's pretty loose, with no firm deadline (seeing as we're all volunteers) but, still, I'd only be willing to add additional tasks needing completion before first release if the person asking to include the task were willing to accept responsibility for following through.

In essence, this is all I mean when I ask that folks work within the project framework. I'd like folks to 1) work through me so I can make sure the appropriate team members are involved 2) work with those who already may own responsibility for the task or an overlapping task (in this case, responsibility for delivering a search solution has already been assigned and it seems reasonable to expect anyone proposing additional approaches to work with those responsible) and 3) be willing to accept responsibility for completion and delivery for whichever pieces of work they agree to take on.

That's just about it. This is a loose and relatively informal framework, one that others haven't found overly restrictive.

If Jerry's willing to work within this framework, I would *love* to see him jump in and work with Neophytos and Dan on ideas as to how best integrate various search technologies into the site-wide search package, if they're willing to take the time ot do so. I would leave the final decision up to them, though - if they felt it would be too distracting to work towards that goal during today's push to finish OpenFTS integration by our first release, I'd ask that Jerry live with that decision and I'd hope that he'd be willing to revisit the issue once integration was complete.

As far as whether or not I should be project manager, I mostly want to make it clear that I took on the role because the job needed doing, and no other candidate was popping up asking to take it on. Ben's too busy, Dan's never expressed an interest in taking it on (if he had, I would've done my best to stick him with it from the beginning!), etc.

I'm not doing it because I love power, want to enhance my resume, etc. If someone else were able to do as good or, more likely, a better job well ... that would free me up to spend more time digging into architectural issues, doing a more complete rewrite of the APM and cleaner multi-db support, etc etc etc.

So ... I'm not asking to be kicked out. Nor do I think I'm doing as bad a job as Jerry's comments would imply. But ... I'm not devoted to maintaining myself in the role as project manager, either.

What I am devoted to is doing what I can to get this project out the door in roughly seven weeks, if at all possible.

I apologize for losing my temper with Jerry, and take no offense as a result of him having lost his temper with me. Jerry's made significant contributions to AOLserver and I know he wants to see OpenACS be a great success. I think he knows I want OpenACS to be a great success, too. I think we have slightly different visions as to what that means, and perhaps as to how the OpenACS world ought to be structured, but I can live with that.

Well, I was interested, until I heard how much the job paid :).