Forum OpenACS Development: Enhanced KM in OpenACs 4.0 -- interface to OpenCyc

There is a lot of discussion of KM throughout these boards, and as we look to the future of OpenACS, I want to suggest using OpenCyc (also at sourceforge) to enhance the KM capabilities of OpenACS.

My project's needs extend from sense disambiguation and contextualization of query strings to automatic document classification with metadata extraction and generation to general knowledge discovery of the content of forum and other OpenACS databases, among others. The goals are enhanced user experience, improvements in CRM functionality, and increased "self awareness" of unstructured textual content in the application.

From what I know about OpenCyc, these goals (and more) should be readily achievable through the creation of an interface to the OpenCyc API.

I am not a programmer. If I were, I would take this on myself. I do want to interest a member of the OpenACS community (who is also a programmer) with interest in these areas to work with me to get started. OpenCyc would provide capability that was originally a part of Oracle's ConText package that has evolved into Intermedia, and is missing (from what I can glean) from the full text search working currently going on.

IMO, this sort of integration can rapidly expand the scope and usefulness of OpenACS and make it more competitive (assuming that is a goal of the development) with alternatives while fulfilling one of the basic promises (and premises) of Open Source tools.

Reply privately, if you'd like, but I'd like to see a public discussion about the usefulness of this approach and the uses to which a tool like OpenCyc can be put in the framework of OpenACS.

One major challenge in full-text searching is query
contextualization. Basically, when someone enters a general
search request for a topic like "Java," the search engine doesn't
know if you're interested in the programming language, coffee, or
an Indonesian island. OpenCyc (and WordNet, which is a part of
the package), can help that contextualization, in part by a)
engaging the requestor in a dialog to ask them which one they
are interested in, and/or b) chunking the return results into these
parent categories.

One way to use query contextualization and chunking is for the
preprocessor (OpenCyc) to formulate queries that will return a
separate results set for each sense of a word  or phrase, a form
of metasearch, if you will. The results are aggregated and
presented to the requestor using some presentation method
that supports i) serendipity and/or ii) helping the requestor
eliminate from consideration results which are obviously not of
interest. General terms return so many results (java returns 2+
million pages at Google while java+coffee returns 190k pages)
enabling users to say, "I am not interested in these things," is
incredibly useful.

Because OpenCyc "knows" about these aspects of language, it
can examine texts to determine what they are "about." In this way
OpenCyc can be used extract meaning from a document/db
record (e-mail, web page, forum posting, etc.) and automatically
generate metadata that can be inserted in a db in a form that is
optimized for querying or query assistance. In one use I have,
OpenCyc would scan the contents of e-mails directed to
customer support and route them based on content.

OpenCyc can be used for general knowledge discovery in a
e-doc collection, and it can be retrospective in addition to current.
That is, I can run OpenCyc against an e-doc collection and let it
generate metadata about the collection, which metadata gets
indexed. I can imagine having a historical reporting function that
included trending to let me know the change in frequency of
occurrence of certain topics. Important in a CRM context.

Among other things.

<p>It's very easy to get pie-in-the-sky with these sorts of things
so I'm going to list only those ideas that I'm guessing wouldn't
be outrageously difficult to implement (assuming OpenCyc
works as advertized).</p>

<ol>
<li>Enhance the OpenACS search results page by grouping the
results into folders, similar to the way NorthernLight.com does.
(This might actually be a good pilot project for
OpenACS/OpenCyc integration.)</li>

<li>Shovel a huge amount of pre-existing content (Word files,
PDFs, static web pages, etc.) into an OpenACS system and have
OpenCyc generate one or more category trees for it. Then
generate a Yahoo!-style content directory. This would be
enornously useful for medium to large-sized organizations that
generate a lot of content that doesn't get re-used because
people who don't know it exists already have no way of finding
it.</li>

<li>Shovel all of the content (bboard posts, documents in
file-storage, SDM entries, etc.) created by one person into
OpenCyc and produce a ranked list of categories in which this
person seems interested. Show that ranked list, along with links
to the content items in each category, on the person's profile
page. (This could be hidden at the user's option.) So, for
example, you could see that Jane Doe has posted a lot of stuff
about caching to the OpenACS.org site, and you could
immediately drill down to those particular posts to see if she's
the expert you're looking for. Alternatively, you could allow
somebody to search the directory based on keywords to find all
the people who have posted stuff about caching, in rank
order.</li>

<li>Building on the idea above, whenever the person's interest
level score passes a threshold on a particular topic, have the
system ask the person if s/he wants to receive email alerts when
new content items within that category are entered into the
system.</li>
</ol>

<p>Yeah. This could be interesting.</p>

After that's done, we'll need to wait for OpenCyc to be released,
which will hopefully be in the next two weeks. Then somebody
needs to whip together a prototype to show off the potential. Clay,
is that something you're up to trying?

I suspect we'll get more potential volunteers once (a) the initial
4.0 port is finished and tested, and (b) people have a clearer
understanding of what we're talking about here. It still seems a
little abstract.

Implementation can wait until the next beta release of OpenCYC and the 4.0 port is complete/stable. I'd like to begin the preliminary design work so that, perhaps, we may be done or nearly so when the above two conditions are met.

Clay, I'm not a programmer, so I can't do the integration for you
(though I would if I could). It'll be especially important to outline
some concrete use cases so that those who can do the work
see a reason for putting in the time. Likewise, there are some of
us here who, while we can't do the work ourselves, might be
able to get funding for it if we could make a compelling and
concrete business case for it.

So let's keep talking about good pilot applications, let's draw up
some use cases, and let's mock up some HTML wire frames.

Michael: By all means lets put together some uses cases and do some wireframes. Where would you like to start?

Let's start with your first two ideas:

It seems to me that these two are the simplest to think about first, since they need only to interface with search and not any other part of OpenACS. I'm not really clear on how hard (3) would be or how necessary it is for achieving (1) and (2), and I'm fairly certain that (4) would require more work.

OK, so with that in mind, how would you envision "query optimization" to work? What's it like from the user's perspective? Are there examples of existing pre-search dialogs you can point to as similar?

Likewise, how do you picture the "chunking" of results into "contexts"? How is that like or unlike search results pages that are out there now? For example, I'm not really clear yet on how, from a user's perspective "a collection of phrases and keywords grouped into the smallest possible number of headings" differs from Yahoo! or from Northern Light.

BTW, according to the latest Sourceforge update, it looks like we will have plenty of time to think this through:

Although many of the constants with incomplete minimal definitions do have many other assertions in the Cyc knowledge base, the minimal definitions are a prerequisite for any release of OpenCyc. This should not affect the release date of OpenCyc 1.0 (currently slated for delivery before the end of 2001), but it will delay the open beta release of OpenCyc 0.9B for several months. New dates for OpenCyc 0.9B will be provided as they become available.

OK, this helps quite a bit.

The query optimiztion requires taking the search request and passing it through OpenCyc to try to figure out the context of the request. Java is a good example: does the requestor want to know about the programming language, coffee, or the island in Indonesia.

If I understand correctly, OpenCyc implemented on an OpenACS site could, for example, recognize that a content item is likely to be about Java even if the term "Java" isn't used in it because it knows that the words "Swing," "classes," "beans," and "introspection" are all terms that relate to Java the language. It should even be able to guess that an OpenACS.org page that uses the word "Swing" is probably relevant to somebody looking for information about user interfaces.

Now, OpenCyc could accomplish this goal simply by presenting the search results in three categories: programming language, coffee, and geographic location. To do this much, you would just have to pass the search results through OpenCyc and display by category. (Right so far?)

It would be more convenient, though, if OpenCyc knew that people searching the OpenACS site are extremely unlikely to be looking for information about coffee or geographic location. This information would reduce the number of user clicks required to get the info they want and also reduce the number of false positives. That, by itself, doesn't sound like a huge gain on a site the size and focus of OpenACS.org, although it could be more significant in a large corporate intranet.

However, there is more here:

So, if I'm getting this right, OpenCyc could learn to be more useful by being able to access the keywords in the content repository. It could, for example, start to recognize that certain proc names are associated with, say, the request processor. So Cyc could tell the search engine to look for those proc names as additional search terms when a user enters "request processor" as the term. Alternatively, if the proc is used in several different applications, it could group search results by application. (I have no idea if this particular example would be useful to anybody; I'm just making it up.)

Category organizers, in my way of thinking, should spend as much time thinking about metadata as they do about the category structure. For example, I have a personal interest in chocolate. If I was responsible for the chocolate category at a site that incorporated both categories and a page-based search engine, I would want to create a thesaurus of concepts that describe chocolate to aid a classification engine. With OpenCyc, I would use the OpenCyc tools to do this work using assertions like:

I had a similar thought to this a while back:

I was originally thinking about relationships between content items only, but you could do it with relationships to categories or keywords as well. How much would you have to change in the content repository data model in order to do this?

Regards..

A couple of replies rolled into one post:

Talli:
So as you're working out your use cases, it might be nice to work one out that somehow defines how OpenCyc would benefit a CRM system. Or actually to write a use case for a CRM system would be great too. A $6.5 billion dollar industry could always use an open source solution

Improving the CRM capabilities of OpenACS is definitely one of the uses of OpenCyc that I see and I have been discussing this off-line with a couple of people. One simple use is auto-classification of trouble tickets with a business rules engine for routing. This would require training the system by describing a vocabulary, but there are tools in OpenCyc to do that. The rules are a separate issue, though. As always, in the OpenACS context, is how to create a framework for customization with the correct level of integration with all of the existing modules in the system that could take advantage of such capabilities.

Roger:
Here is another article about Cyc and its descendant OpenCyc. This article places its creator somewhere other than the top of the AI field. In particular, it questions the methodology of his approach.

I am aware of the criticisms of Lenat's work and approach, and, frankly, I don't know that they apply here because I am not interested in a general solution to the AI problem, which as I see it, can be thought of as how to apply "what I know" to situations that don't fit "what I already know." There is a general learning problem which may have to do with a combination of crystalline intelligence (which is based on experience) and fluid intelligence (which is a set of rules that help you apply crystalline intelligence in unfamiliar contexts). I am a big fan of what I call "artificial artificial intelligence." I don't really care if it's "true" AI -- what I want is something that behaves intelligently, which in this case means that it "knows" when it "doesn't know what it doesn't know" and organizes things the best it can so that a person can make the decision.

Michael:
I was originally thinking about relationships between content items only, but you could do it with relationships to categories or keywords as well. How much would you have to change in the content repository data model in order to do this?

I have no clue, which is why I hope that someone with a deeper technical understanding of the OpenACS data model will jump in here and elucidate the magnitude of effort involved in getting something like this started. I know that this is something I want to do for my project, I just want to make entirely sure that I am not doing something that won't benefit the community at large.

Code is available here:

http://sourceforge.net/project/showfiles.php?group_id=27274

Documentation is available here:

http://www.opencyc.org/doc