Forum OpenACS Development: Trying the search package

Posted by Neophytos Demetriou on
First, let me apologize for the lack of documentation. Hopefully, we
will be able to provide some documentation before the final release.


As you might already know, the following packages are now available
from CVS:

<li> ACS Service Contract
<li> Search
<li> OpenFTS Driver

(1) and (2) are core packages. To install the openfts-driver package
follow the instructions in the <code>doc/INSTALL</code> file. When you
are done visit the ACS Service Contract package link from the
top-level page. You will see something like this:

873 FtsEngineDriver, 2376 openfts-driver [Install]

Click on the Install link. <code>FtsEngineDriver</code> is a contract
(an abstract set of operations like index,search,etc).
<code>openfts-driver</code> is an implementation of that contract
(functions -- concrete implementations of the operations). So, if
somebody wants to use a different search engine (htdig, swish, etc)
all s/he has to do is write a new implementation of the
<code>FtsEngineDriver</code> contract. <small><b>[Note: If you plan to
do so,
please contact me so that I can provide some guidelines for
consistency among the implementations of a contract]</b></small>


Next, download the enhanced notes package from here:
This package implements/supports the <code>FtsContentProvider</code>
contract which
includes two operations, namely datasource and url. The former returns
data for the given object_id and the latter returns a url for the
given object_id. Now, visit the ACS Service Contract link from the
top-level page and install the binding that looks like this:

892 FtsContentProvider, 2408 note [Install]

Again, if you want your package to support indexing/searching
capabilities, you will have to implement
<code>FtsContentProvider</code>. We are already working to provide
these implementations for the packages that are currently available
(i.e. CVS) and we will also post some guidelines about this very soon.


Finally, mount the <code>search</code> and <code>note</code> packages
and restart your server. Try adding some notes. We have configured the
search package to check a queue of content-objects that need indexing
every 30 seconds (change this setting from the search package
parameters). Then, visit the search page and submit some queries. Here
are some features of the openfts-driver you can enjoy:

<li> No full re-indexing: A document is indexed only once. You don't
have to rebuilt the index for all the objects every time (intermedia).
<li> Proximity: Ranking results w.r.t how close the keywords were
found in a document. So if you have a document which include the
phrase "full text search" and another document which includes the
words "full", "text", "search", the first one will be ranked higher.
<li> Stemming: It allows to find same words with different
endings. For example, if words "testing" or "tests" are found in a
document, the word "test" will be stored by indexer instead. Search
will also try to find the word "test" if "testing" or "tests" is given
in search query. Note that this scheme lacks exact search possibility,
but usually reduces database size and makes search faster
<li> Different weights of words in the title and body. This affects
ranking of the search results
<li> Much, much more, ...

If there are any questions, please post to the forums or email me
directly (<a
href="";></a> , <a
href="";></a>). I will try be as
responsive as possible. Soon, I will post a
list of todo items that are related to the acs-service-contract
package and how you can help.

Posted by Don Baccus on
Neophytos has now committed his changes to the notes package to the CVS tree, so there's no need to grab it from new-file-storage.  Just update your local copy and you'll get it.
Posted by Ola Hansson on
How or where can I get the OpenFTS driver? I was able to checkout the notes package but not "openfts-driver":
[nsadmin@infogettable packages]$ cvs checkout openfts-driver
cvs server: cannot find module `openfts-driver' - ignored
cvs [checkout aborted]: cannot expand modules
Posted by Jonathan Marsden on
Just to confirm, the install is really smooth for the OpenACS parts of this; I just updated from CVS and reinstalled from scratch using my little script oacs4-startover (!), and it all came up fine.  In installed the search and note packages, and told acs-service-contract to "install" (strange choice of word here?) the bindings, and mounted the search and note packages.  All that "just works".

The creation of the ns_fts module for AOLserver is a little more involved, of course, since it involves recompiling C source code and assumes you have the AOLserver source code around too, and Tcl installed on your machine.  I'm playing with adding ns_fts to my AOLserver RPMs to make that part of the install easy also... an aolserver-nsfts RPM may well emerge.

BTW, the supplied is not very Red Hat friendly...
a patch will be available as soon as I am confident it works :-)

There is also some sort of issue where Search-OpenFTS-tcl-0.1 uses the Tcl installation on the machine, and apparently assumes the version used within AOLserver will be the same version of Tcl?  Red Hat 6.2 comes with Tcl 8.0.5; AOLserver 3.3.1 comes with Tcl 8.3...

So far, after patching and supplying a --with-tcl parameter to the resulting configure script, I can at least build ns_fts under Red Hat 7.1 (which comes with Tcl 8.3.1)  -- but not under Red Hat 6.2.  Overall there is something not quite "feeling right" about assuming the version of Tcl on the machine is the same one as the one used when nsd was compiled, I think...?

Is there a way around this by pointing the Search-OpenFTS to look for the installed copy of Tcl under AOLserver's lib dir, maybe?  And meanwhile, is the AOLserver using Tcl 8.3 and Red Hat 7.1 using Tcl 8.3.1 slight mismatch likely to mean I am building an ns_fts that will behave badly?

Posted by Jonathan Marsden on

I just did a cvs checkout openacs-4 originally, and now cvs -q -z3 update -d -P brings in new modules as they arrive in CVS.

If you can't handle the idea of "just grab everything" for bandwidth reasons, then you may have to try and create a directory called packages/openfts-driver and also one called packages/search, and then do a cvs update -d -P to pull them in??

OK, here's my patch to Running autoconf after applying this, and then doing ./configure --with-tcl=/usr/lib seems to build ns_ftp OK. All the patch does is have configure look in one more place for the PG include files, the RPMs as distributed by the PGDG put them in /usr/include/pgsql not /usr/include/postgresql.

---   Fri Aug 10 01:28:55 2001
+++        Tue Sep  4 13:33:35 2001
@@ -339,15 +339,18 @@
 if test -d "$PG_INSTALLATION_DIR"; then
         AC_MSG_CHECKING([looking for postgresql include files])
-        if test ! -f $PG_INSTALLATION_DIR/include/postgres.h; then 
-                if test ! -f $PG_INSTALLATION_DIR/include/postgresql/postgres.h; then
-                        AC_MSG_ERROR([couldn't find postgresql include files])
-                else
+        if test -f $PG_INSTALLATION_DIR/include/postgres.h; then 
+                PG_INC="$PG_INSTALLATION_DIR/include"
+       else
+                if test -f $PG_INSTALLATION_DIR/include/postgresql/postgres.h; then
+                else
+                       if test -f $PG_INSTALLATION_DIR/include/pgsql/postgres.h; then
+                               PG_INC="$PG_INSTALLATION_DIR/include/pgsql"
+                       else
+                               AC_MSG_ERROR([couldn't find postgresql include files])
+                       fi
-        else
-                        PG_INC="$PG_INSTALLATION_DIR/include"
         AC_MSG_CHECKING([looking for postgresql libraries])

Posted by Dan Wickstrom on
Thanks for the patch Jonathan.  The configure script does indeed need some work, it's just that I've been busy with other things, so I haven't had a chance to get back to openfts and clean up a few things that need fixing.  Once the alpha-release dust settles, I should have a chance to get back and clean it up, but in the meantime patches, such as the one that you've included, are always welcome.
Posted by Ola Hansson on
Ehrm, I forgot to download OpenFTS. Hmm.. I applied your patch and I get this:
checking looking for postgresql include files... configure: error: couldn't find postgresql include files
I have RH 7.1 and installed PG 7.1.3 from source in the normal place...I wonder what I'm doing wrong?
Posted by Neophytos Demetriou on
Ola, try gmake-install-headers in the postgresql src directory.
Posted by Ola Hansson on
Nope, it doesn't work.
[root@infogettable src]# gmake-install-headers
bash: gmake-install-headers: command not found
Posted by Neophytos Demetriou on
Sorry, that should be gmake install-all-headers But I didn't actually meant the postgresql-X.X.X/src/. Try from postgresql-X.X.X/
Posted by Neophytos Demetriou on
Before adding any notes, you have to mount the openfts-driver. Visit the administration page and Initialize the OpenFTS Engine (This is mentioned in doc/INSTALL of the openfts-driver package). Then you are ready to go.
Posted by Ola Hansson on
Whew! It's working now. It's a lot easier, in fact, if one follows the docs at /packages/openfts-driver/www/doc/INSTALL...

The only thing I needed to alter was the configure statement:

./configure --with-aolserver-src=/usr/local/src/aolserver-3.3-ad13-nsvhr/aolserver/ --with-tcl=/usr/lib

Thanks to everyone involved and Neophytos who helped me via email!

Posted by jay he on
Hi. All. I installed openfts, following the instruction of Neophytos Demetriou in the /packages/openfts-driver/doc/INSTALL. But after restarted the server, I still can't bind search with openfts-driver.
Here are my steps:
1. gmake install-all-headers

2. installed intarray (in /usr/local/src/postgresql-7.1.3/contrib/intarray, make install)

3. - Download Search-OpenFTS-tcl-0.2.tar.gz from and save it to /tmp/Search-OpenFTS-tcl-0.2.tar.gz

4. - Log in as the root user:
su -
- Untar the archive file:
	cd /usr/local/src
	tar -xzvf /tmp/Search-OpenFTS-tcl-0.2.tar.gz

5. chown -R nsadmin.web Search-OpenFTS-tcl-0.2/
	cd Search-OpenFTS-tcl-0.2/
	./configure --with-aolserver-src=/usr/local/src/aolserver/aolserver/ --with-tcl=/usr/lib 
Then I got the following errors:
checking for strlen... (cached) yes
checking platform specific compile options... linux
checking looking for aolserver ns.h include file... /usr/local/src/aolserver/aol
checking looking for aolserver nsd.h include file... /usr/local/src/aolserver/ao
checking looking for libtcl8... /usr/local/src/aolserver/aolserver/tcl8.3.2/unix
checking looking for libthread... /usr/local/src/aolserver/aolserver/thread/libn
which: no postmaster in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local
./configure: test: =: unary operator expected
configure: error: directory:  doesn't exist!
I guess something wrong my postgresql installation. But I did install all those prerequisite. Any idea? Thanks,
Posted by Vinod Kurup on
Hi Jay,

This is the last error reported:

which: no postmaster in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local
So, my questions are:
  1. which user are you using to run ./configure?
  2. what is the PATH of that user -> "echo $PATH"?
The location of the postgresql binaries should be in the PATH of the user that runs the configure script. If you've followed the canonical openacs installation instructions, then the nsadmin user should have the proper PATH.
Posted by jay he on
Thanks. Vinod. I didn't notice that I should su - nsadmin. I did ./configure as root, which path doesn't include postmaster. After I tried ./configure as nsamdin, I can make and copy it to aolserver/bin directory. But when I tried to load load.sql file, I got a problem.
psql:/usr/local/src/postgresql-7.1.3/contrib/intarray/_int.sql:297: NOTICE:  current transaction is aborted, queries ignored until end of transaction block
psql:/usr/local/src/postgresql-7.1.3/contrib/intarray/_int.sql:303: NOTICE:  current transaction is aborted, queries ignored until end of transaction block
psql:/usr/local/src/postgresql-7.1.3/contrib/intarray/_int.sql:309: NOTICE:  current transaction is aborted, queries ignored until end of transaction block
psql:/usr/local/src/postgresql-7.1.3/contrib/intarray/_int.sql:315: NOTICE:  current transaction is aborted, queries ignored until end of transaction block
psql:/usr/local/src/postgresql-7.1.3/contrib/intarray/_int.sql:321: NOTICE:  current transaction is aborted, queries ignored until end of transaction block
psql:/usr/local/src/Search-OpenFTS-tcl-0.2/func_pgsql/create_func.sql:25: ERROR:
  stat failed on file '/usr/local/pgsql/lib/': No such file or directory
psql:/usr/local/src/Search-OpenFTS-tcl-0.2/func_pgsql/create_func.sql:29: ERROR:
  stat failed on file '/usr/local/pgsql/lib/': No such file or directory
I have Redhat7.2 installed, which has a default postgresql rpm. I uninstalled the rpm version and installed postgresql from source, which is in the directory /usr/local/src/postgresql-7.1.3/. I installed intarray under this directory. I don't know why load.sql looks for /usr/local/pgsql and it doesn't have lib/ file. And there is no lib directory under /usr/local/src/postgresql-7.1.3. Where did I make a mistake? Any idea? Thanks,