Forum OpenACS Q&A: Re: Need Help OpenFTS

Collapse
27: Re: Need Help OpenFTS (response to 26)
Posted by Demon Cypher on 08/02/06 05:59 PM
Hi Mageshwari,

OpenACS uses the OpenFTS package to implement site-wide-search. I installed openfts for the openacs
version is 5-1-5.

There are good instructions included in the OpenFTS instructions which I will repeat here.
Be sure to look at those instructions .... .

The following instructions taken from openACS forums and some from my own experience of installing
Sitewide-search. This doc consists of installing openFTS-Search module and configuring aolserver,
postgres accordingly and also configuring your openACS installation to enable search package,
implementing service contracts

====== Install OpenFTS (Search-OpenFTS-tcl-0.3.2) =====
#!/bin/tcsh

# Start run as root (will su as needed to do installs).

# Make and install AOLserver nsfts module and postgres support for FTS. (Full Text Search)

# 1. Install aolserver before running this install. See the reconf file in AOLserver src dir.

make distclean

# Can't use "set" since it won't export variable to shell when su.

setenv SRCDIR "/usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2"
setenv AOLSERVER_SRC "/usr/local/src/oacs/aolserver/aolserver"
setenv AOLSERVER "/usr/local/aolserver"
setenv TCL "/usr/local/tcl"
setenv POSTGRES_SRC "/usr/local/src/oacs/postgres/postgresql"
setenv DB_NAME "openacs-dev"

# 2. Build aolserver module:

su --preserve-environment --shell=/bin/tcsh nsadmin
cd $SRCDIR
$SRCDIR/configure --with-aolserver-src=$AOLSERVER_SRC --with-tcl=$TCL/lib
cd aolserver
make
cp nsfts.so $AOLSERVER/bin
chmod 700 nsfts.so $AOLSERVER/bin/nsfts.so
exit

# 3. Build and install postgresql tsearch option from contrib directory.

# Make sure you have installed all headers (for example, spi.h) using
# gmake-install-headers during the PostgreSQL installation.
su --preserve-environment --shell=/bin/tcsh postgres
cd $POSTGRES_SRC/contrib/tsearch

# NOTE: If you are using openACS 5-1 use tsearch2.. this the module used by tsearch2-driver(openACS)
make clean
make
make install
psql database_name < tsearch.sql

# NOTE: Also load tsearch.sql into all existing db's you will use FTS with.
# Installing in template1 makes openfts available to all new dbs that are created
# If you don't want openfts in all of your newly created dbs, then only load this file
# into the db you want to use openfts with.

exit

# root again.

cp -r $SRCDIR/pgsql_contrib_openfts $POSTGRES_SRC/contrib
chown -R postgres $POSTGRES_SRC/contrib
su --preserve-environment --shell=/bin/tcsh postgres
cd $POSTGRES_SRC/contrib/pgsql_contrib_openfts
make clean
make
make install

# Installing in database_name makes openfts available to all new dbs that are created.
psql database_name < openfts.sql
# NOTE: Also load openfts.sql into all existing db's you will use FTS with.
exit

Edit config.tcl (your openACS config file) file and uncomment following entry for nsfts module:

# ns_param nsfts ${bindir}/nsfts.so

====== Done installing OpenFTS (Search-OpenFTS-tcl-0.3.2) =====

Restart AOLserver so nsfts module gets loaded.

In APM: Install packages for OpenFTS Driver and tsearch2-driver

Restart AOLserver

=================================================================

Mount the OpenFTS Full Text Search Engine in the Site Map.

Click the new sub folder link on the Main Site.
Type openfts and click New.
On the openfts line, click the mount linkClick OpenFTS Driver.

Add application Search interface in the site map.

Click the new sub folder link on the Main Site line.
Type search and click New.
Click the new application link on the search line.
Type Search where it says untitled, choose search from the drop-down list, and click New.

Restart AOLserver

In the Site Map:
#NOTE: This is important step this where your openfts driver get recognized and loaded

Go to the admin/site-map and

Visit the parameters of the newly created openfts-driver package and set the
tcl-src-path to the path to the openfts source code. (e.g. /usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2)

#NOTE that OpenACS will require access to OpenFTS source every time it starts up (don't remove the source):

"[20/Mar/2003:14:23:36][9006.1024][-main-] Notice: sourcing files from /usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2"

Restart AOLserver (so the openfts tcl source gets sourced).

In the Site Map, click on OpenFTS Driver link.

Click on Administration.
Click on Initialize OpenFTS Engine. Click Initialize OpenFTS Engine.

If this step fails, (it will if you have an older FTS version installed or you had
to drop and reload the OpenFTS Driver), the error would look like
"Relation 'txt' already exists." Then you will need to drop the txt table using psql on
your database. Then Initialize OpenFTS Engine again (as just above).

In Site Map, Click on the ACS Service Contract link.
On the FtsEngineDriver line, click Install.

Restart AOLserver.

Finally Install News Package

Mount the News package in the Site Map.

Click the new sub folder link on the Main Site line.
Type test and click New.
Click the new application link on the test line.
Type News where it says untitled, choose news from the drop-down list, and click New.
In Site Map, Click on the ACS Service Contract link.
On the "FtsContentProvider, news" click Install.

=======================================================

Add a News item..

And search for that news item using search package

For future use of Search with any other packages:

Install any packages (in APM) you want to use Search with and install their corresponding FTSContentProvider service contracts from the acs-service-contract
admin page (from Site Map).

========================================================

IMPLEMENTING SERVICE CONTRACT WITH BUG-TRACKER

There are defalut service contracts for Forums,News,Static pages,file-storage packages
But If you want to implement the service contract for othere package such as BUG-TRACKER
follow these instructions

How to make an object type searchable?

Making an object type searchable involves three steps:

* Choose the object type
* Implement FtsContentProvider
* Add triggers

1) Choose the object type

In most of the cases, choosing the object type is straightforward. However, if your object type uses the content repository
then you should make sure that your object type is a subclass of the "content_revision" class. You should also make sure all
content is created using that subclass, rather than simply create content with the "content_revision" type.

#NOTE : Selecting correct object type is the most important thing in creating service contract

In Case of bugtracker package the object type is "bt_bug"

2) Implement FtsContentProvider

FtsContentProvider is comprised of two abstract operations, namely datasource and url. The specification for these
operations can be found in packages/search/sql/postgresql/search-sc-create.sql. You have to implement these operations for
your object type by writing concrete functions that follow the specification. For example, the implementation of datasource
for the object type "bt_bug", looks like this:
ad_proc bugtracker__datasource {
object_id
} {
@author Sriharsha Chintalapani (mailto:sriharsha.ch@azri.biz)
} {
db_0or1row bugs_datasource {
select b.bug_id as object_id,
b.comment_content as title,
b.summary as content,
'text/plain' as mime,
'' as keywords,
'text' as storage_type
from bt_bugs b
where bug_id = :object_id
} -column_array datasource
return [array get datasource]
}

ad_proc bug_tracker::url {
object_id
} {
returns a url for a message to the search package

} {

set package_id [db_string get_url "select package_id from cr_folders where folder_id = ( select parent_id from bt_bugs
where bug_id = :object_id) "]

set bug_number [db_string get_bugno "select bug_number from bt_bugs where bug_id=:object_id"]

set url_stub [ site_node::get_url_from_object_id -object_id $package_id]

set url "${url_stub}bug?bug_number=$bug_number"
return $url
}

copy the above code into a file (i named it as bugtracker-sc-procs.tcl) and put it in bugtracker/tcl/ dir

When you are done with the implementation of FtsContentProvider operations, you should let the system know of your
implementation. This is accomplished by an SQL file which associates the implementation with a contract name.
The implementation of FtsContentProvider for the object type note looks like:

select acs_sc_impl__new(
'FtsContentProvider', -- impl_contract_name
'bt_bug', -- impl_name
'bug-tracker' -- impl_owner_name
);You should adapt this association to reflect your implementation. That is, change impl_name with your object type and
the impl_owner_name to the package key. Next, you have to create associations between the operations of FtsContentProvider
and your concrete functions. Here's how an association between an operation and a concrete function looks like:

select acs_sc_impl_alias__new(
'FtsContentProvider', -- impl_contract_name
'bt_bug', -- impl_name
'datasource', -- impl_operation_name
'bug_tracker::bugtracker__datasource', -- impl_alias
'TCL' -- impl_pl
);

select acs_sc_impl_alias__new(
'FtsContentProvider', -- impl_contract_name
'bt_bug', -- impl_name
'url', -- impl_operation_name
'bug_tracker::url', -- impl_alias
'TCL' -- impl_pl
);

Again, you have to make some changes. Change the impl_name from bt_bug to your object type and the impl_alias from
bugtracker__datasource to the name that you gave to the function that implements the operation datasource.


Add triggers

If your object type uses the content repository to store its items, then you are done. If not, an extra step is required to
inform the search_observer_queue of new content items, updates or deletions. We do this by adding triggers on the table
that stores the content items of your object type. Here's how that part looks like for bt_bug.

create function bugtracker__itrg ()
returns opaque as '
begin
insert into search_observer_queue values (new.bug_id,now(),''INSERT'');
return new;
end;' language 'plpgsql';

create function bugtracker__dtrg ()
returns opaque as '
begin
perform search_observer__enqueue(old.bug_id,''DELETE'');
return old;
end;' language 'plpgsql';
create function bugtracker__utrg ()
returns opaque as '
begin
perform search_observer__enqueue(old.bug_id,''INSERT'');
return old;
end;' language 'plpgsql';

create trigger bugtracker__itrg after insert on bt_bugs
for each row execute procedure bugtracker__itrg ();

create trigger bugtracker__dtrg after delete on bt_bugs
for each row execute procedure bugtracker__dtrg ();

create trigger bugtracker__utrg after update on bt_bugs
for each row execute procedure bugtracker__utrg ();

In Site Map, Click on the ACS Service Contract link.
On the "FtsContentProvider, news" click Install. In the Valid Installed Bindings look for
FtsContentProvider bt_bug bt_bug bug-tracker install . Click on install

Restart AOL server..

Add a new bug to your bug tracker. Search for the bug using search package...
voila.....you got the results..

Now following the same procedure you can implement the search for any package you want to search..

---------------------------------------------HAPPY SEARCHING---------------------------------------------------

#IMPORTANT NOTES :

Indexing existing content:

If you already have content in a package before you install Search, you will need to queue the existing items to be indexed.
A query something like this will work nicely:

insert into search_observer_queue (select news_id, now(), 'INSERT' from cr_news);
Use whatever type-specific storage table for your package, "news_id" will be whatever field references acs_objects.object_id.


The search package supports two contracts, namely FtsEngineDriver and FtsContentProvider.

* FtsEngineDriver contains abstract descriptions of operations (search,index,unindex,update_index) that are common to search
engines (openfts, htdig,swish,intermedia.) The search package will only make use of one implementation of the FtsEngineDriver
contract. So far, only openfts-driver provides an implementation of FtsEngineDriver but efforts are under way to support other
search engines as well. The choice of implementation for FtsEngineDriver leads all indexing/searching done by the search
package.
* FtsContentProvider is comprised of two operations, namely datasource and url. The former returns information about a
content item (given its id) like title, content, mime, etc and it is used both while indexing and for displaying the results.
The latter returns the url of a content item given its id and it is used for displaying the search results.

When inserting/updating/deleting a content item that is being "observed" (triggers have been added), the search_observer_queue
table is updated with a record composed of the object_id, event (INSERT,UPDATE,DELETE), and the timestamp. With respect to the
event type, we call the appropriate operation from FtsEngineDriver, for example "index". Since operations are nothing more but
abstract descriptions, we are actually calling a chosen (a-priori -- in our case openfts-driver) implementation. In the case
of indexing, we are choosing the function that does the indexing using FtsEngineDriver implementations. Next, we retrieve the
content for an object_id using the implementation (with name the type of the object) of FtsContentProvider. Finally, the
chosen indexing function is called with the content passed as an argument.


**** THE ABOVE TUTORIAL BUILT ON THE BASIS OF OPENACS FORUMS AND MY OWN EXPERIENCE****

links you might be interested are:

https://openacs.org/forums/message-view?message_id=89360
https://openacs.org/forums/message-view?message_id=25295
https://openacs.org/doc/openacs-4-6-3/nextsteps.html