Forum OpenACS Q&A: Installing OpenFTS, Search, Static Pages + General Comments

Just FYI, for all the newbies like myself out there, this is a step by step process of how I finally installed StaticPages, General Comments, Search + OpenFTS 0.3.2 with OpenACS 4.6.1.

This documents steps to install and configure StaticPages,
In the StaticPages docs, the first thing that is mentioned to do is Install Site-Wide-Search. If you are using postgres db, instead of SWS you need to install and setup OpenFTS and the Search package.

We need to do all the following if we want to use FTS and the Search package for postgres (to search StaticPages).

====== Install OpenFTS (Search-OpenFTS-tcl-0.3.2) =====
#!/bin/tcsh

# Start run as root (will su as needed to do installs).

# Make and install AOLserver nsfts module and postgres support for FTS. (Full Text Search)
# 1. Install aolserver before running this install.  See the reconf file in AOLserver src dir.

make distclean

# Can't use "set" since it won't export variable to shell when su.

setenv SRCDIR "/usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2"
setenv AOLSERVER_SRC "/usr/local/src/oacs/aolserver/aolserver"
setenv AOLSERVER "/usr/local/aolserver"
setenv TCL "/usr/local/tcl"
setenv POSTGRES_SRC "/usr/local/src/oacs/postgres/postgresql"
setenv DB_NAME "openacs-dev"

# 2. Build aolserver module:

  su --preserve-environment --shell=/bin/tcsh nsadmin

  cd $SRCDIR
  $SRCDIR/configure --with-aolserver-src=$AOLSERVER_SRC --with-tcl=$TCL/lib
  cd aolserver
  make
  cp nsfts.so $AOLSERVER/bin
  chmod 700  nsfts.so $AOLSERVER/bin/nsfts.so

  exit

# 3. Build and install postgresql tsearch option from contrib directory.

  # Make sure you have installed all headers (for example, spi.h) using
  # gmake-install-headers during the PostgreSQL installation.

  su --preserve-environment --shell=/bin/tcsh postgres

  cd $POSTGRES_SRC/contrib/tsearch
  make clean
  make
  make install
  psql template1 < tsearch.sql
  # NOTE:  Also load tsearch.sql into all existing db's you will use FTS with.
  # Installing in template1 makes openfts available to all new dbs that are created
  # If you don't want openfts in all of your newly created dbs, then only load this file
  # into the db you want to use openfts with.

  exit

  # root again.
  cp -r $SRCDIR/pgsql_contrib_openfts $POSTGRES_SRC/contrib
  chown -R postgres $POSTGRES_SRC/contrib

  su --preserve-environment --shell=/bin/tcsh postgres

  cd $POSTGRES_SRC/contrib/pgsql_contrib_openfts
  make  clean
  make
  make install
  # Installing in template1 makes openfts available to all new dbs that are created.
  psql template1 < openfts.sql
  # NOTE:  Also load openfts.sql into all existing db's you will use FTS with.

  exit

# 4. Edit nsd.tcl file and add entry for nsfts module:

# ns_section "ns/server/${server}/modules"
#        ns_param  nssock          ${bindir}/nssock.so
#        ns_param  nslog          ${bindir}/nslog.so
#  ...
#        ns_param  nsfts          ${bindir}/nsfts.so

# 5. Openacs-4 has a search package which uses Openfts.
#    If you are using a vanilla aolserver installation, the openfts/tcl/aolserver/test
#    directory has a few scripts that can used to test the search package.

exit

====== Done installing OpenFTS (Search-OpenFTS-tcl-0.3.2) =====

Restart AOLserver so nsfts module gets loaded.

In APM: Install packages for ACS Automated testing 4.0a and OpenFTS Driver 4.2.

Restart AOLserver

Mount the Test package in the Site Map.

  Click the new sub folder link on the Main Site line.
  Type test and click New.
  Click the new application link on the test line.
  Type Automated Test where it says untitled, choose ACS Automated Test from the drop-down list, and click New.

Mount the OpenFTS Full Text Search Engine in the Site Map.

  Click the new sub folder link on the Main Site.
  Type openfts and click New.
  On the openfts line, click the mount link.
  Click OpenFTS Driver.

(I had to uninstall Search and reinstall it in the APM first... don't know why)

Add application Search interface in the site map.

  Click the new sub folder link on the Main Site line.
  Type search and click New.
  Click the new application link on the search line.
  Type Search where it says untitled, choose search from the drop-down list, and click New.

Restart AOLserver

In the Site Map:
Visit the parameters of the newly created openfts-driver package and set the tcl-src-path to the path to the openfts source code. (e.g. /usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2)
NOTE that OpenACS will require access to OpenFTS source every time it starts up (don't remove the source):
[20/Mar/2003:14:23:36][9006.1024][-main-] Notice: sourcing files from /usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2

Restart AOLserver (so the openfts tcl source gets sourced).

In the Site Map, click on OpenFTS Driver link.

  Click on Administration.
  Click on Initialize OpenFTS Engine. Click Initialize OpenFTS Engine.

  If this step fails, (it will if you have an older FTS version installed or you had
  to drop and reload the OpenFTS Driver), the error would look like
  "Relation 'txt' already exists." Then you will need to drop the txt table using psql on
  your database. Then Initialize OpenFTS Engine again (as just above).

In Site Map, Click on the ACS Service Contract link.
  On the FtsEngineDriver line, click Install.

Restart AOLserver.

Finally, install StaticPages.

Install StaticPages from APM.
  This requires GeneralComments. Install this as well.

Restart AOLserver.

Add application StaticPages interface in the site map.
  Click the new sub folder link on the Main Site line.
  Type static-pages and click New.
  Click the new application link on the search line.
  Type Static Pages where it says untitled, choose Static Pages from the drop-down list, and click New.

In Site Map, Click on the ACS Service Contract link.
  On the "FtsContentProvider, content_revision" click Install.
  On the "FtsContentProvider, static_page" click Install.

Static pages has support for OpenFTS search. To index your static pages, you need to scan the filesystem. That is the only time static pages are indexed. You do this from the StaticPages admin page.
  Index your existing static pages in your site:
    In Site Map, click on Static Pages.
    Click on "Scan filesystem for static pages"

  If this fails, with "nsv array 'sp_sync_cr_fs_times' does not exist!" attempt the scan
  more than once (reload the page showing the error). The nsv_array seems to get defined
  at some point!

Test your installation of Static Pages:
  In Site Map, click on Search.
    Enter a keyword to search for. Try this with keywords that do and don't exist on your site;)

Add application GeneralComments interface in the site map.
  Click the new sub folder link on the Main Site line.
  Type general-comments and click New.
  Click the new application link on the search line.
  Type General Comments where it says untitled, choose General Comments from the drop-down list, and click New.

Restart AOLserver.

Now that you have Static Pages and General Comments installed, you can add General Comments links (Add Comment / Add Link) to your Static Pages. You needed Static Pages installed to use General Comments as Static Pages stuffs your static pages into the Content Repository in the database, which also makes OACS objects out of the static pages. General Comments uses the object ids to associate comments/links with the static pages.
Also notice that the GeneralComments administrative functions of toggling page comment-ability  are located on the Static Pages admin page;)

DONE:)

For future use of Search with any other packages:

Install any packages (in APM) you want to use Search with and install their corresponding FTSContentProvider service contracts from the acs-service-contract admin page (from Site Map).

Indexing existing content:

If you already have content in a package before you install Search, you will need to queue the existing items to be indexed. A query something like this will work nicely:

insert into search_observer_queue (select news_id, now(), 'INSERT' from cr_news);
Use whatever type-specific storage table for your package, "news_id" will be whatever field references acs_objects.object_id.

Where would you safe these pieces of information on the offical documentation of OpenACS
Well I would cleanup a couple typos I noticed rereading it right now! I think this could go in the Install Guide as an extra bit of info on installing the OpenFTS, Search, Static Pages + General Comments packages for postgres users. If it was put in with the docs on the individual packages then I think it might be broken up too much for people to follow easily. Maybe the individual package docs could link to this info for an overview of the install process? I'm new to OpenACS, and it wasn't clear to me how these packages interacted and I didn't know where to start looking for info on how to install them (or know what, exactly, I had to install) to get General Comments or Search working for instance.
I just noticed a section called "Set up site-wide search" in the Next Steps portion of the Install Guide! I wish I had found this when I was trying to install SWS. Maybe this could have it's own link in the Install Guide, since the rest of Next Steps looks like administrative stuff and I didn't even think to look for SWS install in there.
I have one more bit of information to add right after the step of testing Search (in my first post):

Test your installation of Static Pages: In Site Map, click on Search. Enter a keyword to search for. Try this with keywords that do and don't exist on your site;)

If the URLs returned by Search are broken (/www is hardcoded in the URLs when it shouldn't be), then apply these changes and restart AOLserver.

You have given the example of the news package in serach. Is the News search working for you?

I have posted this message https://openacs.org/forums/message-view?message_id=90800, could you please take a look.

Thanks,
Kiran

I reproduced the request error in case this will help.

Request Error
Database operation "dml" failed
    while executing
"error "$err""
    invoked from within
"if [catch {
    db_dml create_table "create table $table_name ( ${table_id} int not null primary key, path varchar unique, fts_index txtidx, last_modi..."
    ("uplevel" body line 21)
    invoked from within
"uplevel {
          ad_page_contract {

    Initialize OpenFTS

    @author Neophytos Demetriou

} {
    table_name
    table_id
    dict
    numbergroup..."
    (procedure "code::tcl::/home/phill/web/itgroup/packages/openfts-driver/w..." line 2)
    invoked from within
"code::tcl::$__adp_stub"
    invoked from within
"if { [file exists $__adp_stub.tcl] } {

      # ensure that data source preparation procedure exists and is up-to-date
      adp_init tcl $__adp_stub
..."
    ("uplevel" body line 3)
    invoked from within
"uplevel {

    if { [file exists $__adp_stub.tcl] } {

      # ensure that data source preparation procedure exists and is up-to-date
      adp_init t..."
    (procedure "adp_prepare" line 2)
    invoked from within
"adp_prepare "
    (procedure "template::adp_parse" line 30)
    invoked from within
"template::adp_parse [file root [ad_conn file]] {}"
    (procedure "adp_parse_ad_conn_file" line 7)
    invoked from within
"$handler"
    ("uplevel" body line 2)
    invoked from within
"uplevel $code"
    invoked from within
"ad_try {
    $handler
      } ad_script_abort val {
    # do nothing
      }"
    invoked from within
"rp_serve_concrete_file [ad_conn file]"
    (procedure "rp_serve_abstract_file" line 60)
    invoked from within
"rp_serve_abstract_file "$root/$path""
    ("uplevel" body line 2)
    invoked from within
"uplevel $code"
    invoked from within
"ad_try {
    rp_serve_abstract_file "$root/$path"
    set tcl_url2file([ad_conn url]) [ad_conn file]
    set tcl_url2path_info([ad_conn url]) [ad_conn path_inf..."

Hi I'm sorry, I'm not sure which error you are referring to. When does this Request Error occur? It doesn't seem to be related to the News Datasource problem as the error logs posted are not the same.