Forum OpenACS Q&A: Excellent New Site

Posted by Simon at TCB on
Yup, might fine piece of work!

However, just a small quibble. The forums stuff seems a bit 'rough'?

The testing forum has about 80 odd page navigation numbers? Why so high? Where can I admin this forum now?

Messages I post up don't seem to appear on the list? Is this info cached?

Also, the messages sent out (I think someones noted this) have something like mailto:notification-33432-43423@openacs.orf in the sender? This is a little bit crappy. Any chance of at least having the forum it was from in the header? none of my email filters works now :o(

Just on a 'polish' point. The three links at the bottom of the messages received don't actually say what they're for.

Despite having not asked for it I appear to be getting two mails for each posting I put up. I also noted it says at the bottom of a post, something like 'subscribe to this message, warning you're already subscribed so you may get this twice'... urm, shouldn't it really *know* that?

Oh, and the timestamp is a bit meaningless without a timezone ;o)... Did we ever get any further with including timezone in someones profile? We could adjust the time displays for individuals then..

I don't remember most of these glitches in bboard when I recently tested it? Is this an older version or is it an entirely different package?

(I've just noticed it hasn't warned me about being double subscribed... but I am already receiving mail from this forum??)

2: Forum content (response to 1)
Posted by Andrei Popov on a bit slow too.  I guess the time is taken up to put the little blue "New" thingies next to a thread...
3: Re: Excellent New Site (response to 1)
Posted by Roberto Mello on
Like Jack, the Ripper would say, "let's do it by parts":

Regarding messages you post not showing up on the list, yes the information is cached, so there might be a delay in them showing up, although notifications will have been sent. Paginator does the caching.

Weird FROM addresses: Notifications has a feature that you can reply to a forums message via e-mail. That weird FROM address is how it keeps track of what message the e-mail is replying to. We're working to improve the forums messages "usability" with better headers and all that.

The three links at the bottom have [1],[2],[3]. Look at your message and you'll see the [1],[2],[3] pointing to those links.

The two e-mails is probably an artifact resulting from a combination of events. You probably changed your e-mail in your account from when we first migrated (August) and a couple days ago. Our migration script was supposed to catch that, but it didn't, so you now probably have two accounts created and subscribed. Look at and see if you find two of you, and delete one of them. We're finding a way to merge accounts.

Timezone: dunno.

The "glitches" as you can see, are not forums glitches (it has other glitches, but none that you pointed out). They are the result of a massive migration from a totally different code base and data model :-)

Our apologies for the temporary inconveniences. We are working on them.

4: Re: Excellent New Site (response to 1)
Posted by Walter McGinnis on
Thanks to all those involved for the new site!

I'll add another observation, I lost all my notifications for various bboards.  Its not  a big deal for me to turn them on again individually, but we may lose some readers that never re-up their notifications.

5: Re: Re: Excellent New Site (response to 3)
Posted by Bart Teeuwisse on
--Notifications has a feature that you can reply to a forums message via e-mail.--


whenever I reply to a notification by e-mail my message bounces with the following message:

Hi. This is the qmail-send program at
I'm afraid I wasn't able to deliver your message to the following addresses.
This is a permanent error; I've given up. Sorry it didn't work out.

Sorry, no mailbox here by that name. vpopmail (#5.1.1)

Is the reply by e-mail feature currently in place?


Posted by Tilmann Singer on
Apparently reply-by-email isn't set up yet. Judging from the comments in the file notifications/tcl/notifications-security-procs.tcl that is propably responsible for dealing with incoming replies it wouldn't be a good idea to activate this at a public site now, because the security checks necessary to make sure that a reply really came from the user that s/he claims to be are not implemented yet. Or maybe the code in that file should serve another purpose and I'm missing something, but there are procs like this in it:
    ad_proc -public can_notify_object_p {
        {-user_id ""}
    } { 
        # hack
        return 1
7: Re: Excellent New Site (response to 1)
Posted by Ben Adida on
Tilmann: although what you point out definitely needs to be fixed, that part doesn't apply to the security check for incoming emails. The incoming email feature is ready to be turned on.

Note that I'm personally not a huge fan of incoming email forum postings as I think it may lead to more duplication of posts and generally lower quality of postings. But the feature is there, and trying it for a bit would probably be a good idea.

8: Re: Excellent New Site (response to 1)
Posted by Jeff Davis on
Simon, the testing forum numbering is fixed (it was a bug in the paginator code; I fixed on 4.6 as well).

Ben, one pretty negative aspect with the "reply by email" is that it is so easily spoofed since it is all public information (the object id and user id). Maybe it does not matter that it is so easy to masquerade as another user but I think it would not be too hard to fix. We could add a signature to the reply-to address which could just be the first part hashed together with a server secret so you would have a reply-to like "" which could then be checked to make sure the reply to is genuine.

9: Re: Excellent New Site (response to 1)
Posted by Ben Adida on
Jeff: I agree with you: doing the reply by email correctly is a pain in the butt. There are other issues: what happens if someone forwards a notification to someone? In general, we found this to be complicated enough that we didn't try to add too much security.

That said, the hash solution you propose is one we considered. It could be done, even if it's going to make the From addresses even uglier.

Posted by Christian Eva on
"Our migration script was supposed to catch that, but it didn't, so you now probably have two accounts created and subscribed. Look at and see if you find two of you, and delete one of them. "

How can I delete an old registration? I logged in into on of my old registrations (fortunately remebering the password) but there is no 'remove your subscription" link anywhere?

Posted by Tilmann Singer on
If there was both a From: and a Reply-To: header then the hash could be added in the Reply-To: which would leave the From: header non-ugly (maybe even set to the posters address, but that's another topic).

I wonder what happens to the Reply-To: header when an email is forwarded - maybe it even gets discarded by most mail programs which would be nice in this case. I just tested it with mutt: it discards the Reply-To: header.

Another issue, propably solvable with a smart regex, would be the ignorant full-quote-at-the-bottom that many people just seem unable to avoid, and which would just look terrible in a forum.

I agree with Ben that is the wrong place to try this out, though. Maybe it should never be enabled for the public site, but this functionality certainly makes sense in other settings, e.g. an intranet that consists of trusted users only.

12: Re: Excellent New Site (response to 1)
Posted by Jeff Davis on
We had reply via email in the acs3 ticket tracker and it worked ok (maybe even well) for people reading and sending email from emacs or mutt or other text based email handlers.

It was disaster (or at least pretty unpleasant) when we had clients using outlook or even netscape to reply with mime attachments, bad html, and awful quoting.

I vote no (at least until someone takes the time to write good filters to rip out the more unpleasant stuff you will get with email replies).

Posted by Roberto Mello on
Bart, reply-by-email is not setup yet. I forgot to mention that in my previous e-mail.


14: Re: Excellent New Site (response to 1)
Posted by Dave Bauer on

We plan to merge duplicated accounts so the forums posting history will be retained. We haven't quite worked out the procedure for this yet though.

To manage subscriptions to the forums go to

There is a link to this in every email notification and also from "Your Workspace".

15: Re: Excellent New Site (response to 1)
Posted by Walter McGinnis on
I signed up for daily notifications, but I haven't received any since doing so.


Posted by Simon at TCB on
Thanks Jeff, much appreciated!
Posted by Christian Eva on
Thanks Dave

It is clear to me how to manage subscriptions. My question is still, how can you get rid of an old email registration. Only the Site-wide administrator has an option to delete a user.

Do you plan to incorporate a merge option to integrate the contributions from obsolete registrations into the actual one?

Should we send an email to to delete (or merge) old registrations?

18: Re: Excellent New Site (response to 1)
Posted by Janine Ohmer on
Quoted text in e-mailed replies is a big problem.  But the most insidious problem of all is "out of office" messages.  They're sent by the recipient with all the proper hashes, etc, and there are so many variations on the text that I don't think it would ever be possible to strip them all out.

We had one hosting client who had this happen to them.  It turned into a huge feedback loop (each notification containing the out of office reply is replied to again) and completely backed up incoming mail, not to mention really junking up the forum.

Maybe it's solvable, but I never figured out how to do it.  I think the best you can hope for is getting the really obvious ones, like "out of office reply" and "vacation".

19: Re: Excellent New Site (response to 1)
Posted by David Walker on
Assuming "out of office" messages always to go the "From" address and replies always go to "Reply To" it shouldn't be too hard to handle that particular issue.

Still the lack of any security in the smtp protocol and email makes putting words in someone else's mouth too easy.  Perhaps the "Reply-To" address should have a special encoding that only works with messages sent "From" the address that message goes to.

20: Re: Excellent New Site (response to 1)
Posted by Jun Yamog on
I just noticed my recent notification is a bit different.  The links below are now gone, and only a link back to the thread is present.  Whoever did this.  Thanks.  Its a useful and good change in my opinion.

Hopefully someone with spare time can improve search or make it up to par with the old search... then we are back to business.

21: Re: Excellent New Site (response to 1)
Posted by Dave Bauer on

Are you using the Forums search box? Or the sitewide search in the header. The forums specific search does not seem to be as effective as the OpenFTS based site-wide search. The forums instance installed here has been enabled to work with the Search package. The only disadvantage is that you cannot specify a Forums only search, but as most of the content on the site is in the forums, it is not that big a disadvantage.

22: Re: Excellent New Site (response to 1)
Posted by Jun Yamog on
Hi Dave,

I am using the Forums search.  Does the site wide search differ from the forums search?  I haven't tried using side wide search.

I just find the old 3.x search yeilds better results, maybe I am not yet used to the new search.

23: Re: Excellent New Site (response to 1)
Posted by Dave Bauer on
Forums specific search for postgresql is implemented with just a few database queries. The site-wide search uses the Search package with OpenFTS full text indexing. It is quite effective. Every page has a search form in the header that uses this site-wide search.
24: Re: Excellent New Site (response to 1)
Posted by Jun Yamog on
Hi Dave,

The site wide search is definitely better and maybe better than the old 3.x search.  Maybe someone can note in forums search results to try the SWS in case it does not yeild good results.  While we are waiting for a volunteer to make the forums OpenFTS, or maybe the other way around.. make OpenFTS forum aware.

In fact why not just remove the search on forums.  Then when someone is able to make OpenFTS work on segments of the site, we will just add a drop down box next to the search box.  That has "All, Forums, Documents, Comments, Site Page, ...".  Lets just remove the search on forums.

25: Re: Excellent New Site (response to 1)
Posted by Robert Locke on
The problem with Forums search has been documented here:

Basically, the issue is that it does a phrase search.  The ideal solution would be to replace the current forum search with the site-wide search solution (ie, OpenFTS) restricted to the forums package.  But I'm not sure if package-level searching has been implemented yet.

26: Need Help OpenFTS (response to 24)
Posted by mageshwari magi on
Dear Sir,

      We are trying to integrate the OpenFTS with our site. The OpenFTS package contains many files such as Install, Make File, Manifest,meta.yml,OpenFTS pm file and many folder. Each and every folder also contains many files. For integration which is the main file? We need your help to get the main file to integrate? please give your guidance.

Warm Regards


27: Re: Need Help OpenFTS (response to 26)
Posted by Demon Cypher on
Hi Mageshwari,

OpenACS uses the OpenFTS package to implement site-wide-search. I installed openfts for the openacs
version is 5-1-5.

There are good instructions included in the OpenFTS instructions which I will repeat here.
Be sure to look at those instructions .... .

The following instructions taken from openACS forums and some from my own experience of installing
Sitewide-search. This doc consists of installing openFTS-Search module and configuring aolserver,
postgres accordingly and also configuring your openACS installation to enable search package,
implementing service contracts

====== Install OpenFTS (Search-OpenFTS-tcl-0.3.2) =====

# Start run as root (will su as needed to do installs).

# Make and install AOLserver nsfts module and postgres support for FTS. (Full Text Search)

# 1. Install aolserver before running this install. See the reconf file in AOLserver src dir.

make distclean

# Can't use "set" since it won't export variable to shell when su.

setenv SRCDIR "/usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2"
setenv AOLSERVER_SRC "/usr/local/src/oacs/aolserver/aolserver"
setenv AOLSERVER "/usr/local/aolserver"
setenv TCL "/usr/local/tcl"
setenv POSTGRES_SRC "/usr/local/src/oacs/postgres/postgresql"
setenv DB_NAME "openacs-dev"

# 2. Build aolserver module:

su --preserve-environment --shell=/bin/tcsh nsadmin
$SRCDIR/configure --with-aolserver-src=$AOLSERVER_SRC --with-tcl=$TCL/lib
cd aolserver
chmod 700 $AOLSERVER/bin/

# 3. Build and install postgresql tsearch option from contrib directory.

# Make sure you have installed all headers (for example, spi.h) using
# gmake-install-headers during the PostgreSQL installation.
su --preserve-environment --shell=/bin/tcsh postgres
cd $POSTGRES_SRC/contrib/tsearch

# NOTE: If you are using openACS 5-1 use tsearch2.. this the module used by tsearch2-driver(openACS)
make clean
make install
psql database_name < tsearch.sql

# NOTE: Also load tsearch.sql into all existing db's you will use FTS with.
# Installing in template1 makes openfts available to all new dbs that are created
# If you don't want openfts in all of your newly created dbs, then only load this file
# into the db you want to use openfts with.


# root again.

cp -r $SRCDIR/pgsql_contrib_openfts $POSTGRES_SRC/contrib
chown -R postgres $POSTGRES_SRC/contrib
su --preserve-environment --shell=/bin/tcsh postgres
cd $POSTGRES_SRC/contrib/pgsql_contrib_openfts
make clean
make install

# Installing in database_name makes openfts available to all new dbs that are created.
psql database_name < openfts.sql
# NOTE: Also load openfts.sql into all existing db's you will use FTS with.

Edit config.tcl (your openACS config file) file and uncomment following entry for nsfts module:

# ns_param nsfts ${bindir}/

====== Done installing OpenFTS (Search-OpenFTS-tcl-0.3.2) =====

Restart AOLserver so nsfts module gets loaded.

In APM: Install packages for OpenFTS Driver and tsearch2-driver

Restart AOLserver


Mount the OpenFTS Full Text Search Engine in the Site Map.

Click the new sub folder link on the Main Site.
Type openfts and click New.
On the openfts line, click the mount linkClick OpenFTS Driver.

Add application Search interface in the site map.

Click the new sub folder link on the Main Site line.
Type search and click New.
Click the new application link on the search line.
Type Search where it says untitled, choose search from the drop-down list, and click New.

Restart AOLserver

In the Site Map:
#NOTE: This is important step this where your openfts driver get recognized and loaded

Go to the admin/site-map and

Visit the parameters of the newly created openfts-driver package and set the
tcl-src-path to the path to the openfts source code. (e.g. /usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2)

#NOTE that OpenACS will require access to OpenFTS source every time it starts up (don't remove the source):

"[20/Mar/2003:14:23:36][9006.1024][-main-] Notice: sourcing files from /usr/local/src/oacs/fts/Search-OpenFTS-tcl-0.3.2"

Restart AOLserver (so the openfts tcl source gets sourced).

In the Site Map, click on OpenFTS Driver link.

Click on Administration.
Click on Initialize OpenFTS Engine. Click Initialize OpenFTS Engine.

If this step fails, (it will if you have an older FTS version installed or you had
to drop and reload the OpenFTS Driver), the error would look like
"Relation 'txt' already exists." Then you will need to drop the txt table using psql on
your database. Then Initialize OpenFTS Engine again (as just above).

In Site Map, Click on the ACS Service Contract link.
On the FtsEngineDriver line, click Install.

Restart AOLserver.

Finally Install News Package

Mount the News package in the Site Map.

Click the new sub folder link on the Main Site line.
Type test and click New.
Click the new application link on the test line.
Type News where it says untitled, choose news from the drop-down list, and click New.
In Site Map, Click on the ACS Service Contract link.
On the "FtsContentProvider, news" click Install.


Add a News item..

And search for that news item using search package

For future use of Search with any other packages:

Install any packages (in APM) you want to use Search with and install their corresponding FTSContentProvider service contracts from the acs-service-contract
admin page (from Site Map).



There are defalut service contracts for Forums,News,Static pages,file-storage packages
But If you want to implement the service contract for othere package such as BUG-TRACKER
follow these instructions

How to make an object type searchable?

Making an object type searchable involves three steps:

* Choose the object type
* Implement FtsContentProvider
* Add triggers

1) Choose the object type

In most of the cases, choosing the object type is straightforward. However, if your object type uses the content repository
then you should make sure that your object type is a subclass of the "content_revision" class. You should also make sure all
content is created using that subclass, rather than simply create content with the "content_revision" type.

#NOTE : Selecting correct object type is the most important thing in creating service contract

In Case of bugtracker package the object type is "bt_bug"

2) Implement FtsContentProvider

FtsContentProvider is comprised of two abstract operations, namely datasource and url. The specification for these
operations can be found in packages/search/sql/postgresql/search-sc-create.sql. You have to implement these operations for
your object type by writing concrete functions that follow the specification. For example, the implementation of datasource
for the object type "bt_bug", looks like this:
ad_proc bugtracker__datasource {
} {
@author Sriharsha Chintalapani (
} {
db_0or1row bugs_datasource {
select b.bug_id as object_id,
b.comment_content as title,
b.summary as content,
'text/plain' as mime,
'' as keywords,
'text' as storage_type
from bt_bugs b
where bug_id = :object_id
} -column_array datasource
return [array get datasource]

ad_proc bug_tracker::url {
} {
returns a url for a message to the search package

} {

set package_id [db_string get_url "select package_id from cr_folders where folder_id = ( select parent_id from bt_bugs
where bug_id = :object_id) "]

set bug_number [db_string get_bugno "select bug_number from bt_bugs where bug_id=:object_id"]

set url_stub [ site_node::get_url_from_object_id -object_id $package_id]

set url "${url_stub}bug?bug_number=$bug_number"
return $url

copy the above code into a file (i named it as bugtracker-sc-procs.tcl) and put it in bugtracker/tcl/ dir

When you are done with the implementation of FtsContentProvider operations, you should let the system know of your
implementation. This is accomplished by an SQL file which associates the implementation with a contract name.
The implementation of FtsContentProvider for the object type note looks like:

select acs_sc_impl__new(
'FtsContentProvider', -- impl_contract_name
'bt_bug', -- impl_name
'bug-tracker' -- impl_owner_name
);You should adapt this association to reflect your implementation. That is, change impl_name with your object type and
the impl_owner_name to the package key. Next, you have to create associations between the operations of FtsContentProvider
and your concrete functions. Here's how an association between an operation and a concrete function looks like:

select acs_sc_impl_alias__new(
'FtsContentProvider', -- impl_contract_name
'bt_bug', -- impl_name
'datasource', -- impl_operation_name
'bug_tracker::bugtracker__datasource', -- impl_alias
'TCL' -- impl_pl

select acs_sc_impl_alias__new(
'FtsContentProvider', -- impl_contract_name
'bt_bug', -- impl_name
'url', -- impl_operation_name
'bug_tracker::url', -- impl_alias
'TCL' -- impl_pl

Again, you have to make some changes. Change the impl_name from bt_bug to your object type and the impl_alias from
bugtracker__datasource to the name that you gave to the function that implements the operation datasource.

Add triggers

If your object type uses the content repository to store its items, then you are done. If not, an extra step is required to
inform the search_observer_queue of new content items, updates or deletions. We do this by adding triggers on the table
that stores the content items of your object type. Here's how that part looks like for bt_bug.

create function bugtracker__itrg ()
returns opaque as '
insert into search_observer_queue values (new.bug_id,now(),''INSERT'');
return new;
end;' language 'plpgsql';

create function bugtracker__dtrg ()
returns opaque as '
perform search_observer__enqueue(old.bug_id,''DELETE'');
return old;
end;' language 'plpgsql';
create function bugtracker__utrg ()
returns opaque as '
perform search_observer__enqueue(old.bug_id,''INSERT'');
return old;
end;' language 'plpgsql';

create trigger bugtracker__itrg after insert on bt_bugs
for each row execute procedure bugtracker__itrg ();

create trigger bugtracker__dtrg after delete on bt_bugs
for each row execute procedure bugtracker__dtrg ();

create trigger bugtracker__utrg after update on bt_bugs
for each row execute procedure bugtracker__utrg ();

In Site Map, Click on the ACS Service Contract link.
On the "FtsContentProvider, news" click Install. In the Valid Installed Bindings look for
FtsContentProvider bt_bug bt_bug bug-tracker install . Click on install

Restart AOL server..

Add a new bug to your bug tracker. Search for the bug using search package... got the results..

Now following the same procedure you can implement the search for any package you want to search..

---------------------------------------------HAPPY SEARCHING---------------------------------------------------


Indexing existing content:

If you already have content in a package before you install Search, you will need to queue the existing items to be indexed.
A query something like this will work nicely:

insert into search_observer_queue (select news_id, now(), 'INSERT' from cr_news);
Use whatever type-specific storage table for your package, "news_id" will be whatever field references acs_objects.object_id.

The search package supports two contracts, namely FtsEngineDriver and FtsContentProvider.

* FtsEngineDriver contains abstract descriptions of operations (search,index,unindex,update_index) that are common to search
engines (openfts, htdig,swish,intermedia.) The search package will only make use of one implementation of the FtsEngineDriver
contract. So far, only openfts-driver provides an implementation of FtsEngineDriver but efforts are under way to support other
search engines as well. The choice of implementation for FtsEngineDriver leads all indexing/searching done by the search
* FtsContentProvider is comprised of two operations, namely datasource and url. The former returns information about a
content item (given its id) like title, content, mime, etc and it is used both while indexing and for displaying the results.
The latter returns the url of a content item given its id and it is used for displaying the search results.

When inserting/updating/deleting a content item that is being "observed" (triggers have been added), the search_observer_queue
table is updated with a record composed of the object_id, event (INSERT,UPDATE,DELETE), and the timestamp. With respect to the
event type, we call the appropriate operation from FtsEngineDriver, for example "index". Since operations are nothing more but
abstract descriptions, we are actually calling a chosen (a-priori -- in our case openfts-driver) implementation. In the case
of indexing, we are choosing the function that does the indexing using FtsEngineDriver implementations. Next, we retrieve the
content for an object_id using the implementation (with name the type of the object) of FtsContentProvider. Finally, the
chosen indexing function is called with the content passed as an argument.


links you might be interested are: