Forum OpenACS Development: Extend Forums to suport Email List archives

First some background to the itch I want to scratch: With the sole exceptions of the various versions of the OpenACS Forums/BBoard package and Google Groups, every web-based bulletin board software I've ever seen sucks. And as I've briefly mentioned elsewhere, every web-based interface to email list archives I've ever used also sucks, without exception.

To my mind this is a ridiculous state of affairs, and I propose to remedy it, like so:

1. Extend the OpenACS Forums package to support importing mailing list archives and displaying them as read-only web Forums.

2. Also add the ability to take arbitrary mailboxes of saved emails and import them as if they were a list archive. Distinguish and filter out duplicate emails.

3. Add admin UI to control permissions, categorization, content editing, etc. of such imported email data. (For private non-list emails, on initial import default the permissions to those users who actually received the emails.)

4. Extend the functionality of Forums search. In addition to plain full-text search, it should support searching for content by particular authors, within particular dates, in particular combinations of Forums, possibly by various metadata tags, etc.

   The above features should make OpenACS very useful for archiving and searching the bazillions of emails saved at most companies and other organizations. Important business or technical details are often contained only in emails, which are then mis-filed and effectively lost. Yet most employees have big folders of old work emails lying around, they just can't organize or search them effectively, and can't let anyone else do so either.

   But that's a solvable problem... Take all of those email folders, automatically suck them into your single centralized store, and type some queries into the "search" box.

5. Add functionality to also export web-based Forums content to email archive and/or Usenet format.

   A very serious drawback of all current web-based forums is that they are 100% centralized, there is no easy built in distribution or redundancy at all for all that data. This has always worried me...

   When ArsDigita went away, its years of BBoard content did not survive in any format amenable for importing into openacs.org or another OpenACS instance. We were lucky that any of that content surived as static HTML pages at Red Hat at all. If the OpenACS server ever melted down, would we lose content? I don't know, but I do know that we would be 100% dependent on retrieving a recent PostgreSQL backup, etc.

   It shouldn't be that way. I should be able to provide a read-only mirror of all the OpenACS Forums content, by just asking the openacs.org maintainers to flip one switch.

I haven't actually looked at the OpenACS Forums code at all, so I don't know how difficult this would be. I believe OpenACS and/or Tcllib also both have various email handling functions which should help this work, but I'm not yet familiar with those either.

I would like to work on this, but probably won't have time until summer 2006. Anyone else also interested in this project?

Any comments or further thoughts?

Links to any other projects with similar goals are definitely appreciated. But unfortunately, the guy doing MLwiki has given almost no info on just what exactly he wants to accomplish and how he's going about it - not very useful.

Btw, "Lurch", we strongly prefer real names in these forums, and registering with bogus "spam@" email addresses is seriously frowned upon.

as for 4). The mail tracking package already does part of that. At least it allows you to store an e-mail, the recipient, sending date and the object_id (why the e-mail was send in the first place). The resulting list is searchable and can be limited to a data range.

Which reminds me to change forums to call acs-mail-lite::send (or complex_send, which supports multiple mime types) with the "no_callback_p" switch, if it is using acs-mail-lite in the first place...

Daniël, of course I would prefer to save all useful mail header metadata so that going from email file archive to database and back does not lose anything significant. That will need investigation.

Mailing list managers should be more or less irrelevant. AFAIK they all store any archived emails just like a client email reader does, in one of three or four different standard file formats. At least for the first pass that's all that matters. As the project actually gets going we'd want to look into that more, as it's likely that different mailing list manager programs insert additional headers into the emails which may contain useful metadata.

I have no opinion yet whether this new functionality should be added into the existing Forums package, constructed as a second package, or what. I definitely think that the existing Forums data model, code, and especially UI should be re-used as much as is possible and practical. Web-based forums, email lists, and Usenet groups are all merely differently flavored implementations of the same basic concept, the "discussion list", and the software design should reflect that. I seek to unify them.

I recently stumbled across one little known web email list archive tool which doesn't totally stuck, ezmlm-browse (search on google). It has various flaws, e.g., no full-text search, and its UI features seem confusingly inconsistent. But, shockingly, it actually has a link to see all messages by a particular user. OpenACS/ACS has had that obvious feature since 1996 or so, Google Groups (the former DejaNews) has it, and now ezmlm-browse. So that's (only) three I'm aware of.

Another feature a good mailing list archive should have is a button to say, "Please forward me a copy of this particular email." (Of course you must be required to login under SSL and have an authenticated email address in order to use that feature.) I've never yet seen an archive with that.

There is also the issue of managing email lists and keeping spam off them. That's not really something I want to address in this project, but it is clearly related, and has substantial synergies with it and with OpenACS.

Donald Becker just recently raised this issue on the Beowulf list, and I made a few comments about it there.

I'm told that grepmail is useful for regexp and date-based searching of email archive files, and various MUAs have interfaces to it. grepmail's implementation is a Perl script, about 2400 lines total (including comments, etc.).

Caroline, no, no progress that I'm aware of. Carl Robert Blesius and I talked about it back in March. He was going to write up a draft spec., and I was going to read through the current Forums codebase to get an idea of how suitable it might be as a base for this project. Unfortunately, I never did that.   :(   I haven't been in touch with Carl since then, and don't know what progress he made on his spec.