Forum OpenACS Development: Extend Forums to suport Email List archives
To my mind this is a ridiculous state of affairs, and I propose to remedy it, like so:
- Extend the OpenACS Forums package to support importing mailing list archives and displaying them as read-only web Forums.
- Also add the ability to take arbitrary mailboxes of saved emails and import them as if they were a list archive. Distinguish and filter out duplicate emails.
- Add admin UI to control permissions, categorization, content editing, etc. of such imported email data. (For private non-list emails, on initial import default the permissions to those users who actually received the emails.)
- Extend the functionality of Forums search. In addition to
plain full-text search, it should support searching for content by
particular authors, within particular dates, in particular
combinations of Forums, possibly by various metadata tags, etc.
The above features should make OpenACS very useful for archiving and searching the bazillions of emails saved at most companies and other organizations. Important business or technical details are often contained only in emails, which are then mis-filed and effectively lost. Yet most employees have big folders of old work emails lying around, they just can't organize or search them effectively, and can't let anyone else do so either.
But that's a solvable problem... Take all of those email folders, automatically suck them into your single centralized store, and type some queries into the "search" box.
- Add functionality to also export web-based Forums content
to email archive and/or Usenet format.
A very serious drawback of all current web-based forums is that they are 100% centralized, there is no easy built in distribution or redundancy at all for all that data. This has always worried me...
When ArsDigita went away, its years of BBoard content did not survive in any format amenable for importing into openacs.org or another OpenACS instance. We were lucky that any of that content surived as static HTML pages at Red Hat at all. If the OpenACS server ever melted down, would we lose content? I don't know, but I do know that we would be 100% dependent on retrieving a recent PostgreSQL backup, etc.
It shouldn't be that way. I should be able to provide a read-only mirror of all the OpenACS Forums content, by just asking the openacs.org maintainers to flip one switch.
I haven't actually looked at the OpenACS Forums code at all, so I don't know how difficult this would be. I believe OpenACS and/or Tcllib also both have various email handling functions which should help this work, but I'm not yet familiar with those either.
I would like to work on this, but probably won't have time until summer 2006. Anyone else also interested in this project?
Any comments or further thoughts?
if nothing else, it's interesting...
Btw, "Lurch", we strongly prefer real names in these forums, and registering with bogus "spam@" email addresses is seriously frowned upon.
The system should also be able to deal with attachments and also make them searchable.
Does anyone know which IMS specification would be used for forums content? That would be worth looking at as you looked into export.
Andrew, I have projects in the pipeline who might require us to do some of the pieces of this vision. I'll keep you in the loop if/when the projects happen.
Which reminds me to change forums to call acs-mail-lite::send (or complex_send, which supports multiple mime types) with the "no_callback_p" switch, if it is using acs-mail-lite in the first place...
So I'm quite interrested in this feature. However, I would make a separate package, because:
- Not all features of forums make sense for mailinglists
- There is metadata in e-mails which can be put to good use (message-id, references). The database tables for forums have not been designed for this purpose.
- User interface issues. E-mail might need a slightly different user interface than forums.
Mailman stores all its e-mail archives both in html format as in mbox format. The mbox format could be nicely used to build an user interface uppon and provide a search wide search service contract.
Of course the situation could be very different in a different mailinglist manager. What mailinglist manager were you thinking about?
Mailing list managers should be more or less irrelevant. AFAIK they all store any archived emails just like a client email reader does, in one of three or four different standard file formats. At least for the first pass that's all that matters. As the project actually gets going we'd want to look into that more, as it's likely that different mailing list manager programs insert additional headers into the emails which may contain useful metadata.
I have no opinion yet whether this new functionality should be added into the existing Forums package, constructed as a second package, or what. I definitely think that the existing Forums data model, code, and especially UI should be re-used as much as is possible and practical. Web-based forums, email lists, and Usenet groups are all merely differently flavored implementations of the same basic concept, the "discussion list", and the software design should reflect that. I seek to unify them.
For me it has to do with importing MASSIVE amounts of listserv archives (which are in mbox format) and making them categorizable and searchable (lists go back to the mid 1990s). This includes storing attachments (e.g. medical images) so they can be used and categorized.
Please contact me per email if you have time to meet soon (I will post my final proposal in the forums, but anyone else in the area is welcome)
Another feature a good mailing list archive should have is a button to say, "Please forward me a copy of this particular email." (Of course you must be required to login under SSL and have an authenticated email address in order to use that feature.) I've never yet seen an archive with that.
Clearly a new feature the got rolled out in the recent openacs.org upgrade. Whoever implemented that, kudos to you, it's nice.
We may also need to deal with email attachments when someone replies by email. Has anyone done this?