Forum OpenACS Development: Incoming email handling

Collapse
1: Incoming email handling
Posted by Guan Yang on 12/14/03 12:51 PM

A bunch of us are in Copenhagen working on the blogger. We want to implement email blogging/moblogging and we're discussing the best way to handle incoming email in OpenACS.

Approach 1: Script Handler

In qmail and most other mail servers it is possible to have email bound for a given address (or domain) to be handled by a script. In qmail this is done by having a .qmail-default file with 

|/path/to/script
/path/to/script will then be called and the message contents will be piped to the script in standard input.

That script would probably do some MIME parsing. It will then send the mail to OpenACS, either with a HTTP POST or an XML-RPC call. (Possibly it could also put the mail contents directly into the database with nstcl. However this would not be very different from the next approach.) An OpenACS Tcl script or XML-RPC handler will then process the mail according to some mapping defined by service contracts.

Pro: This approach is supported by almost all mail servers on Unix. If the script is written in Perl or Python, it could take advantage of the excellent MIME libraries that exist for these languages. Email can be handled much more synchronously, perhaps immediately, which is useful for "live" applications such as moblogging. (Note: If OpenACS is too busy the script should return an error, and the email server should retry later. Mail servers are good at queueing stuff.)

Con: This approach would probably not scale as gracefully for things like bounce handling. It requires an external script which might have dependencies like MIME libraries.

Approach 2: Sweeper

This is the approach currently implemented in acs-mail-lite. It works well for somewhat asynchronous applications like bounce handling and notification replies.

The approach involves the mail server putting all the messages in a Maildir-type directory. A scheduled proc then sweeps the directory, loads all the messages into the database, and sends them off to various handles registered with service contracts.

Pro: This approach will generally be faster than approach 1 above. It is already implemented in acs-mail-lite and appears to work well.

Con: It won't work well for applications where the user expects a near-immediate response. Procs scheduled to run more frequently than once a minute often cause problems.

Which approach is better? Would it be a good idea to support two types of incoming email, a script handler for immediate response and a sweeper for less time-sensitive apps?

Collapse
2: Re: Incoming email handling (response to 1)
Posted by Jonathan Ellis on 12/14/03 06:09 PM
It does seem that the sweeper approach is more straightforward -- instead of an external script having to POST to an oacs page to communicate you just have one scheduled proc.  Moving code out of nsd creates a whole new layer of complexity and IMO the case for it here isn't very compelling.

I've never noticed a problem with scheduled procs running too often and I'm pretty sure I have at least a couple running every 30s now...  Just move any procs that take so long that they're confusing the scheduler into their own thread.

Collapse
3: Re: Incoming email handling (response to 1)
Posted by Mark Aufflick on 12/15/03 06:35 AM
We had a discussion about this functionality not so long ago. The outcome of many hours of frustration with qmail (installed in this case with vpopmail) was to have a scheduled proc check a pop server using the tcl pop api. The tcl pop code had to be sourced rather than loaded as a library under nsd 3.x - I never did figure out why. The original module can of course be required under nsd 4.x instead of hacked like mine is, although ulimately it would be nice for it to be ad_proc'ed.

FYI i currently have the functionality hacked into my install of the 4.6 blogger, but it's pretty nasty.

Useful threads:

Collapse
4: Re: Incoming email handling (response to 1)
Posted by Sean Redmond on 12/15/03 07:25 PM

It will then send the mail to OpenACS, either with a HTTP POST or an XML-RPC call. (Possibly it could also put the mail contents directly into the database with nstcl. However this would not be very different from the next approach.)

Why use something like an HTTP POST instead of having the script insert the data itself, via DBI, for instance if it's a Perl script? (I guess nstcl-database is the equivalent of Perl's DBI, but I'm willing to bet that even in the OACS community a lot more people are familiar with the basic Perl modules.) I've done this on a couple of occasions for an intranet, giving the script a different DB user with just as much read and write privilege as it needs to do it's job. It's a lot easier to install, debug and enhance, and it adds hardly any complexity proportional to the overall complexity of OACS. It is also more Unix-y (it conforms to the rules of modularity, clarity, simplicity and separation, and I would argue also of parsimony, transparency, robustness, least surprise and diversity)

Also, instead of .qmail-default which will handle any otherwise unhandled address, you should use a more specific alias, e.g. .qmail-blogger (which would handle only emails to blogger@whatever.com). This leaves open the possibility of using other addresses for other purposes, and using subaliases, such as blogger-bob or blogger-alice, which would both be handled by .qmail-blogger, but the script would know that one was intended for Bob's blog and one for Alice's (à la ezmlm), without having to set up any mailboxes (not even the one required to use a sweeper.)

I'm curious about the "many hours of frustration with qmail" since I've found it to be about the least frustrating piece of software ever (no experience with vpopmail, though).

Collapse
5: Re: Incoming email handling (response to 4)
Posted by Guan Yang on 12/15/03 09:08 PM

Sean: It would of course be possible to have the incoming email handler put the contents directly into the database. Perhaps that's a better way of implementing it. I just thought that using HTTP POST or XML-RPC would involve slightly fewer dependencies for the script. A pure Perl implementation is easier to install than having to compile/link database libraries. For example Oracle client libraries are a bit hairy to link against.

Regarding the dot-qmail files, that should be up to the user/administrator and would depend on the application. On some sites you would obviously want more granular control of what email goes where. If you were implementing OpenACS Webmail (there was actually a webmail package for ACS Classic once) you would use a .qmail-default file. But using .qmail-default does not preclude the possibility of having other .qmail files; .qmail-blah-default would IIRC override .qmail-default.

I agree that qmail is a breeze to install and administer. But let's not turn this into a mail server flamewar 😉. Both of the approaches I've outlined work equally well with all mail servers (I think; if that turns out not to be the case, then start the flamewar).

Collapse
6: Re: Incoming email handling (response to 5)
Posted by Sean Redmond on 12/15/03 09:47 PM

I certainly don't mean to open a flame war! I'm just wondering if people think the sweeper approach is easier because they had trouble configuring qmail, when their trouble with qmail could easily be cleared up. Delivering mail to a script is so easy and reliable (from the perspective of the MTA) that, to me, setting a mailbox and having a sweeper read the mail like a human would, just faster, seems like a hopeless kludge.

Collapse
7: Moblogging revisited (response to 1)
Posted by Ben Koot on 02/24/07 08:51 PM
I realize this is an old discussion, but I was wondering what options the current OACS environment offers to set up a moblogging service. Many of my contacts are based in Africa and prefer to communicate via mobile phone since internet connectivity is limited or to expensive.

Maltes SMS / contacts solution is an interesting start, but I hope there are more options by now.

Thanks
Ben