Forum OpenACS Development: Forums Reply-by-email problem
Recently, I set up reply-by-email for forums and ran into some problems, so wanted to document them for everyone to comment.
Reply-by-email allows forum participants to reply to messages directly from their email client. Hamilton has written up a nice document on how to do this.
When a user makes a post, notifications are sent out with a special Reply-To address. You then make sure that you can catch emails sent to those addresses and put them in a Maildir that the OpenACS instance can access. OpenACS then sweeps that maildir, reads the messages and inserts the messages into the forum in the appropriate place, based on the special Reply-To address.
Problems
procmail recipe didn't catch all out-of-office replies
The procmail recipe that Ham posted catches most "Out of Office" replies, but it missed one. That message then got inserted into the forums as a reply and sent out to users as a notification. That generated another autoreply from the same user. Autorepliers are supposed to be smart and not autoreply to the same address multiple times, but our changing Reply-To: header confuses some of them. So, basically, this causes an infinite mail loop.
RECOMMENDATION: When you're doing this for the first time on a production system, keep the EmailQmailQueueScanP parameter 0. Then, when you send out your first post, you can personally monitor the Maildir to make sure nothing has snuck through your recipes before letting OpenACS sweep up the replies.
FIX: Here are my revised procmail filters (I'm no expert, so please comment):
:0 w * ^subject:.*Out of Office AutoReply /dev/null :0 w * ^subject:.*Out of Office /dev/null :0 w * ^subject:.*out of the office /dev/null :0 w * ^subject:.*NDN /dev/null :0 w * ^subject:.*[QuickML] Error: /dev/null :0 w * ^subject:.*autoreply /dev/null :0 w * ^from.*mailer.*daemon /dev/null
That last one is important as some clients apparently don't bounce to the Envelope address, but rather to the reply-to. There's no other way to identify these emails as autoreplies. Note that these should really be swept up as bounces rather than being discarded to /dev/null.
FIX #2: We add a "Precedence: list" header to mail that we sent out. I read somewhere that autorepliers don't respond if you use "Precedence: bulk", so I changed to that. Not sure that it really made a difference.
couldn't turn off maildir sweeping without restarting server
As soon as I saw the autoreply posted to the forum, I wanted to turn off the notification scanner, so I set EmailQmailQueueScanP to 0. Unfortunately, this does NOT turn off mail scanning. The parameter is only read at server startup, so I had to restart the server as well.
FIX: I fixed this by making the scanner check the parameter before each scan, which I think is a much more useful behavior.
acs-mail-lite sweeper silently deletes forum replies
The sweeper that I have been talking about until now is the notification sweeper. acs-mail-lite has its own sweeper which also runs. It's supposed to take care of bounced emails and also take care of any other type of reply that comes in. Unfortunately, it seems to silently delete any forum replies. I haven't looked into why this is the case yet.
FIX: Separate replies from bounces. Make all forum-replies go into Maildir, and have all bounces go into Maildir-bounce, then point the notification sweeper at Maildir and the acs-mail-lite sweeper at Maildir-bounce.
BounceDomain wasn't set, so bounced emails were re-bouncing
The next problem was that the BounceDomain parameter was blank, so the bounce address on our envelopes was incorrect causing bounces to double bounce.
FIX: This may be fixed in more recent versions of OpenACS, but if it's not, just set BounceDomain in acs-mail-lite properly.
acs-mail-lite sweeper silently deletes bounces
So now I had bounces going into Maildir-bounce, but the sweeper was
just deleting them without marking the user_id's as it's supposed to. The
bounce address on outgoing emails is
bounce-${user_id}-${key}-${package_id}@[BounceDomain]. For
some reason, in our outgoing emails, package_id was blank, so
when the bounce sweeper tried to match the address with this regexp:
set regexp_str "^[bounce_prefix]-(\[0-9\]+)-(\[^-\]+)-(\[0-9\]+)\@"
it failed because of package_id being blank.
FIX: I changed it to:
set regexp_str "^[bounce_prefix]-(\[0-9\]+)-(\[^-\]+)-(\[0-9\]*)\@"
and it is now handling bounces correctly.
I have a feeling that the problem is that package_id is not getting set somewhere and this might fix both the Bounce problem and the notification problem, but I haven't found exactly where that's supposed to happen.
Questions
My main concern is that the "autoreply" filter is not robust and even 1 autoreply sneaking through can cause havoc. Has anyone run into this? I see that others have mentioned this, including Mark Aufflick in reply to Ham's post. How did you solve this? There must be other projects that have already solved the same issues, right?
In general you should not have a seperate directory for incoming notification e-mails. This is too much hassle to setup.
What should be done is to make sure that notifications has a callback implementation for acs_mail_lite::incoming_email that actually works. The callback mechanism itself works just fine (since 5 days ago).
In the long run, as forums wants to achieve the following goals:
* (1) Respond to post
* (1) Starting new threads
* (2) Subscribing to forum
* (2) Unsubscribing to a forum
we should not rely on a notifications callback but write forums callbacks instead.
As an example, writing a callback for "forums-subscribe-$forum_id@" and "forums-unsubscribe-$forum_id@" is a piece of cake.
For responding to a post I would have a convention like "forums@reply-$message_id@", which is essentially what notifications does at the moment.
As for starting a new thread: I would suggest to make forums parties and allow the administrator to set an E-Mail address for the forum, instead of relying on the standard e-mail "forums-post-$forum_id". "mailto:openacs@yoursite.com"; is much easier to remember.
My Idea was to use something like this on the subject:
set no_callback_p 0
set no_callback_p -nocase [regexp {.*Out of Office$} $subject]
set no_callback_p -nocase [regexp {.*Out of Office Autoreply$} $subject]
set no_callback_p -nocase [regexp {.*out of the office$} $subject]
aso. for all your examples. But it this acutally right (to assume that "out of office" will come at the end..) ?
How about international E-Mails? Are they using out of office as well?
Should all auto replies be considered bounces? If yes, can't we just say we check for mailer daemon instead of the various forms of "Out of Office" in the subject line?
I agree it's better to get all these filtering rules into OpenACS, rather than rely on installing procmail.
Here are what the regexp's should look like (partially stolen from Mark Aufflick's comment):
# check subject set no_callback_p [regexp -nocase "(out of.*office|automated response|autoreply)" $subject] set no_callback_p [regexp "NDN" $subject] set no_callback_p [regexp "[QuickML] Error" $subject] # check from set no_callback_p [regexp -nocase "mailer.*daemon" $from] # check body set no_callback_p [regexp -nocase "out of (the )?office" $body](Not sure if those are the right variable names for 'from' and 'body')
The out-of-office pieces can come anywhere in the subject, not necessarily at the beginning or end, so I would match without beginning or end-of-line markers.
International Emails are a good question. I don't know how mail clients in different countries format their out of office replies.
Unfortunately autoreplies are not bounces, so we can't just check for mailer-daemon in the From: field. Autoreplies created by mail clients are new, valid emails and are sent from the user's email address.
