Forum OpenACS Development: Re: RFC: New OpenACS Mail Procedures

Note:  I had some discussion til Til about this and I'll be adding his ideas and comments tonite.

One of the larger problems that is creeping into the current work on OpenACS
is a reliable way to make sure email messages can be sent from and delivered
to OpenACS.  There has been several packages added (and depreciated) dedicated
to the problem.  Much of it stems from a missing core package dedicated to
the process that can be expanded in a simple matter.

Currently I'm not defining specific API's and only some small test code has
been written to help me think out some of the process that are needed.  I
will be using the pseudo package name of mek::mail not because of an inflated
ego but to avoid confusion between OpenACS packages and a unwritten pseudo
package that currently exists in my head.

Please note, I will describe some processes that are currently do exists in
OpenACS without specifically naming them.  The goal of this document is to
present the core ideas and then and provide a framework for code.

There are two common ways to send mail, either create the actual SMTP
transaction or use the common UNIX /usr/sbin/sendmail or /usr/lib/sendmail
(called sendmail throughout this document ) system utility.  Both have pros
and cons and typically you don't have a sendmail mechanism on Windows
platforms.  Support for Windows and UNIX presents unique issues in designing
any email situation and most applications in general.  OpenACS (and
AOLserver) use TCL which erases some of the design barriers but not
completely.  The design of this system will attempt to create a design that
works on both platforms but initial attempts will not have that type of
compatibility.  It will be a future goal.

Sending Mail:

Sending mail will happen by either creating a SMTP transaction to a SMTP server
or using a sendmail process.  This will establish two mek::send send procedures
smtp and sendmail.  The mek::mail::send will call the process as
defined by a server parameter that can be controlled by the server admin(s).

mek::send {
  -package_id    #sending package id
  -env          #message envelope sender
                #(defaults to <server_parameter>@server_hostname)
  -to_addrs      #list of user_ids to send the message to (Required)
  -from_addr    #From: address of message (If not set, defaults to envelope
                #sender)
  -subject      #Subject of message (Required)
  -body          #Body (text) of message (Required)
  -extraheaders  #Additional headers for message body
  -bcc          #For compatibility, but will be ignored.
}

A message-id header will be created using the format
[ns_time].message_id@hostname.  This will allow each message (and its
recipients) to be recorded with the sending package id and insure
uniqueness for the message id.  The message_id  header will then be added to
the extraheaders paramater.  mail::send will then call the sending program
with the parameters (env,to_addr_from_addr (taken from the user_id),subject,
body, extraheaders),  The sending procedure will return 0 if the message is
successfully delivered and 1 if there was a failure.
The status for each to_address will be saved as each address is tried.  Each
message will be tried 3 times if there is a failure.  If each message fails,
each time, sending mail will be suspended and a error message will be recorded
in the log file.  Currently I am unaware of an non-email way to alert the
administrator of an error but logging.  If a message fails to send to
all addresses, an error in sending email will be indicated and an error in
the log files appears to be on the only way to notify the admin.  The
notification will appear in the logs periodically.

If less than 100% of the message return error, then for each user:

If mail is successful:
o email_failing will be set to null for the user.

If mail is failing:
o A check for email_failing in the user profile will be check, if unset the
  current time will be set for email_failing.
o The current time will be entered in email_last_failure in the user's
  profile.
o If both email_failing and email_last_failure are set, a server parameter
  (disable_failing_mail_days) will be use to determine if email should be
  disabled for the user.

mek::send::smtp will be a wrapper around SMTP compliant ns_sendmail
replacement that returns 0 if the transaction is successful and 1 if
there is a failure.

mek::send::sendmail will be a wrapper around sendmail, controlled by a
server parameter to defined the path to the sendmail program.

OpenACS also needs a way to accept and process incoming mail.  For one reason,
email addresses do become inactive and a user may forget to de-active that
address on the system.  It would also be desirable to accept replies to
messages send out from OpenACS or, for example, to allow a email user to
start a new thread in the forums package.

To handle bounces for the system we will define a server parameter that is
an address that will be set in the envelope header.  In addition we will
add the to email address to the end of the envelope address to create an
envelope address in the form parameter-email=domain.com@hostname.  This is
referred to as a Variable Envelope Return Path (VERP)
(http://cr.yp.to/proto/verp.txt).  Mail servers will bounce message back to the
envelope sender.  All modern mail servers have a process to handle VERP's
and will be an important part in the incoming mail process.

The server's MTA will feed the message through standard input, to a script
that will parse the message and insert the message data into the OpenACS
incoming mail database table.  The script will parse out the destination and
from address from the message and record the arrival time.  Each message
inserted into the database will be given a unique id.

Periodically a process within OpenACS will be run to process all messages in
the incoming mail queue.  Each OpenACS system will have a sever parameter
defined as to the address that will handled bounces, in addition each
OpenACS package that wants to receive mail will be required to create a
process_mail procedure and a unique email address for that package.  If you
assign the email address forums to a forum instance, all messages delivered
to forums-<anything> would be handled by that package's process_mail procedure.
That procedure could, in turn, look at the second part of the message
forums-<second_part>-<anything> and give each forum in the package an address.
For example, OpenACS's site could have email address for the following:

mailto:forums-development@openacs.org
mailto:forums-qanda@openacs.org

They could then check the third part, forums-development-<third> and use the
third part to insert a message in reply to a forum message, or if the the
third part is not present create a new thread in the forum.  The package will
be able to control how they process mail.

The bounce case will be handled in the exact same way as bad addresses went
sent.

o A check for email_failing in the user profile will be check, if unset the
  current time will be set for email_failing.
o The current time will be entered in email_last_failure in the user's
  profile.
o If both email_failing and email_last_failure are set, a server parameter
  (disable_failing_mail_days) will be use to determine if email should be
  disabled for the user.

Any message that are not handled by the incoming queue process will
be bounced to the sender.