Forum OpenACS Development: RFC: New OpenACS Mail Procedures
One of the large problems that is creeping into the current work on OpenACS is a
reliable way to make sure email messages can be sent from and delivered to Open
ACS. There has been several packages added (and depreciated) dedicated to the p
roblem. Much of it stems from a missing core package dedicated to the process t
hat can be expanded in a simple matter.
Currently I'm not defining specific API's and only some small test code has been
writing to help me think out some of the process that are needed. I will be us
ing the pseudo package name of
mek::mail not because of an inflated ego but to avoid confusion between OpenACS
packages and a unwritten pseudo package that currently exists in my head.
Please note, I will describe some process that are currently do exists in OpenAC
S without specifically naming them. The goal of this document is to present the
core ideas and then and provide a framework for code.
There are two common ways to send mail, either create the actual SMTP transactio
n or use the common UNIX /usr/sbin/sendmail or /usr/lib/sendmail (called sendmai
l throughout this document ) system utility. Both have pros and cons and typica
lly you don't have a sendmail mechanism on Windows platforms. Support Windows
and UNIX present unique issues in designing any email situation and well, most a
pplication is general. OpenACS (and AOLserver) use TCL which erase some of the
design barriers but does not completely erase them. The design on this system w
ill attempt to create a design that works on both platforms but initial attempts
will not have that type of compatibility. It will be a future goal.
Sending Mail:
Sending mail will happen by either creating a SMTP transaction to a SMTP server
or using a sendmail process. This will establish two mek::send send procedures
smtp and sendmail. The mek::mail::send will call the process defined as defined
by a server parameter that can be controlled by the server admin(s).
mek::send {
-package_id #sending package id
-env #message envelope sender (defaults to <server_parameter>@server_
hostname)
-to_addrs #list of user_ids to send the message to (Required)
- from_addr #From: address of message (If not set, defaults to envelope send
er)
- subject #Subject of message (Required)
- body #Body (text) of message (Required)
- extraheaders #Additional headers for message body
-bcc #For compatibility, but will be ignored.
}
Procedure will call mek::send::smtp or mek::send::sendmail as directed by server
parameter. Will create a message-id header using the format "[ns_time].message
_id@hostname". This will allow each message (and its recipients) to be recorded
with the sending package id and insure uniqueness for the message id. The mess
age_id header will then be added to the extraheaders paramater. mail::send will
then call the sending program with the parameters (env,to_addr_from_addr (taken
from the user_id),subject,body,extraheaders), The sending procedure will retur
n 0 if the message is successfully delivered and 1 if there was a failure.
The status for each to address will be saved add each address is tried. Each me
ssage will be tried 3 times if there is a failure. If each message fails, each
time mail sending will be suspended and a error message will be records in the l
og file. Currently I am unaware of an non-email way to alert the administrator
of an error unless mail is used. If a message fails to send to all addresses, an error is sending email would be indicated and an error in the log files appear
s to be on the only way to notify the admin.
If less than 100% of the message return error, then for each user:
o A check for email_failing in the user profile will be check, if unset the curr
ent time will be set for email_failing.
o The current time will be entered in email_last_failure in the user's profile.
o If both email_failing and email_last_failure are set, a server parameter (dis
able_failing_mail_days) will be use to determine if email should be disabled for
the user.
If mail is successful:
o email_failing will be set to null for the user.
mek::send::smtp will be a wrapper around SMTP compliant ns_sendmail replacement
that returns 0 if the transaction is successful and 1 if there is a failure.
mek::send::sendmail will be a wrapper around sendmail, controlled by a server pa
rameter to defined the path to the sendmail program.
Handling incoming mail.
To be completed 4/24/2003.
Besides the error handling I don't see anything in this proposal that is not already implemented in the modified acs-mail-lite package I sent you, yet I think there is some useful stuff in that package that you are not mentioning, most notably the maintainance of a queue for outgoing mail. Since we can always rely on having a working database connection in openacs (otherwise the whole system would be down), but mail sending capability might be subject to temporary failure, a queue as implemented in acs-mail-lite makes a whole lot of sense and should be used for all outgoing mail in OpenACS in my opinion.
Since you suggest to change the return value depending on delivery success you obviously don't plan to use a queue, where the send command would just enqueue an entry and return almost immediately. I would rather suggest to let the send proc return a queue_entry_id which uniquely identifies the new entry and enables the package to later query the delivery status of that entry if it desires (would be discarded in most cases).
Message-Id creation is already in the modified acs-mail-lite package as well, and also the possibility for callers of the ::send proc the possibility to provide their own Message-Id if desired. Currently the created Message-Id is being returned by the send proc, but it occured to me that it might not be unique so it should rather return an internal identifier like queue_entry_id.
Something that needs to be adressed is the handling of rfc style email addresses. Should the caller be responsible for generating an rfc string like this 'Fritz Fitzke <mailto:fritz@blitz.com>' or should the send proc take email and optionally pretty name seperately and assemble that string itself? The way it is currently solved in my modified package is that for each address to pass in there are two parameters, e.g. -from_addr and -from_pretty, where the _pretty argument is optional. I think that is more convenient to use, but it does not allow for parameters that need to pass in multiple addresses, e.g. for a Cc: list. Unsolved problem
Regarding the usage of different mail backends, that is also already implemented partially in the modified package. Don has recommended to use a service contract for each implementation, which is something that I would like to add to that package, now that service contracts don't need tedious sql declarations anymore. Also Don suggested to add one implementation that does not actually send the mail but simply displays it in some web UI, for testing purposes.
Error handling: wouldn't an error when sending rather be a (temporary) failure of the mail system as a whole, than an error with the individual recipient? I guess the individual failures can only be identified after receiving and processing bounce messages, no? The only kind of immediate failure would be a malformed email address, something that would propably happen rarely in real life. So I think that is part of the incoming email part, which will have to deal with bounces.
Bottom line: much of the stuff we need for outgoing email is already implemented. I volunteer to add the missing parts listed here so that we finally get on with this.
Should we add these features to acs-mail? I am just worried about it's complicated datamodel and acs-messaging integration. Not sure what it is good for, besides the possibility to send multipart mails (text and html, the receiving client chooses which it wants to display) - do we really need that?
(Also available at http://shell1.alal.net/~kovachme/acs-mail.txt).
Note: I had some discussion til Til about this and I'll be adding his ideas and comments tonite.
One of the larger problems that is creeping into the current work on OpenACS
is a reliable way to make sure email messages can be sent from and delivered
to OpenACS. There has been several packages added (and depreciated) dedicated
to the problem. Much of it stems from a missing core package dedicated to
the process that can be expanded in a simple matter.
Currently I'm not defining specific API's and only some small test code has
been written to help me think out some of the process that are needed. I
will be using the pseudo package name of mek::mail not because of an inflated
ego but to avoid confusion between OpenACS packages and a unwritten pseudo
package that currently exists in my head.
Please note, I will describe some processes that are currently do exists in
OpenACS without specifically naming them. The goal of this document is to
present the core ideas and then and provide a framework for code.
There are two common ways to send mail, either create the actual SMTP
transaction or use the common UNIX /usr/sbin/sendmail or /usr/lib/sendmail
(called sendmail throughout this document ) system utility. Both have pros
and cons and typically you don't have a sendmail mechanism on Windows
platforms. Support for Windows and UNIX presents unique issues in designing
any email situation and most applications in general. OpenACS (and
AOLserver) use TCL which erases some of the design barriers but not
completely. The design of this system will attempt to create a design that
works on both platforms but initial attempts will not have that type of
compatibility. It will be a future goal.
Sending Mail:
Sending mail will happen by either creating a SMTP transaction to a SMTP server
or using a sendmail process. This will establish two mek::send send procedures
smtp and sendmail. The mek::mail::send will call the process as
defined by a server parameter that can be controlled by the server admin(s).
mek::send {
-package_id #sending package id
-env #message envelope sender
#(defaults to <server_parameter>@server_hostname)
-to_addrs #list of user_ids to send the message to (Required)
-from_addr #From: address of message (If not set, defaults to envelope
#sender)
-subject #Subject of message (Required)
-body #Body (text) of message (Required)
-extraheaders #Additional headers for message body
-bcc #For compatibility, but will be ignored.
}
A message-id header will be created using the format
[ns_time].message_id@hostname. This will allow each message (and its
recipients) to be recorded with the sending package id and insure
uniqueness for the message id. The message_id header will then be added to
the extraheaders paramater. mail::send will then call the sending program
with the parameters (env,to_addr_from_addr (taken from the user_id),subject,
body, extraheaders), The sending procedure will return 0 if the message is
successfully delivered and 1 if there was a failure.
The status for each to_address will be saved as each address is tried. Each
message will be tried 3 times if there is a failure. If each message fails,
each time, sending mail will be suspended and a error message will be recorded
in the log file. Currently I am unaware of an non-email way to alert the
administrator of an error but logging. If a message fails to send to
all addresses, an error in sending email will be indicated and an error in
the log files appears to be on the only way to notify the admin. The
notification will appear in the logs periodically.
If less than 100% of the message return error, then for each user:
If mail is successful:
o email_failing will be set to null for the user.
If mail is failing:
o A check for email_failing in the user profile will be check, if unset the
current time will be set for email_failing.
o The current time will be entered in email_last_failure in the user's
profile.
o If both email_failing and email_last_failure are set, a server parameter
(disable_failing_mail_days) will be use to determine if email should be
disabled for the user.
mek::send::smtp will be a wrapper around SMTP compliant ns_sendmail
replacement that returns 0 if the transaction is successful and 1 if
there is a failure.
mek::send::sendmail will be a wrapper around sendmail, controlled by a
server parameter to defined the path to the sendmail program.
OpenACS also needs a way to accept and process incoming mail. For one reason,
email addresses do become inactive and a user may forget to de-active that
address on the system. It would also be desirable to accept replies to
messages send out from OpenACS or, for example, to allow a email user to
start a new thread in the forums package.
To handle bounces for the system we will define a server parameter that is
an address that will be set in the envelope header. In addition we will
add the to email address to the end of the envelope address to create an
envelope address in the form parameter-email=domain.com@hostname. This is
referred to as a Variable Envelope Return Path (VERP)
(http://cr.yp.to/proto/verp.txt). Mail servers will bounce message back to the
envelope sender. All modern mail servers have a process to handle VERP's
and will be an important part in the incoming mail process.
The server's MTA will feed the message through standard input, to a script
that will parse the message and insert the message data into the OpenACS
incoming mail database table. The script will parse out the destination and
from address from the message and record the arrival time. Each message
inserted into the database will be given a unique id.
Periodically a process within OpenACS will be run to process all messages in
the incoming mail queue. Each OpenACS system will have a sever parameter
defined as to the address that will handled bounces, in addition each
OpenACS package that wants to receive mail will be required to create a
process_mail procedure and a unique email address for that package. If you
assign the email address forums to a forum instance, all messages delivered
to forums-<anything> would be handled by that package's process_mail procedure.
That procedure could, in turn, look at the second part of the message
forums-<second_part>-<anything> and give each forum in the package an address.
For example, OpenACS's site could have email address for the following:
mailto:forums-development@openacs.org
mailto:forums-qanda@openacs.org
They could then check the third part, forums-development-<third> and use the
third part to insert a message in reply to a forum message, or if the the
third part is not present create a new thread in the forum. The package will
be able to control how they process mail.
The bounce case will be handled in the exact same way as bad addresses went
sent.
o A check for email_failing in the user profile will be check, if unset the
current time will be set for email_failing.
o The current time will be entered in email_last_failure in the user's
profile.
o If both email_failing and email_last_failure are set, a server parameter
(disable_failing_mail_days) will be use to determine if email should be
disabled for the user.
Any message that are not handled by the incoming queue process will
be bounced to the sender.
An issue that is not that easy to answer in my opinion is the question of the format of the message body. The old acs-mail package is capable of sending a mail either in text/plain or in text/html, or in multipart/alternative, which contains both a text and a html version and lets the receiving mail program decide which version to display. Also I think it was capable of sending aolhtml, which is a subset of html and a different mime-type. Maybe it is even capable of including any mime type in a multipart/mixed message. I wonder what of this functionality if at all we want to retain, and suggest to limit ourselves to a message body content type of text and html for simplicitys sake. We could then include a body_text and body_html field in the queue table, sending out whichever is filled or multipart/alternative if both are not NULL.
Unless someone comes up with a good reason why message body in a mime-type different than text/plain or text/html would make sense one time. (text/xml? what for?)
Limiting the message body that way would make the datamodel a lot simpler than that of the current acs-mail package. Which I don't fully understand to be honest.
Another functionality of the old acs-mail package that I suggest to drop is that it tries to minimize database storage by separating message bodies from the metadata and possibly reusing message bodies, in case the same message was sent to multiple recipients. This only makes sense when the message is intended to be stored for longer as reference (which no package uses right now I think), not when it is meant to be sent out quickly and removed from the queue anyway.
It would be good to add to the requirements that we should have a service contract based send framework that supports pluggable transport mechanisms (ns_sendmail, qmail-inject, ...).
Why does your mek::send procedure take a package_id and what does it do with it? Do you plan to keep the entries generated in the queue after a successful send in general or would you agree that they only stay in the queue as short as necessary?
Should the rfc Message-ID really be unique? What about a bulkmail process that sends out a mass mailing which is slightly personalised, e.g. by replacing the name of the recipient in the body. Wouldn't one in this case want to have the possibility to send out each of these messages with the same Message-Id? This could be achieved if mek::generate_message_id would be publicly callable and mek::send would optionally accept -rfc_message_id as parameter. I can't think of another example where this would be useful right now, but somehow I have the feeling that we should support non-unique rfc Message-IDs.
On a minor side-note I suggest to add .oacs to the rfc message-id as implemented in the modified acs-mail-lite I sent you, which makes it immediately clear to someone looking at the headers that the message was generated by openacs.
We need a good flexible solution to pass email addresses in procedure parameters. It should deal with user_id's, single email addresses ('fritz@fritz.com') and pretty addresses ('Fritz Fitzke <fritz@fritz.com>'), possibly not burdening the caller with the assembly of the pretty string. Some parameters allow for single addresses only (e.g. From:), others for multiple addresses (e.g. Cc:).
What about liberally accepting a list and interpreting it so that all the following calls were legal:
mek::send -to 12345
mek::send -to {12345 12346 132457}
mek::send -to fritz@fritz.com
mek::send -to {"Fritz Fitzke" <fritz@fritz.com>}
mek::send -to {{"Fritz Fitzke" <fritz@fritz.com>} {"Test User" <test@user.com>}}
The way to correctly distinguish the last two forms would be a bit kludgy but it would work somehow, and this method would at least save us from having to define several variants per parameter such as -to, -to_pretty etc.
The generation of the pretty form could be done by a utility proc, e.g.
mek::build_address -email fritz@fritz.com -pretty_name "Fritz Fitzke"would return {"Fritz Fitzke" <fritz@fritz.com>}
So that the last from the examples above could be also written like that:
mek::send -to [list \ [mek::build_address -pretty_name "Fritz Fitzke" -email "fritz@fritz.com"] \ [mek::build_address -pretty_name "Test User" -email "test@user.com"] \ ]
what is the status with this package. We will start tomorrow (tuesday, 17th of June) with the extension of acs-mail-lite to support the bouncemanagement most likely using VERPs. Anything that you have developed would be cool to lay hands on so we could save some work.
For the rest of the commmunity, we will develop a bouncemanagement system for acs-mail-lite within the next 7-10 days, we already enabled immediate sending of emails (without storing them in the database for the scheduler).
SMTP or /usr/sbin/sendmail support.
You can now send emails using SMTP (a modified version of ns_sendmail) or /usr/sbin/sendmail (which all major UNIX MTA's support).
Message-ID: support.
If not provided, a Message-ID: header will be added to the provide email before it is sent. The message id that is created is:
<[ns_time].acs_mail_lite_sequence_id.[ns_rand 32767].oacs@domain]>
Some public functions have been addeded to parse an email address, among others.
Bounce support.
Simple bounce support. For each email sent, and envolope address is created in the form:
bounce-email=domain.com@a.defined.domain.com
(where bounce can be a use set option).
Bounces are recorded in the database. If the time of the first recorded bounce is more that 10 days ago, a notification email will be sent and all bounces for the user will be removed from the database. If a bounce is received after the notification is sent, email_bouncing_p is set to 't' in the users table.
I have tested this lightly with bulk-mail and notifications.
Currently oracle support is limited. Anybody that has a working oracle installation a bit for time feel free to update the oracle sql.
The following problems will be worked on this week:
Schedule processes to remove bounces older than 10 days and remove old notifications that did not receive a bounce after 24 hours.
Additional parameters to allow adminstrators change the queue run period, bounce measurement period, and the notifcation monitor period.
Documentation.
Clean up various bits of the code.
We need to decieded on a way to advise the user their mail is currently bouncing to the system and has been disabled. We should also include a way to allow the user to verify their email address to re-enable email.
Note: I bumped up the version of acs-mail-lite to 0.36a. A for alpha and .36 because my parents celebrate 36 years of wedding bliss today :->
You can download the code:
acs-mail-lite-0.36a.tar.gz
at
http://shell1.alal.net/~kovachme
or
http://www.alal.com/files (in the openacs directory)
Questions, comments, patches? please send to mailto:mkovach@alal.com.
