Forum OpenACS Q&A: RFC: New system for incoming and outgoing email

1: RFC: New system for incoming and outgoing email

Posted by Talli Somekh on 05/03/04 09:15 PM

Musea needs to build some email intensive applications for a couple of projects. As a result, we've designed a new system for incoming and outgoing email, as a replacement for acs-mail-*. The design docs can be found here

The summaries of both docs are:

Incoming mail:

The purpose of this document is to describe a new universal method to allow email to be injected into OpenACS system. The goals of this system will be:
Independence: Administrators should be able to use any common UNIX mail transfer agents (MTA). OpenACS should not require a specific mail server.
Security: While complete security is hard to maintain in any dynamic system good practices should be used to minimize potential security related issues.
Outgoing mail:
The purpose of the document is describe a new universal method for handling outgoing mail in OpenACS. The goals of this system will be to:
Interface with the new OpenACS Incoming Mail Handler to set the correct headers for outgoing mail.
Allow for multiple queues. Packages should be able to create separate outoing mail queues to control deliver of mail messages.
One time and periodic queue deliver. Queues should be able to be controlled when they will deliver mail and if the queue will be used once and destroyed or run on a periodic basis for a specific or unlimited amount of time.
Message creators and administrators should be able to freeze queues and messages in queues.
Administrators and delegated users should be able to edit messages in the queues.

Comments are encouraged and welcome.

talli

2: Re: RFC: New system for incoming and outgoing email (response to 1)

Posted by Malte Sussdorff on 05/04/04 09:04 AM

Hi Talli,

if you take a closer look at acs-mail-lite in 5-1 you will see it does bounce management already and is prepared to do most of your other things as well. As for outgoing mail, the mailing-list manager module in contrib is already prepared to handle some of the cases and just needs to be ripped apart and stitched into acs-mail :).

A more detailed answer later.

3: Re: RFC: New system for incoming and outgoing email (response to 1)

Posted by Mat Kovach on 05/04/04 03:27 PM

The object is to describe a system and procedure for incoming and outgoing mail. Specifically we didn't look INTO current packages but design a reasonable way to work with email in the OpenACS system.

The idea is not to describe a current package but to help develop a plan how FUTURE packages should look. If some of the concepts are already implemented all the better.

I know that the latest version of OpenACS's mail lite does this but the real point is not about specific packages but if people agree with the general ideas. More specific documentation can be found at:

http://dev.museatech.net/mail-dev/Files/

4: Re: RFC: New system for incoming and outgoing email (response to 3)

Posted by Jeff Davis on 05/04/04 11:02 PM

Mat, on the incoming mail doc I had some questions. You say:

The separator will be a system wide parameter with a requirement that the unique tag will not be allowed to contain the separator. Packages will be allowed to configure unique tags and domains. The additional information will be passed to packages but will not be used internally by the incoming email handler.

and

The incoming processor would then call the service contract for the package, if any, handling mail for the unique tag/domain.

Is the implementation key for the service contract the "unique" tag + the domain? Is the unique tag supposed to be the package key? I don't really see the connection between unique tag, packages and a particular service contract.

Also you say:

The separator will be a system wide parameter

If the separator can be changed, you should probably provide a function to construct the reply address from the components (take the pathological case where I choose the letter "l" for my seperator -- packages with l in them would break unless there was something that could escape and unescape l's).

You should probably define the behavior when the service contract returns an error as well.

I suspect we should also include the user_id as well as the users email in the envelope since you want to be able to recover which user the bounce belonged to even if they have changed their email address in the meantime.

5: Re: RFC: New system for incoming and outgoing email (response to 4)

Posted by Mat Kovach on 05/05/04 12:06 AM

Jeff,

Hey, it made a lot of since to me.

I should clarify.

The seperate will be something like:

-, +, |, etc. Now that you have raised the question I think a simple list of possible values that can be choosen would be better.

The TAG would be something else. For notificatitons you could set that as 'notifications'. I do plan on creating a function to create proper strings. That function will give you the options of also using the user_id in the string OR the email address. Yes using the email address on OpenACS does cause the problem.

The unique tag COULD be the package name, it would be something that is defined.

Maybe an example might help:

Say we have a server where '+' is the seperator and the domain is mek.cc. You create a few forums and give each one a unique tag, say qanda and development (this would be added to the forum administration).

This would create the following email addresses that could be used:

mailto:qanda@mek.cc
mailto:qanda+user_id+message_id@mek.cc
mailto:developer@mek.cc
mailto:developer+user_id+message_id@mek.cc

The incoming processor would do the following:

Look at the email and parse it, basically looking
for the <identifier>@mek.cc or <identifier+stuff>@mek.cc.
Once it has the indentifier it will send the message to the proper area. There the forum package could say:

Okay qanda is for forum Q and A. If the message has
no extra stuff we create a new forum thread with
this subject but we check the From address to make sure
the user is a valid user. If the message has extra stuff,
we make sure the user_id is valid and post as based on the message id. But this part would be up to the package
maintainer.

Does this clear things up a bit?

6: Re: RFC: New system for incoming and outgoing email (response to 5)

Posted by Jeff Davis on 05/05/04 09:52 AM

I guess given this example I am still sort of stuck knowing how the unique tag is mapped to a service contract. Above you say "Okay qanda is for forum Q and A." but I don't see anything in the design that lets you tell that. Especially since qanda maps to something below a package (a forum). Is there a mapping table that says:

unique tag   package   extra stuff
qanda        forums    {forum_id 200}

and you invoke a "forums" service contract for forums with the "extra stuff" passed in to indicate which forum it is?

Or do you invoke a service contract implementation "qanda" which has the forum_id either curried in or require that forums maintain it's own table to map unique tag to forum_id and bind a service contract per unique tag?

7: Re: RFC: New system for incoming and outgoing email (response to 5)

Posted by Malte Sussdorff on 05/05/04 11:15 AM

In my opinion the unique tag should be named after the service contract that is executed in it's behalf. In the case of forums it should be called "forums" and the additional information would contain items like "forum_id", "parent_id" (if in reply to something). Alternatively it would be "bounce" for the bounce management service contract. You should also keep in mind that you want to support multiple packages of one package type, so you need to pass on the package_id as well.

In addition to the default naming (e.g. forums, bounce, blogger), you should have a mechanism to define named special cases. E.g. you will map the "qanda" unique tag to the "forums" service contract with prefilled value for forum_id. For obvious reason, the user interface needs to be well thought through and we should *NOT* have to create the named special cases by going into the code.

Some more remarks based on previous challanges:

How do you make sure that all mails belonging to OpenACS are delivered to the mail-dir, but not the other mails, if you do not know which packages are installed? Wouldn't it make sense to have a prefix like "oacs" for each email address (incoming and outgoing) so that the MTA can differentiate between mailto:malte@sussdorff.de and mailto:oacs-malte@sussdorff.de, where the later goes to the "malte" package at the same domain ?

This is especially true if you want to allow easy postings to forums. The Email should be easy to remember (e.g. mailto:Forum-openacs_dev@openacs.org). Alternatively you can configure your MTA to try all other delivery methods first and only if all failed, store the Email in the maildir for OpenACS.

Furthermore, make sure you do not interfere between multiple installations of your mail-package on the same site. Therefore the system name (from your aolserver config) should make it into the mail scanning as well, to prevent server1 reading and storing mails destined for server2.

8: Re: RFC: New system for incoming and outgoing email (response to 3)

Posted by Malte Sussdorff on 05/05/04 11:38 AM

This is all fair and good, but my question points to the fact that this *again* reinvents the wheel. Take bounce management for example:

What you describe for bounce management is already in acs-mail-lite, unless you *demand* that you use the email address instead of the user_id (which I find more convenient). It would be something like bounce-<user_id>-<optional unique tag>-<optional additional information>@somedomain.com.

Though this is only a minor change (email vs. user_id) it would yet again change the behaviour of the acs-mail package.

Talking about bounces, how would you handle bounces due to "out of office" replies ?

Furthermore, do you plan to add multipart email handling to the mail package, how about alternative emails, do you want to include internationlization using quoted printables (or other methods) and last but not least what about attachements? Will you create a unique message identifier with each sent mail ?

You talk about queues in your outgoing document. Does this mean you think also about having multiple SMTP servers (which is what I'd understand under a queue for mail sending). Otherwise I do not really understand the need for special queues, as this could be handled by an addition to the notification package where you would see, which packages have what scheduled notifications, what has been send through the notification package lately and (if done correctly) how many messages bounced for a certain mailing. This would enable us to use the same mechanisms you describe not only for mail but also for SMS, IM and whatever other methods of notification we happen to create in the future.

I think the whole queue description (as per your document) belongs to notifications, not to the system for incoming and outgoing mail. Though, obviously the outgoing mail system needs to have an interface with notifications to deliver the information about sent mails and such.

9: Re: RFC: New system for incoming and outgoing email (response to 8)

Posted by Jeff Davis on 05/05/04 03:47 PM

Talking about bounces, how would you handle bounces due to "out of office" replies ?

An out of office reply won't come to the bounce address as the bounce address is the envelope header, not the reply-to or from addresses.

If you mean how will you distinguish an "out of office" reply from a regular reply, that is a good question and I for one would like to have that be something the mail handler figured out (and possibly passed down to the service contract as an advisory bit of information).

I would like to be able to map a short unique identifiers which are not a package names or adorned with any extra information into something which knows to invoke the forums contract for a given forum_id (as an example).

I don't want to see the wheel reinvented, but I would like to see this stuff work a lot better than it does now and in particular I would like to make non-notifications based inbound email work in a few contexts (bug state manipulation via email, posting photos and file storage objects via attachments, etc where there is no initial notification to reply to for example).

10: Re: RFC: New system for incoming and outgoing email (response to 9)

Posted by Tilmann Singer on 05/09/04 11:47 PM

A major aspect of any outgoing mail system is mime support IMHO. At least choosing between text/plain and text/html when sending should be possible. Some people may argue it should support multipart/alternative as well - I don't really think it is necessary (the current acs-mail package supports it, acs-mail-lite afaik doesn't). Attachments of any mime type should be possible.

What I think is important too is the character encoding of mails - I don't know how widespread utf-8 capable mail clients are. If they are rare it would be nice to be able to encode the mail in the recipient users' preferred encoding.

Representing hierarchy with References and In-Reply-To headers - some way of tracking message_ids so that we can send out mails that contain information about their parent messages, so that they find their correct place in the threaded view of mail programs that read those headers. Typical example are forum threads.

11: Re: RFC: New system for incoming and outgoing email (response to 1)

Posted by Rocael Hernández Rizzardini on 09/18/04 06:44 PM

Talli & Mat,
what's the state of this?
was finally developed?

BTW, I would like to have a look at your document, would you pls send it to me? (the link is not working anymore)

The reason I ask is because we are looking for that functionality and if its not there, we might start the develoment...

TIA

12: Re: RFC: New system for incoming and outgoing email (response to 1)

Posted by Jun Yamog on 09/30/04 07:19 AM

Hi,

It has been already developed. Although I don't know the exact details, you can ask Mat on that.

13: Re: RFC: New system for incoming and outgoing email (response to 12)

Posted by Rocael Hernández Rizzardini on 09/30/04 03:56 PM

but Mat's email doesn't work :(