E-Mail: Incoming E-Mail
Created by Malte Sussdorff, last modified by Gustaf Neumann 24 Aug 2020, at 01:43 PM
Incoming E-Mail in OpenACS works with the latest version of acs-mail-lite in a general fashion using callbacks.
The original version of this documentation is found via archive.org at: http://www.cognovis.de/developer/en/incoming_email
We will take a look on what needs to be done to get incoming e-mail working and then continue on to see how packages can benefit.
Project notes: ACS Mail Lite sends via SMTP which permits the use of an external server to handle email. For scalability, consider expanding the incoming E-mail paradigm to likewise use Tcllib's imap4 or NaviServer's nsimap so that most all email can be handled on separate servers.
Install incoming E-Mail
First, one must have an understanding of postfix basics. See http://www.postfix.org/BASIC_CONFIGURATION_README.html.
These instructions use the following example values:
- hostname: www.yourserver.com
- oacs user: service0
- OS: Linux
- email user: service0
- email's home dir: /home/service0
- email user's mail dir: /home/service0/mail
Important: The email user service0 does not have a ".forward" file. This user is only used for running the OpenACS website. Follow careful use of email rules by following strict guidelines to avoid email looping back unchecked.
For postfix, the email user and oacs user do not have to be the same. Furthermore, postfix makes distinctions between virtual users and user aliases. Future versions of this documentation should use examples with different names to help distinguish between standard configuration examples and the requirements of ACS Mail Lite package.
Postfix configuration parameters:
myhostname=www.yourserver.com myorigin=$myhostname inet_interfaces=$myhostname, localhost mynetworks_style=host virtual_alias_domains = www.yourserver.com virtual_maps=regexp:/etc/postfix/virtual home_mailbox=mail/
Here is the sequence to follow if installing email service on system for first time. If your system already has email service, adapt these steps accordingly:
- Install postfix
- Install smtp (for postfix)
- Install metamail (for acs-mail-lite)
- Edit /etc/postfix/main.cf
- Edit /etc/postfix/virtual Add a regular expression to filter relevant incoming emails for processing by OpenACS.
@www.yourserver.com service0
- Edit /etc/postfix/master.cf - uncomment this line so postfix listens to emails from internet
smtp inet n - n - - smtpd
- Create a mail directory as service0
mkdir /home/service0/mail
- Configure ACS Mail Lite parameters
BounceDomain: www.yourserver.com
BounceMailDir: /home/service0/mail
EnvelopePrefix: bounce
The EnvelopePrefix is for bounce e-mails only.
NOTE: Parameters should be renamed:
BounceDomain to IncomingDomain
BounceMailDir to IncomingMaildir
EnvelopePrefix to BouncePrefix
..to reflect that acs-mail-lite is capable of dealing with other types of incoming e-mail.
Furthermore, setting IncomingMaildir parameter clarifies that incoming email handling is setup. This is useful for other packages to determine if they can rely on incoming e-mail working (e.g. to set the reply-to email to an e-mail address which actually works through a callback if the IncomingMaildir parameter is enabled). - Configure Notifications parameters
EmailReplyAddressPrefix: notification
Configure other packages likewise
EmailQmailQueueScanP: 0
We want acs-mail-lite incoming handle the Email Scanning, not each package separately.
- Invoke postmap in OS shell to recompile virtual db:
postmap /etc/postfix/virtual
- Restart Postfix.
/etc/init.d/postfix restart
- Restart OpenACS
Processing incoming e-mail
A sweeper procedure like acs_mail_lite::load_mails should:
- scan the e-mails which are in the IncomingMaildir directory on a regular basis.
- check if any email came from an auto mailer.
- Parse new ones, and
- process them by firing off callbacks.
Vinod has made a check for auto mailers by using procmail as follows. Maybe we could get this dragged into Tcl code (using regexp or a Procmail recipe parser) instead, thereby removing the need for setting up procmail in the first place.
:0 w * ^subject:.*Out of Office AutoReply /dev/null :0 w * ^subject:.*Out of Office /dev/null :0 w * ^subject:.*out of the office /dev/null :0 w * ^subject:.*NDN /dev/null :0 w * ^subject:.*[QuickML] Error: /dev/null :0 w * ^subject:.*autoreply /dev/null :0 w * ^from.*mailer.*daemon /dev/null
To make things granular a separate parsing procedure should deal with loading the e-mail into the Tcl interpreter and setting variables in an array for further processing.
ad_proc parse_email {
-file:required
-array:required
} {
...
}
An email is split into several parts: headers, bodies and files.
The headers consists of a list with header names as keys and their corresponding values. All keys are lower case.
The bodies consists of a list with two elements: content-type and content.
The files consists of a list with three elements: content-type, filename and content.
An array with all the above data is upvarred to the caller environment.
Processing an email should result in an array like this:
HEADERS
- message_id
- subject
- from
- to
- date
- received
- references
- in-reply-to
- return-path
- .....
X-Headers:
- X-Mozilla-Status
- X-Virus Scanned
- .....
We do not know which headers are going to be available in the e-mail. We set all headers found in the array. The callback implementation then checks if a certain header is present or not.
#get all available headers
set keys [mime::getheader $mime -names]
set headers [list]
# create both the headers array and all headers directly for the email array
foreach header $keys {
set value [mime::getheader $mime $header]
set email([string tolower $header]) $value
lappend headers [list $header $value]
}
set email(headers) $headers
Bodies
An e-mail usually consists of one or more bodies. With the advent of complex_send, OpenACS supports sending of multi-part e-mails which are needed if you want to send out and e-mail in text/html and text/plain (for old mail readers).
switch [mime::getproperty $part content] {
"text/plain" {
lappend bodies [list "text/plain" [mime::getbody $part]]
}
"text/html" {
lappend bodies [list "text/html" [mime::getbody $part]]
}
}
Files
OpenACS supports tcllib mime functions. Getting incoming files to work is a matter of looking for a part where there exists a "Content-disposition" part. All these parts are file parts. Together with scanning for email bodies, code looks something like this:
set bodies [list]
set files [list]
#now extract all parts (bodies/files) and fill the email array
foreach part $all_parts {
# Attachments have a "Content-disposition" part
# Therefore we filter out if it is an attachment here
if {[catch {mime::getheader $part Content-disposition}]} {
switch [mime::getproperty $part content] {
"text/plain" {
lappend bodies [list "text/plain" [mime::getbody $part]]
}
"text/html" {
lappend bodies [list "text/html" [mime::getbody $part]]
}
}
} else {
set encoding [mime::getproperty $part encoding]
set body [mime::getbody $part -decode]
set content $body
set params [mime::getproperty $part params]
if {[lindex $params 0] == "name"} {
set filename [lindex $params 1]
} else {
set filename ""
}
# Determine the content_type
set content_type [mime::getproperty $part content]
if {$content_type eq "application/octet-stream"} {
set content_type [ns_guesstype $filename]
}
lappend files [list $content_type $encoding $filename $content]
}
}
set email(bodies) $bodies
set email(files) $files
Note that the files ie attachments are actually stored in the /tmp directory from where they can be processed further. It is up to the callback to decide if to import the file into OpenACS or not. Once all callbacks have been fired files in /tmp will have to be deleted again though.
Firing off callbacks
Now that we have the e-mail parsed and have an array with all the information, we can fire off the callbacks. The firing should happen in two stages.
The first stage is where we support a syntax like "object_id@yoursite.com".
Second, incoming e-mail could look up the object_type, and then call the callback implementation specific to this object_type. If object_type = 'content_item', use content_type instead.
ad_proc -public -callback acs_mail_lite::incoming_object_email { -array:required -object_id:required } { }
callback acs_mail_lite::incoming_object_email -impl $object_type -array email -object_id $object_id
ad_proc -public -callback acs_mail_lite::incoming_object_email -impl user {
-array:required
-object_id:required
} {
Implementation of mail through support for incoming emails
} {
# get a reference to the email array
upvar $array email
# make the bodies an array
template::util::list_of_lists_to_array $email(bodies) email_body
if {[exists_and_not_null email_body(text/html)]} {
set body $email_body(text/html)
} else {
set body $email_body(text/plain)
}
set reply_to_addr "[party::get_by_email $email(from)]@[ad_url]"
acs_mail_lite::complex_send \
-from_addr $from_addr \
-reply_to $reply_to_addr \
-to_addr $to_addr \
-subject $email(subject) \
-body $body \
-single_email \
-send_immediately
}
Object id based implementations are useful for automatically generating "reply-to" addresses. With ProjectManager and Contacts object_id is also handy, because Project / TaskID is prominently placed on the website. If you are working on a task and you get an e-mail by your client that is related to the task, just forward the email to "$task_id@server.com" and it will be stored along with the task. Highly useful :).
Obviously you could have implementations for:
-
forums_forum_id: Start a new topic
-
forums_message_id: Reply to an existing topic
-
group_id: Send an e-mail to all group members
-
pm_project_id: add a comment to a project
-
pm_task_id: add a comment to a task and store the files in the projects folder (done)
Once the e-mail is dealt with in an object oriented approach we are either done with the message (an object_id was found in the to address) or we need to process it further.
ad_proc -public -callback acs_mail_lite::incoming_email {
-array:required
-package_id
} {
}
array set email {}
parse_email -file $msg -array email
set email(to) [parse_email_address -email $email(to)]
set email(from) [parse_email_address -email $email(from)]
# We execute all callbacks now
callback acs_mail_lite::incoming_email -array email
For this a general callback should exist which can deal with every leftover e-mail and each implementation will check if it wants to deal with this e-mail. How is this check going to happen? As an example, a package could have a prefix, as is the case with bounce e-mails as handled in acs_mail_lite::parse_bounce_address (see below):
ad_proc -public -callback acs_mail_lite::incoming_email -impl acs-mail-lite {
-array:required
-package_id:required
} {
@param array An array with all headers, files and bodies. To access the array you need to use upvar.
@param package_id The package instance that registered the prefix
@return nothing
@error
} {
upvar $array email
set to [acs_mail_lite::parse_email_address -email $email(to)]
ns_log Debug "acs_mail_lite::incoming_email -impl acs-mail-lite called. Recepient $to"
util_unlist [acs_mail_lite::parse_bounce_address -bounce_address $to] user_id package_id signature
# If no user_id found or signature invalid, ignore message
# Here we decide not to deal with the message anymore
if {[empty_string_p $user_id]} {
if {[empty_string_p $user_id]} {
ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: No equivalent user found for $to"
} else {
ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Invalid mail signature $signature"
}
} else {
ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Bounce checking $to, $user_id"
if { ![acs_mail_lite::bouncing_user_p -user_id $user_id] } {
ns_log Debug "acs_mail_lite::incoming_email impl acs-mail-lite: Bouncing email from user $user_id"
# record the bounce in the database
db_dml record_bounce {}
if {![db_resultrows]} {
db_dml insert_bounce {}
}
}
}
}
Alternatively we could just check the whole to address for other things, e.g. if the to address belongs to a group (party)
ad_proc -public -callback acs_mail_lite::incoming_email -impl contacts_group_mail {
-array:required
{-package_id ""}
} {
Implementation of group support for incoming emails
If the to address matches an address stored with a group then send out the email to all group members
@author Malte Sussdorff (malte.sussdorff@cognovis.de)
@creation-date 2005-12-18
@param array An array with all headers, files and bodies. To access the array you need to use upvar.
@return nothing
@error
} {
# get a reference to the email array
upvar $array email
# Now run the simplest mailing list of all
set to_party_id [party::get_by_email -email $email(to)]
if {[db_string group_p "select 1 from groups where group_id = :to_party_id" -default 0]} {
# make the bodies an array
template::util::list_of_lists_to_array $email(bodies) email_body
if {[exists_and_not_null email_body(text/html)]} {
set body $email_body(text/html)
} else {
set body $email_body(text/plain)
}
acs_mail_lite::complex_send \
-from_addr [lindex $email(from) 0] \
-to_party_ids [group::get_members -group_id $to_party_id] \
-subject $email(subject) \
-body $body \
-single_email \
-send_immediately
}
}
Or check if the to address follows a certain format.
ad_proc -public -callback acs_mail_lite::incoming_email -impl contacts_mail_through {
-array:required
{-package_id ""}
} {
Implementation of mail through support for incoming emails
You can send an e-amil through the system by sending it to user#target.com@yoursite.com
The email will be send from your system and if mail tracking is installed the e-mail will be tracked.
This allows you to go in direct communication with a customer using you standard e-mail program instead of having to go to the website.
@author Malte Sussdorff (malte.sussdorff@cognovis.de)
@creation-date 2005-12-18
@param array An array with all headers, files and bodies. To access the array you need to use upvar.
@return nothing
@error
} {
# get a reference to the email array
upvar $array email
# Take a look if the email contains an email with a "#"
set pot_email [lindex [split $email(to) "@"] 0]
if {[string last "#" $pot_email] > -1} {
....
}
}
Alternatives to this are:
- ${component_name}-bugs@openacs.org (where component_name could be openacs or dotlrn or contacts or whatever), to store a new bug in bug-tracker
- username@openacs.org (to do mail-through using the user name, which allows you to hide the actual e-mail of the user whom you are contacting).
Cleanup
Once all callbacks have been fired off, e-mails need to be deleted from the Maildir directory and files which have been extracted need to be deleted as well from the /tmp directory.
